I have seen a couple blog entries about executing MSTest unit tests from MSBuild. Most recently I saw the entry by Scott A. Lawrence. So I decided to share how I execute MSTest unit tests from MSBuild
I created a file named Build.Common.UnitTest.targets which contains all the behavior that will execute the test cases. This file can then be imported into whatever scripts that need to execute test cases. The entire file is shown below. We will discuss afterwards.
Build.Common.UnitTest.targets
<?xml
version="1.0" encoding="utf-8"?>
<Project
ToolsVersion="3.5" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<!-- =======================================================================
TESTING
======================================================================= -->
<!-- Default build settings go here -->
<PropertyGroup>
<RunMSTest Condition="'$(RunMSTest)'==''">true</RunMSTest>
<BuildInParallel Condition="'$(BuildInParallel)'==''">true</BuildInParallel>
</PropertyGroup>
<Target Name="MSTestValidateSettings">
<!-- Cleare out these items -->
<ItemGroup>
<_RequiredProperties Remove="@(_RequiredProperties)"/>
<_RequiredItems Remove="@(_RequiredItems)"/>
</ItemGroup>
<ItemGroup>
<_RequiredProperties Include="BuildContribRoot">
<Value>$(BuildContribRoot)</Value>
</_RequiredProperties>
<_RequiredProperties Include="OutputRoot">
<Value>$(OutputRoot)</Value>
</_RequiredProperties>
<_RequiredItems Include="MSTestProjects">
<RequiredValue>@(MSTestProjects)</RequiredValue>
<RequiredFilePath>%(MSTestProjects.FullPath)</RequiredFilePath>
</_RequiredItems>
<_RequiredItems Include="AllConfigurations">
<RequiredValue>@(AllConfigurations)</RequiredValue>
</_RequiredItems>
<_RequiredItems Include="AllConfigurations.Configuration">
<RequiredValue>%(AllConfigurations.Configuration)</RequiredValue>
</_RequiredItems>
</ItemGroup>
<!-- Raise an error if any value in _RequiredProperties is missing -->
<Error Condition="'%(_RequiredProperties.Value)'==''"
Text="Missing required property [%(_RequiredProperties.Identity)]"/>
<!-- Raise an error if any value in _RequiredItems is empty -->
<Error Condition="'%(_RequiredItems.RequiredValue)'==''"
Text="Missing required item value [%(_RequiredItems.Identity)]" />
<!-- Validate any file/directory that should exist -->
<Error Condition="'%(_RequiredItems.RequiredFilePath)' != '' and
!Exists('%(_RequiredItems.RequiredFilePath)')"
Text="Unable to find expeceted path
[%(_RequiredItems.RequiredFilePath)] on item [%(_RequiredItems.Identity)]" />
</Target>
<UsingTask
TaskName="TestToolsTask"
AssemblyFile="$(BuildContribRoot)TestToolsTask-1.3\Microsoft.VisualStudio.QualityTools.MSBuildTasks.dll"/>
<!-- TODO: Create a ValidateTestSettings target and put it on this list -->
<PropertyGroup>
<MSTestDependsOn>
BuildMSTestProjects;
BeforeMSTest;
CoreMSTest;
AfterMSTest
$(MSTestDependsOn);
</MSTestDependsOn>
</PropertyGroup>
<Target Name="MSTest" DependsOnTargets="$(MSTestDependsOn)"/>
<Target Name="BeforeMSTest"/>
<Target Name="AfterMSTest"/>
<Target Name="CoreMSTest" Outputs="%(MSTestProjects.Identity)">
<Message Text="Running MSTest for project [%(MSTestProjects.Identity)]"/>
<Message Text="MSTestProjects.Directory:
%(MSTestProjects.RootDir)%(MSTestProjects.Directory)" />
<PropertyGroup>
<_CurrentConfig>Debug</_CurrentConfig>
</PropertyGroup>
<PropertyGroup>
<_SearchPath>$(OutputRoot)$(_CurrentConfig)\%(MSTestProjects.Filename)\</_SearchPath>
<_TestContainers></_TestContainers>
</PropertyGroup>
<TestToolsTask
SearchPathRoot="%(MSTestProjects.RootDir)%(MSTestProjects.Directory)"
TestContainers="$(OutputRoot)$(_CurrentConfig)\%(MSTestProjects.Filename)\bin\%(MSTestProjects.Filename).dll"/>
<!-- TODO: Read in the results of the tests and get all failures and report
them here -->
</Target>
<PropertyGroup>
<BuildMSTestProjectsDependsOn>
BeforeBuildMSTestProjects;
CoreBuildMSTestProjects;
AfterBuildMSTestProjects;
$(BuildMSTestProjectsDependsOn);
</BuildMSTestProjectsDependsOn>
</PropertyGroup>
<Target Name="BuildMSTestProjects" DependsOnTargets="$(BuildMSTestProjectsDependsOn)"/>
<Target Name="BeforeBuildMSTestProjects"/>
<Target Name="AfterBuildMSTestProjects"/>
<Target Name="CoreBuildMSTestProjects"
Outputs="%(AllConfigurations.Configuration)"
DependsOnTargets="CleanMSTest">
<!-- Make sure to do clean build in case test cases were added -->
<PropertyGroup>
<_CurrentConfig>%(AllConfigurations.Configuration)</_CurrentConfig>
</PropertyGroup>
<Message Text="Building (MSTestProjects.Identity)
%(MSTestProjects.Identity)" Importance="high"/>
<!-- Build the projects here. -->
<MSBuild Projects="%(MSTestProjects.Identity)"
Properties="Configuration=$(_CurrentConfig);
OutputPath=$(OutputRoot)$(_CurrentConfig)\%(MSTestProjects.Filename)\bin\;
BaseIntermediateOutputPath=$(OutputRoot)$(_CurrentConfig)\%(MSTestProjects.Filename)\obj\;
GenerateResourceNeverLockTypeAssemblies=true;
%(ProjectsToBuild.Properties);
$(AllProjectProperties);"
BuildInParallel="$(BuildInParallel)"
>
</MSBuild>
</Target>
<Target Name="CleanMSTest">
<MSBuild Projects="@(MSTestProjects)"
Targets="Clean"
Properties="Configuration=$(_CurrentConfig);
OutputPath=$(OutputRoot)$(_CurrentConfig)\%(MSTestProjects.Filename)\bin\;
BaseIntermediateOutputPath=$(OutputRoot)$(_CurrentConfig)\%(MSTestProjects.Filename)\obj\;
GenerateResourceNeverLockTypeAssemblies=true;"
BuildInParallel="$(BuildInParallel)"
/>
</Target>
</Project>
There are five important targets which are described below.
Name | Description |
MSTestValidateSettings | This validates that the file was provided the needed data values to perform its task, to run the unit tests. For more info on this technique see my previous entry Elements of Reusable MSBuild Scripts: Validation. |
MSTest | This is the target that you would execute to run the test cases. The target itself is empty but it sets up the chain of dependent targets. |
CoreMSTest | This is the target which executes the test cases. This is preformed using the TestToolsTask. |
BuildMSTestProjects | This target is responsible for building (i.e. compiling) the projects which contain the test cases. You don't have to call this it is called automagically. |
CleanMSTest | This target will execute the Clean target for all the test projects defined. |
If you take a look at the CoreBuildMSTestProjects target you can see that I am batching (Target batching to be specific) it on each defined configuration. This is achieved with the attribute Outputs="%(AllConfigurations.Configuration)". If you are not familiar with batching, see the links at the end of this post for more details, and you can always grab my book for even more detailed info J. Then inside that target I build each project by batching (Task batching) the MSBuild task on each project defined in the MSTestProjects item list.
Then inside the CoreMSTest target I execute the test cases. This target is batched for every value in the MSTestProjects item. As I'm writing this I have noticed that I've hard-coded the value for the configuration used in that target to be Debug with the statement
<PropertyGroup>
<_CurrentConfig>Debug</_CurrentConfig>
</PropertyGroup>
This shouldn't be hard coded, but passed in. I will leave it as is for now though. Then the TestToolsTask is invoked to execute the test cases.
Now that we have written the re-usable .targets file to execute the test cases we need to create a file which will "feed" it the necessary data values and let it do its magic. I created a sample solution, which you can download at the end of this post, which demonstrates its usage. The solution is named MSTestExample and you can see the files it contains in the screen shot below.
Here I've highlighted the two MSTest projects as well as a couple build files. I've already shown the contents of the Build.Common.UnitTest.targets file. Here is the contents of the Build.MSTestExample.proj file.
Build.MSTestExample.proj
<?xml
version="1.0" encoding="utf-8"?>
<Project
ToolsVersion="3.5" DefaultTargets="MSTest" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<!--
Some
properties you may be interested in setting:
Name Description
*****************************************************
RunMSTest true/false to
run unit tests or not
-->
<PropertyGroup>
<Root Condition="'$(Root)'==''">$(MSBuildProjectDirectory)\..\</Root>
<BuildRoot Condition="'$(BuildRoot)'==''">$(Root)Build\</BuildRoot>
<BuildContribRoot Condition="'$(BuildContribRoot)'==''">$(BuildRoot)Contrib\</BuildContribRoot>
<SourceRoot Condition="'$(SourceRoot)'==''">$(Root)</SourceRoot>
<BuildArtifactsRoot Condition="'$(BuildArtifactsRoot)'==''">$(BuildRoot)BuildAftifacts\</BuildArtifactsRoot>
<OutputRoot Condition="'$(OutputRoot)'==''">$(BuildArtifactsRoot)Output\</OutputRoot>
<TreatWarningsAsErrors Condition=" '$(TreatWarningsAsErrors)'=='' ">true</TreatWarningsAsErrors>
</PropertyGroup>
<PropertyGroup>
<CodeAnalysisTreatWarningsAsErrors Condition="'$(CodeAnalysisTreatWarningsAsErrors)'==''">true</CodeAnalysisTreatWarningsAsErrors>
<BuildInParallel Condition="'$(BuildInParallel)'==''">true</BuildInParallel>
<RunCodeAnalysis Condition="'$(RunCodeAnalysis)'==''">true</RunCodeAnalysis>
<RunMSTest Condition="'$(RunMSTest)'==''">false</RunMSTest>
<RunStyleCop Condition="''=='$(RunStyleCop)'">true</RunStyleCop>
</PropertyGroup>
<!-- Configurations that we want to build for -->
<ItemGroup>
<AllConfigurations Include="Debug">
<Configuration>Debug</Configuration>
</AllConfigurations>
</ItemGroup>
<!-- =======================================================================
MSTest
======================================================================= -->
<ItemGroup>
<MSTestProjects Include="$(SourceRoot)TestProject1\TestProject1.csproj">
</MSTestProjects>
<MSTestProjects Include="$(SourceRoot)TestProject2\TestProject2.csproj">
</MSTestProjects>
</ItemGroup>
<Import Project="$(BuildRoot)Build.Common.UnitTest.targets"/>
<!-- Inject the MSTest targets into the build -->
<!--<PropertyGroup Condition="'$(RunMSTest)'=='true'">
<BuildDependsOn>
$(BuildDependsOn);
MSTest;
</BuildDependsOn>
<BuildMSTestProjectsDependsOn>
CoreBuild;
$(BuildMSTestProjectsDependsOn)
</BuildMSTestProjectsDependsOn>
</PropertyGroup>-->
</Project> This file is pretty simple. It just creates some properties, and items and then just imports the Build.Common.UnitTest.targets file to do all the heavy lifting. You will probably notice that there are some properties defined that don't make sense here, like CodeAnalysisTreatWarningsAsErrors, this is because this was taken from a build script which does some other tasks. You can ignore those. To see what properties/items are required for the MSTest just look at the MSTestValidateSettings target. Also in the previous code snippet I showed how you could inject the MSTest target into the build process but it is commented out since there is no real build process in this case.
One thing about this approach that may not be ideal is that it will execute the test cases in one assembly at a time and if there is a failure it will not go on to the other assemblies. In my case this is OK because this is for local builds, public builds are executing test cases by Team Build, but this can be looked at.
Questions? Comments?
MSTestExamples.zip
MSBuild Batching Links
Sayed Ibrahim Hashim