I have seen a couple blog entries about executing MSTest unit tests from MSBuild. Most recently I saw the entry by Scott A. Lawrence. So I decided to share how I execute MSTest unit tests from MSBuild
I created a file named Build.Common.UnitTest.targets which contains all the behavior that will execute the test cases. This file can then be imported into whatever scripts that need to execute test cases. The entire file is shown below. We will discuss afterwards.
Build.Common.UnitTest.targets
xml
version="1.0" encoding="utf-8"?>
<Project
ToolsVersion="3.5" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<RunMSTest Condition="'$(RunMSTest)'==''">trueRunMSTest>
<BuildInParallel Condition="'$(BuildInParallel)'==''">trueBuildInParallel>
PropertyGroup>
<Target Name="MSTestValidateSettings">
<ItemGroup>
<_RequiredProperties Remove="@(_RequiredProperties)"/>
<_RequiredItems Remove="@(_RequiredItems)"/>
ItemGroup>
<ItemGroup>
<_RequiredProperties Include="BuildContribRoot">
<Value>$(BuildContribRoot)Value>
_RequiredProperties>
<_RequiredProperties Include="OutputRoot">
<Value>$(OutputRoot)Value>
_RequiredProperties>
<_RequiredItems Include="MSTestProjects">
<RequiredValue>@(MSTestProjects)RequiredValue>
<RequiredFilePath>%(MSTestProjects.FullPath)RequiredFilePath>
_RequiredItems>
<_RequiredItems Include="AllConfigurations">
<RequiredValue>@(AllConfigurations)RequiredValue>
_RequiredItems>
<_RequiredItems Include="AllConfigurations.Configuration">
<RequiredValue>%(AllConfigurations.Configuration)RequiredValue>
_RequiredItems>
ItemGroup>
<Error Condition="'%(_RequiredProperties.Value)'==''"
Text="Missing required property [%(_RequiredProperties.Identity)]"/>
<Error Condition="'%(_RequiredItems.RequiredValue)'==''"
Text="Missing required item value [%(_RequiredItems.Identity)]" />
<Error Condition="'%(_RequiredItems.RequiredFilePath)' != '' and
!Exists('%(_RequiredItems.RequiredFilePath)')"
Text="Unable to find expeceted path
[%(_RequiredItems.RequiredFilePath)] on item [%(_RequiredItems.Identity)]" />
Target>
<UsingTask
TaskName="TestToolsTask"
AssemblyFile="$(BuildContribRoot)TestToolsTask-1.3\Microsoft.VisualStudio.QualityTools.MSBuildTasks.dll"/>
<PropertyGroup>
<MSTestDependsOn>
BuildMSTestProjects;
BeforeMSTest;
CoreMSTest;
AfterMSTest
$(MSTestDependsOn);
MSTestDependsOn>
PropertyGroup>
<Target Name="MSTest" DependsOnTargets="$(MSTestDependsOn)"/>
<Target Name="BeforeMSTest"/>
<Target Name="AfterMSTest"/>
<Target Name="CoreMSTest" Outputs="%(MSTestProjects.Identity)">
<Message Text="Running MSTest for project [%(MSTestProjects.Identity)]"/>
<Message Text="MSTestProjects.Directory:
%(MSTestProjects.RootDir)%(MSTestProjects.Directory)" />
<PropertyGroup>
<_CurrentConfig>Debug_CurrentConfig>
PropertyGroup>
<PropertyGroup>
<_SearchPath>$(OutputRoot)$(_CurrentConfig)\%(MSTestProjects.Filename)\_SearchPath>
<_TestContainers>_TestContainers>
PropertyGroup>
<TestToolsTask
SearchPathRoot="%(MSTestProjects.RootDir)%(MSTestProjects.Directory)"
TestContainers="$(OutputRoot)$(_CurrentConfig)\%(MSTestProjects.Filename)\bin\%(MSTestProjects.Filename).dll"/>
Target>
<PropertyGroup>
<BuildMSTestProjectsDependsOn>
BeforeBuildMSTestProjects;
CoreBuildMSTestProjects;
AfterBuildMSTestProjects;
$(BuildMSTestProjectsDependsOn);
BuildMSTestProjectsDependsOn>
PropertyGroup>
<Target Name="BuildMSTestProjects" DependsOnTargets="$(BuildMSTestProjectsDependsOn)"/>
<Target Name="BeforeBuildMSTestProjects"/>
<Target Name="AfterBuildMSTestProjects"/>
<Target Name="CoreBuildMSTestProjects"
Outputs="%(AllConfigurations.Configuration)"
DependsOnTargets="CleanMSTest">
<PropertyGroup>
<_CurrentConfig>%(AllConfigurations.Configuration)_CurrentConfig>
PropertyGroup>
<Message Text="Building (MSTestProjects.Identity)
%(MSTestProjects.Identity)" Importance="high"/>
<MSBuild Projects="%(MSTestProjects.Identity)"
Properties="Configuration=$(_CurrentConfig);
OutputPath=$(OutputRoot)$(_CurrentConfig)\%(MSTestProjects.Filename)\bin\;
BaseIntermediateOutputPath=$(OutputRoot)$(_CurrentConfig)\%(MSTestProjects.Filename)\obj\;
GenerateResourceNeverLockTypeAssemblies=true;
%(ProjectsToBuild.Properties);
$(AllProjectProperties);"
BuildInParallel="$(BuildInParallel)"
>
MSBuild>
Target>
<Target Name="CleanMSTest">
<MSBuild Projects="@(MSTestProjects)"
Targets="Clean"
Properties="Configuration=$(_CurrentConfig);
OutputPath=$(OutputRoot)$(_CurrentConfig)\%(MSTestProjects.Filename)\bin\;
BaseIntermediateOutputPath=$(OutputRoot)$(_CurrentConfig)\%(MSTestProjects.Filename)\obj\;
GenerateResourceNeverLockTypeAssemblies=true;"
BuildInParallel="$(BuildInParallel)"
/>
Target>
There are five important targets which are described below.
Name | Description |
MSTestValidateSettings | This validates that the file was provided the needed data values to perform its task, to run the unit tests. For more info on this technique see my previous entry Elements of Reusable MSBuild Scripts: Validation. |
MSTest | This is the target that you would execute to run the test cases. The target itself is empty but it sets up the chain of dependent targets. |
CoreMSTest | This is the target which executes the test cases. This is preformed using the TestToolsTask. |
BuildMSTestProjects | This target is responsible for building (i.e. compiling) the projects which contain the test cases. You don't have to call this it is called automagically. |
CleanMSTest | This target will execute the Clean target for all the test projects defined. |
If you take a look at the CoreBuildMSTestProjects target you can see that I am batching (Target batching to be specific) it on each defined configuration. This is achieved with the attribute Outputs="%(AllConfigurations.Configuration)". If you are not familiar with batching, see the links at the end of this post for more details, and you can always grab my book for even more detailed info J. Then inside that target I build each project by batching (Task batching) the MSBuild task on each project defined in the MSTestProjects item list.
Then inside the CoreMSTest target I execute the test cases. This target is batched for every value in the MSTestProjects item. As I'm writing this I have noticed that I've hard-coded the value for the configuration used in that target to be Debug with the statement
<PropertyGroup>
<_CurrentConfig>Debug_CurrentConfig>
PropertyGroup>
This shouldn't be hard coded, but passed in. I will leave it as is for now though. Then the TestToolsTask is invoked to execute the test cases.
Now that we have written the re-usable .targets file to execute the test cases we need to create a file which will "feed" it the necessary data values and let it do its magic. I created a sample solution, which you can download at the end of this post, which demonstrates its usage. The solution is named MSTestExample and you can see the files it contains in the screen shot below.
Here I've highlighted the two MSTest projects as well as a couple build files. I've already shown the contents of the Build.Common.UnitTest.targets file. Here is the contents of the Build.MSTestExample.proj file.
Build.MSTestExample.proj
xml
version="1.0" encoding="utf-8"?>
<Project
ToolsVersion="3.5" DefaultTargets="MSTest" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<Root Condition="'$(Root)'==''">$(MSBuildProjectDirectory)\..\Root>
<BuildRoot Condition="'$(BuildRoot)'==''">$(Root)Build\BuildRoot>
<BuildContribRoot Condition="'$(BuildContribRoot)'==''">$(BuildRoot)Contrib\BuildContribRoot>
<SourceRoot Condition="'$(SourceRoot)'==''">$(Root)SourceRoot>
<BuildArtifactsRoot Condition="'$(BuildArtifactsRoot)'==''">$(BuildRoot)BuildAftifacts\BuildArtifactsRoot>
<OutputRoot Condition="'$(OutputRoot)'==''">$(BuildArtifactsRoot)Output\OutputRoot>
<TreatWarningsAsErrors Condition=" '$(TreatWarningsAsErrors)'=='' ">trueTreatWarningsAsErrors>
PropertyGroup>
<PropertyGroup>
<CodeAnalysisTreatWarningsAsErrors Condition="'$(CodeAnalysisTreatWarningsAsErrors)'==''">trueCodeAnalysisTreatWarningsAsErrors>
<BuildInParallel Condition="'$(BuildInParallel)'==''">trueBuildInParallel>
<RunCodeAnalysis Condition="'$(RunCodeAnalysis)'==''">trueRunCodeAnalysis>
<RunMSTest Condition="'$(RunMSTest)'==''">falseRunMSTest>
<RunStyleCop Condition="''=='$(RunStyleCop)'">trueRunStyleCop>
PropertyGroup>
<ItemGroup>
<AllConfigurations Include="Debug">
<Configuration>DebugConfiguration>
AllConfigurations>
ItemGroup>
<ItemGroup>
<MSTestProjects Include="$(SourceRoot)TestProject1\TestProject1.csproj">
MSTestProjects>
<MSTestProjects Include="$(SourceRoot)TestProject2\TestProject2.csproj">
MSTestProjects>
ItemGroup>
<Import Project="$(BuildRoot)Build.Common.UnitTest.targets"/>
This file is pretty simple. It just creates some properties, and items and then just imports the Build.Common.UnitTest.targets file to do all the heavy lifting. You will probably notice that there are some properties defined that don't make sense here, like CodeAnalysisTreatWarningsAsErrors, this is because this was taken from a build script which does some other tasks. You can ignore those. To see what properties/items are required for the MSTest just look at the MSTestValidateSettings target. Also in the previous code snippet I showed how you could inject the MSTest target into the build process but it is commented out since there is no real build process in this case.
One thing about this approach that may not be ideal is that it will execute the test cases in one assembly at a time and if there is a failure it will not go on to the other assemblies. In my case this is OK because this is for local builds, public builds are executing test cases by Team Build, but this can be looked at.
Questions? Comments?
MSBuild Batching Links
- MSBuild batching Part 1
- MSBuild Batching Part 2
- MSBuild Batching Part 3
- MSBuild RE: Enforcing the Build Agent in a Team Build
Sayed Ibrahim Hashim
Yes, this looks like the blog entry I have been looking for, and I have bought your book, as I am completely new to MSBuild, though I have used other build systems like Ant and, of course, make before.
When I attempt to plug your example into my current solution with its 4 C# projects (1 EXE, 3 DLLs), 1 C++/CLI project, and 2 test projects, I get the following message about the C++/CLI project:
5>c:\Windows\Microsoft.NET\Framework\v3.5\Microsoft.Common.targets(1236,9) : error : MSBuild cannot resolve the reference to the Visual C++ project '..\Shared\CliProj\CliProj.vcproj' when building a stand-alone MSBuild project. To correctly resolve this reference, please build the solution file containing these projects.
I have built the entire solution before attempting to run the MSTest example in this context, but it appears that MSBuild still needs context stored in the solution file.
Indeed the solution is already plugged into a larger build and continuous integration system that currently calls:
msbuild main_exe.sln /p:Configuration=$(CONFIGURATION);OutDir=$(OUTPUT_PATH) /nologo /verbosity:minimal
So, I am wondering how a MSTest.proj file might be integrated into the .sln file, and then invoked with MSBuild, rather than invoked stand-alone, as with the example runall.bat.
I'll keep muddling through this, but any more help you could provide would be much appreciated. Thanks for this example.
--Brendan
Comments are closed.