MSBuild: Executing MSTest Unit Tests

Ibrahim — Thu, 13 Aug 2009 04:11:52 GMT

I have seen a couple blog entries about executing MSTest unit tests from MSBuild. Most recently I saw the entry by Scott A. Lawrence. So I decided to share how I execute MSTest unit tests from MSBuild

I created a file named Build.Common.UnitTest.targets which contains all the behavior that will execute the test cases. This file can then be imported into whatever scripts that need to execute test cases. The entire file is shown below. We will discuss afterwards.

Build.Common.UnitTest.targets

<?xml version="1.0" encoding="utf-8"?>

<!-- =======================================================================

TESTING

======================================================================= -->

</PropertyGroup>

<_RequiredProperties Remove="@(_RequiredProperties)"/>

<_RequiredItems Remove="@(_RequiredItems)"/>

</ItemGroup>

<_RequiredProperties Include="BuildContribRoot">

<Value>$(BuildContribRoot)</Value>

</_RequiredProperties>

<_RequiredProperties Include="OutputRoot">

<Value>$(OutputRoot)</Value>

</_RequiredProperties>

<_RequiredItems Include="MSTestProjects">

<RequiredValue>@(MSTestProjects)</RequiredValue>

<RequiredFilePath>%(MSTestProjects.FullPath)</RequiredFilePath>

</_RequiredItems>

<_RequiredItems Include="AllConfigurations">

<RequiredValue>@(AllConfigurations)</RequiredValue>

</_RequiredItems>

<_RequiredItems Include="AllConfigurations.Configuration">

<RequiredValue>%(AllConfigurations.Configuration)</RequiredValue>

</_RequiredItems>

</ItemGroup>

<Error Condition="'%(_RequiredProperties.Value)'==''"

Text="Missing required property [%(_RequiredProperties.Identity)]"/>

<Error Condition="'%(_RequiredItems.RequiredValue)'==''"

Text="Missing required item value [%(_RequiredItems.Identity)]" />

<Error Condition="'%(_RequiredItems.RequiredFilePath)' != '' and !Exists('%(_RequiredItems.RequiredFilePath)')"

Text="Unable to find expeceted path [%(_RequiredItems.RequiredFilePath)] on item [%(_RequiredItems.Identity)]" />

</Target>

<UsingTask

TaskName="TestToolsTask"

AssemblyFile="$(BuildContribRoot)TestToolsTask-1.3\Microsoft.VisualStudio.QualityTools.MSBuildTasks.dll"/>

BuildMSTestProjects;

BeforeMSTest;

CoreMSTest;

AfterMSTest

$(MSTestDependsOn);

</MSTestDependsOn>

</PropertyGroup>

<_CurrentConfig>Debug</_CurrentConfig>

</PropertyGroup>

<_SearchPath>$(OutputRoot)$(_CurrentConfig)\%(MSTestProjects.Filename)\</_SearchPath>

<_TestContainers></_TestContainers>

</PropertyGroup>

<TestToolsTask

SearchPathRoot="%(MSTestProjects.RootDir)%(MSTestProjects.Directory)"

TestContainers="$(OutputRoot)$(_CurrentConfig)\%(MSTestProjects.Filename)\bin\%(MSTestProjects.Filename).dll"/>

</Target>

BeforeBuildMSTestProjects;

CoreBuildMSTestProjects;

AfterBuildMSTestProjects;

$(BuildMSTestProjectsDependsOn);

</BuildMSTestProjectsDependsOn>

</PropertyGroup>

<Target Name="CoreBuildMSTestProjects" Outputs="%(AllConfigurations.Configuration)"

DependsOnTargets="CleanMSTest">

<_CurrentConfig>%(AllConfigurations.Configuration)</_CurrentConfig>

</PropertyGroup>

<MSBuild Projects="%(MSTestProjects.Identity)"

Properties="Configuration=$(_CurrentConfig);

OutputPath=$(OutputRoot)$(_CurrentConfig)\%(MSTestProjects.Filename)\bin\;

BaseIntermediateOutputPath=$(OutputRoot)$(_CurrentConfig)\%(MSTestProjects.Filename)\obj\;

GenerateResourceNeverLockTypeAssemblies=true;

%(ProjectsToBuild.Properties);

$(AllProjectProperties);"

BuildInParallel="$(BuildInParallel)"

</MSBuild>

</Target>

<MSBuild Projects="@(MSTestProjects)"

Targets="Clean"

Properties="Configuration=$(_CurrentConfig);

OutputPath=$(OutputRoot)$(_CurrentConfig)\%(MSTestProjects.Filename)\bin\;

BaseIntermediateOutputPath=$(OutputRoot)$(_CurrentConfig)\%(MSTestProjects.Filename)\obj\;

GenerateResourceNeverLockTypeAssemblies=true;"

BuildInParallel="$(BuildInParallel)"

</Target>

</Project>

There are five important targets which are described below.

Name	Description
MSTestValidateSettings	This validates that the file was provided the needed data values to perform its task, to run the unit tests. For more info on this technique see my previous entry Elements of Reusable MSBuild Scripts: Validation.
MSTest	This is the target that you would execute to run the test cases. The target itself is empty but it sets up the chain of dependent targets.
CoreMSTest	This is the target which executes the test cases. This is preformed using the TestToolsTask.
BuildMSTestProjects	This target is responsible for building (i.e. compiling) the projects which contain the test cases. You don't have to call this it is called automagically.
CleanMSTest	This target will execute the Clean target for all the test projects defined.

If you take a look at the CoreBuildMSTestProjects target you can see that I am batching (Target batching to be specific) it on each defined configuration. This is achieved with the attribute Outputs="%(AllConfigurations.Configuration)". If you are not familiar with batching, see the links at the end of this post for more details, and you can always grab my book for even more detailed info J. Then inside that target I build each project by batching (Task batching) the MSBuild task on each project defined in the MSTestProjects item list.

Then inside the CoreMSTest target I execute the test cases. This target is batched for every value in the MSTestProjects item. As I'm writing this I have noticed that I've hard-coded the value for the configuration used in that target to be Debug with the statement

<_CurrentConfig>Debug</_CurrentConfig>

</PropertyGroup>

This shouldn't be hard coded, but passed in. I will leave it as is for now though. Then the TestToolsTask is invoked to execute the test cases.

Now that we have written the re-usable .targets file to execute the test cases we need to create a file which will "feed" it the necessary data values and let it do its magic. I created a sample solution, which you can download at the end of this post, which demonstrates its usage. The solution is named MSTestExample and you can see the files it contains in the screen shot below.

Here I've highlighted the two MSTest projects as well as a couple build files. I've already shown the contents of the Build.Common.UnitTest.targets file. Here is the contents of the Build.MSTestExample.proj file.

Build.MSTestExample.proj

<?xml version="1.0" encoding="utf-8"?>

<!--

Some properties you may be interested in setting:

Name Description

*****************************************************

RunMSTest true/false to run unit tests or not

-->

<Root Condition="'$(Root)'==''">$(MSBuildProjectDirectory)\..\</Root>

<BuildRoot Condition="'$(BuildRoot)'==''">$(Root)Build\</BuildRoot>

<BuildContribRoot Condition="'$(BuildContribRoot)'==''">$(BuildRoot)Contrib\</BuildContribRoot>

<BuildArtifactsRoot Condition="'$(BuildArtifactsRoot)'==''">$(BuildRoot)BuildAftifacts\</BuildArtifactsRoot>

<OutputRoot Condition="'$(OutputRoot)'==''">$(BuildArtifactsRoot)Output\</OutputRoot>

</PropertyGroup>

<RunMSTest Condition="'$(RunMSTest)'==''">false</RunMSTest>

</PropertyGroup>

<Configuration>Debug</Configuration>

</AllConfigurations>

</ItemGroup>

<!-- =======================================================================

MSTest

======================================================================= -->

</MSTestProjects>

</MSTestProjects>

</ItemGroup>

<!--<PropertyGroup Condition="'$(RunMSTest)'=='true'">

$(BuildDependsOn);

MSTest;

</BuildDependsOn>

CoreBuild;

$(BuildMSTestProjectsDependsOn)

</BuildMSTestProjectsDependsOn>

</PropertyGroup>-->

</Project>

This file is pretty simple. It just creates some properties, and items and then just imports the Build.Common.UnitTest.targets file to do all the heavy lifting. You will probably notice that there are some properties defined that don't make sense here, like CodeAnalysisTreatWarningsAsErrors, this is because this was taken from a build script which does some other tasks. You can ignore those. To see what properties/items are required for the MSTest just look at the MSTestValidateSettings target. Also in the previous code snippet I showed how you could inject the MSTest target into the build process but it is commented out since there is no real build process in this case.

One thing about this approach that may not be ideal is that it will execute the test cases in one assembly at a time and if there is a failure it will not go on to the other assemblies. In my case this is OK because this is for local builds, public builds are executing test cases by Team Build, but this can be looked at.

Questions? Comments?

MSTestExamples.zip

MSBuild Batching Links

Sayed Ibrahim Hashim

Sayed Ibrahim Hashimi - MSBuild, Web Deploy (MSDeploy), ASP.NET - batching

MSBuild: Executing MSTest Unit Tests

Build.Common.UnitTest.targets

Build.MSTestExample.proj

MSBuild Batching Links