- | rssFeed | My book on MSBuild and Team Build | Archives and Categories Thursday, December 29, 2011

Updating XML files with MSBuild

Today I just saw a question posted on StackOverflow asking how to update an XML file using MSBuild during a CI build executed from Team City.

There is not correct single answer, there are several different ways that you can update an XML file during a build. Most notably:

  1. Use SlowCheetah to transform the files for you
  2. Use the TransformXml task directly
  3. Use the built in (MSBuild 4.0) XmlPoke task
  4. Use a third party task library

#1 Use SlowCheetah to transform the files for you

Before you start reading too far into this post let me go over option #3 first because I think it’s the easiest approach and the most easily maintained. You can download my SlowCheetah XML Transforms Visual Studio add in. Once you do this for your projects you will see a new menu command to transform a file on build (for web projects on package/publish). If you want to get this working from a CI server, its pretty easy see my post at http://sedodream.com/2011/12/12/SlowCheetahXMLTransformsFromACIServer.aspx.

#2 Use the TransformXml task directly

If you want a technique where you have a “main” XML file and you want to be able to contain transformations to that file inside of a separate XML file then you can use the TransformXml task directly. For more info see my previous blog post at http://sedodream.com/2010/11/18/XDTWebconfigTransformsInNonwebProjects.aspx

#3 Use the built in XmlPoke task

Sometimes it doesn’t make sense to create an XML file with transformations for each XML file. For example if you have an XML file and you want to modify a single value but to create 10 different files the XML transformation approach doesn’t scale well. In this case it might be easier to use the XmlPoke task. Note this does require MSBuild 4.0.

Below are the contents of sample.xml (came from the SO question).

<Provisioning.Lib.Processing.XmlConfig instancetype="XmlConfig, Processing, Version=1.0.0.0, Culture=neutral">
  <item>
    <key>IsTestEnvironment</key>
    <value>True</value>
    <encrypted>False</encrypted>
  </item>
  <item>
    <key>HlrFtpPutDir</key>
    <value>C:\DevPath1</value>
    <encrypted>False</encrypted>
  </item>
  <item>
    <key>HlrFtpPutCopyDir</key>
    <value>C:\DevPath2</value>
    <encrypted>False</encrypted>
  </item>
</Provisioning.Lib.Processing.XmlConfig>

So in this case we want to update the values of the value element. So the first thing that we need to do is to come up with the correct XPath for all the elements which we want to update. In this case we can use the following XPath expressions for each value element.

I’m not going to go over what you need to do to figure out the correct XPath because that’s not the purpose of this post. There are a bunch of XPath related resources on the interwebs. In the resources section I have linked to the online XPath tester which I always use.

Now that we’ve got the required XPath expressions we need to construct our MSBuild elements to get everything updated. Here is the overall technique:

  1. Place all info for all XML updates into an item
  2. Use XmlPoke along with MSBuild batching to perform all the updates

For #2 if you are not that familiar with MSBuild batching then I would recommend buying my book or you can take a look at the resources I have online relating to batching (the link is below in resources section). Below you will find a simple MSBuild file that I created, UpdateXm01.proj.

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" DefaultTargets="UpdateXml" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <SourceXmlFile>$(MSBuildProjectDirectory)\sample.xml</SourceXmlFile>
    <DestXmlFiles>$(MSBuildProjectDirectory)\result.xml</DestXmlFiles>
  </PropertyGroup>

  <ItemGroup>
    <!-- Create an item which we can use to bundle all the transformations which are needed -->
    <XmlConfigUpdates Include="ConfigUpdates-SampleXml">
      <XPath>/Provisioning.Lib.Processing.XmlConfig/item[key='HlrFtpPutDir']/value</XPath>
      <NewValue>H:\ReleasePath1</NewValue>
    </XmlConfigUpdates>
    
    <XmlConfigUpdates Include="ConfigUpdates-SampleXml">
      <XPath>/Provisioning.Lib.Processing.XmlConfig/item[key='HlrFtpPutCopyDir']/value</XPath>
      <NewValue>H:\ReleasePath2</NewValue>
    </XmlConfigUpdates>
  </ItemGroup>
  
  <Target Name="UpdateXml">
    <Message Text="Updating XML file at $(DestXmlFiles)" />
    <Copy SourceFiles="$(SourceXmlFile)"
          DestinationFiles="$(DestXmlFiles)" />
    <!-- Now let's execute all the XML transformations -->
    <XmlPoke XmlInputPath="$(DestXmlFiles)"
             Query="%(XmlConfigUpdates.XPath)"
             Value="%(XmlConfigUpdates.NewValue)"/>
  </Target>
</Project>

The parts to pay close attention to is the XmlConfigUpdates item and the contents of the UpdateXml task itself. Regarding the XmlConfigUpdates, that name is arbitrary you can use whatever name you want, you can see that the Include value (which typically points to a file) is simply left at ConfigUpdates-SampleXml. The value for the Include attribute is not used here. I would place a unique value for the Include attribute for each file that you are updating. This just makes it easier for people to understand what that group of values is for, and you can use it later to batch updates. The XmlConfigUpdates item has these two metadata values:

Inside of the UpdateXml target you can see that we are using the XmlPoke task and passing the XPath as %(XmlConfigUpdate.XPath) and the value as %(XmlConfigUpdates.NewValue). Since we are using the %(…) syntax on an item this start MSBuild batching. Batching is where more than one operation is performed over a “batch” of values. In this case there are two unique batches (1 for each value in XmlConfigUpdates) so the XmlPoke task will be invoked two times. Batching can be confusing so make sure to read up on it if you are not familiar.

Now we can use msbuild.exe to start the process. The resulting XML file is:

<Provisioning.Lib.Processing.XmlConfig instancetype="XmlConfig, Processing, Version=1.0.0.0, Culture=neutral">
  <item>
    <key>IsTestEnvironment</key>
    <value>True</value>
    <encrypted>False</encrypted>
  </item>
  <item>
    <key>HlrFtpPutDir</key>
    <value>H:\ReleasePath1</value>
    <encrypted>False</encrypted>
  </item>
  <item>
    <key>HlrFtpPutCopyDir</key>
    <value>H:\ReleasePath2</value>
    <encrypted>False</encrypted>
  </item>
</Provisioning.Lib.Processing.XmlConfig>

So now we can see how easy it was to use the XmlPoke task. Let’s now take a look at how we can extend this example to manage updates to the same file for an additional environment.

How to manage updates to the same file for multiple different results

Since we’ve created an item which will keep all the needed XPath as well as the new values we have a bit more flexibility in managing multiple environments. In this scenario we have the same file that we want to write out, but we need to write out different values based on the target environment. Doing this is pretty easy. Take a look at the contents of UpdateXml02.proj below.

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" DefaultTargets="UpdateXml" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  
  <PropertyGroup>
    <SourceXmlFile>$(MSBuildProjectDirectory)\sample.xml</SourceXmlFile>
    <DestXmlFiles>$(MSBuildProjectDirectory)\result.xml</DestXmlFiles>
  </PropertyGroup>

  <PropertyGroup>
    <!-- We can set a default value for TargetEnvName -->
    <TargetEnvName>Env01</TargetEnvName>
  </PropertyGroup>
  
  <ItemGroup Condition=" '$(TargetEnvName)' == 'Env01' ">
    <!-- Create an item which we can use to bundle all the transformations which are needed -->
    <XmlConfigUpdates Include="ConfigUpdates">
      <XPath>/Provisioning.Lib.Processing.XmlConfig/item[key='HlrFtpPutDir']/value</XPath>
      <NewValue>H:\ReleasePath1</NewValue>
    </XmlConfigUpdates>
    
    <XmlConfigUpdates Include="ConfigUpdates">
      <XPath>/Provisioning.Lib.Processing.XmlConfig/item[key='HlrFtpPutCopyDir']/value</XPath>
      <NewValue>H:\ReleasePath2</NewValue>
    </XmlConfigUpdates>
  </ItemGroup>

  <ItemGroup Condition=" '$(TargetEnvName)' == 'Env02' ">
    <!-- Create an item which we can use to bundle all the transformations which are needed -->
    <XmlConfigUpdates Include="ConfigUpdates">
      <XPath>/Provisioning.Lib.Processing.XmlConfig/item[key='HlrFtpPutDir']/value</XPath>
      <NewValue>G:\SomeOtherPlace\ReleasePath1</NewValue>
    </XmlConfigUpdates>

    <XmlConfigUpdates Include="ConfigUpdates">
      <XPath>/Provisioning.Lib.Processing.XmlConfig/item[key='HlrFtpPutCopyDir']/value</XPath>
      <NewValue>G:\SomeOtherPlace\ReleasePath2</NewValue>
    </XmlConfigUpdates>
  </ItemGroup>
  
  <Target Name="UpdateXml">
    <Message Text="Updating XML file at $(DestXmlFiles)" />
    <Copy SourceFiles="$(SourceXmlFile)"
          DestinationFiles="$(DestXmlFiles)" />
    <!-- Now let's execute all the XML transformations -->
    <XmlPoke XmlInputPath="$(DestXmlFiles)"
             Query="%(XmlConfigUpdates.XPath)"
             Value="%(XmlConfigUpdates.NewValue)"/>
  </Target>
</Project>

The differences are pretty simple, I introduced a new property, TargetEnvName which lets us know what the target environment is. (note: I just made up that property name, use whatever name you like). Also you can see that there are two ItemGroup elements containing different XmlConfigUpdate items. Each ItemGroup has a condition based on the value of TargetEnvName so only one of the two ItemGroup values will be used. Now we have a single MSBuild file that has the values for both environments. When building just pass in the property TargetEnvName, for example msbuild .\UpdateXml02.proj /p:TargetEnvName=Env02. When I executed this the resulting file contains:

<Provisioning.Lib.Processing.XmlConfig instancetype="XmlConfig, Processing, Version=1.0.0.0, Culture=neutral">
  <item>
    <key>IsTestEnvironment</key>
    <value>True</value>
    <encrypted>False</encrypted>
  </item>
  <item>
    <key>HlrFtpPutDir</key>
    <value>G:\SomeOtherPlace\ReleasePath1</value>
    <encrypted>False</encrypted>
  </item>
  <item>
    <key>HlrFtpPutCopyDir</key>
    <value>G:\SomeOtherPlace\ReleasePath2</value>
    <encrypted>False</encrypted>
  </item>
</Provisioning.Lib.Processing.XmlConfig>

You can see that the file has been updated with different paths in the value element.

#4 Use a third party task library

If you are not using MSBuild 4 then you will need to use a third party task library like the MSBuild Extension Pack (link in resources).

Hope that helps.

Resources

Sayed Ibrahim Hashimi @SayedIHashimi

Thursday, December 29, 2011 3:38:56 AM (GMT Standard Time, UTC+00:00)  #     | 
Saturday, December 24, 2011

Package once publish anywhere

When I discuss the web package features in Visual Studio one item that always comes up is that people want to be able to create a single Web Deploy package and then publish that to may different environments. One reason why they have a difficult time in doing this is because when a web package is created the web.config transforms are executed before the package is created and then only the transformed web.config file makes it into the package. Also Web Deploy itself doesn’t have support for web.config transforms. Since I’ve heard this soo many times I decided to take matters into my own hands and address this for existing Visual Studio 2010 scenarios. Here is what I (note that this is not a Microsoft sponsored project, it’s a personal project) have done.

I have create a new Nuget package, PackageWeb, which you can use to address this. In this blog post I will focus on the user experience and not discuss too many details about the implementation, but I will have later blog posts to discuss how this was implemented. Since this is a Nuget package you have to have Nuget installed. If you don’t go to nuget.org and click the huge Install Nuget button. Here are the steps to get you started:

  1. Install the PackageWeb Nuget package
  2. Create a web deployment package
  3. Notice there is a Publish-Interactive.ps1 file in the package folder
  4. Execute the Publish-Interactive.ps1 file from the package folder

#1 Install the PacakgeWeb Nuget package

To install the package just right-click on the web project in the Solution Explorer and select Manage Nuget Packages. In the dialog that pops up just search for PackageWeb and click on the install button.

SNAGHTML6c282a

After you do this a few files will be added to your project and the package creation process will be extended to include those files. Stay tuned to this blog for more details on this part.

#2 Create a web deployment package

For most who are interested they will already know how to do this, but if not don’t worry its simple. Right-click on the project in solution explorer and pick Build Deployment Package.

#3 Notice there is a Publish-Interactive.ps1 file in the package folder

When the package is built (by default it is written to the obj\{Configuration}\Package\ folder. In that folder you will now see a Publish-Interactive.ps1 file.

image

#3 Execute the Publish-Interactive.ps1 file from the package folder

Now that we have everything setup let’s start publishing! First let’s take a look at the project which I am publishing, here is a view of the relevant portions of the project.

SNAGHTML7bc0ff

In this image you can see that I have 4 web.config transforms (web.debug.config,web.release.config,web.Prod.config and web.Test.config). I didn’t add any new build configurations to create the Test and Prod files. I just create them and added them to the project. Now open a PowerShell prompt and cd to the package folder. From there execute the command .\Publish-Interactive.ps1. Once you start it you will see that a file copy progress indicator starts. I’m extracting the package to a temp folder. The first thing that the script will do is to prompt you for the web.config transform to be executed.

image

In the image above you can see that it detected all 4 web.config transforms. At this point you just type in the name of the transform which you want to apply. I will pick Release here for this demo. After that some magic will happen and you will start to be prompted for some values. It will prompt you for Web Deploy endpoint info and for any parameters which exist in the package.

image

The first 5 prompts will always be the same, those are the Web Deploy endpoint values. Notice the whatif here, if you want to simulate a publish then for whatif pass in the value true. This will cause Web Deploy to go into a simulation mode, and it will tell you everything that it would do, but no changes will be made.

After the first 5 values you will be prompted for all the parameters which exist in the package. These usually have default values (you can see default values printed in cyan) so if you want to go with the defaults just hit the enter key.

After you enter all the parameter values you will see the message shown below.

image

You may not want to enter all these values everytime so to help with that when you invoke Publish-Interactive.ps1 it will write a file, PublishConfiguration.ps1.readme, which has all the parameter values (except password, you have to fill that one in).

image

If you want that file to be picked up, just remove the .readme from the extension and the next time you invoke Publish-Interactive.ps1 it will pick up the PublishConfiguration.ps1 file and not annoy you!

Currently I’m listing this project as Alpha quality, I will be able to promote it to beta/RTM only with your help. Please download it and try it out. Send the feedback to me either here or you can create an issue on the project’s issue page.

Note: As I stated previously this is not a Microsoft sponsored project, it is a personal project.

Sayed Ibrahim Hashimi @SayedIHashimi

Note: I work for Microsoft, but these are my own opinions

Saturday, December 24, 2011 5:04:44 AM (GMT Standard Time, UTC+00:00)  #     | 
Monday, December 12, 2011

SlowCheetah XML transforms from a CI server

Please read my updated post, the info below is no longer required for some scenarios SlowCheetah build server support updated

A few months ago I published a Visual Studio add-in, SlowCheetah – XML Transforms, which enables you to transform any XML file during a build in the same way that you can transform the web.config during publish/package for web projects. Since then I’ve received numerous requests to demonstrate how the transforms can be kicked in from a CI server. It’s actually pretty easy because all the logic from a transform perspective is contained inside of MSBuild tasks/targets. The Visual Studio add-in itself is only delivering the support for gestures/preview.

Before I dive into how to enable this for a CI server let me explain how the plugin works and then the CI integration will be come much clearer. When you load a project/solution in Visual Studio for the first time after you installed SlowCheetah a set of files will be written to %localappdata%\Microsoft\MSBuild\SlowCheetah\v1. Those file are:

 

Name

Description

Install-Manifest.xml An XML file which describes the files that are installed. This is used by the plugin itself, you should never have to do anything with this file.
SlowCheetah.Tasks.dll The assembly which contains the MSBuild task which transforms XML files.
SlowCheetah.Transforms.targets The MSBuild file which enables XML file transformation

 

The add-in adds the following menu command to any XML file for supported project types.

image

When you click on this for the first time the add-in needs to make some changes to your project file so you are prompted to accept that. After you do the following changes are applied to your project file.

  1. A new MSBuild property SlowCheetahTargets is added to the project and it points to the SlowCheetah.transforms.targets file in %localappdata%
  2. An import for that file is added

More specifically the elements added to your project file are.

<PropertyGroup>
  <SlowCheetahTargets Condition=" '$(SlowCheetahTargets)'=='' ">$(LOCALAPPDATA)\Microsoft\MSBuild\SlowCheetah\v1\SlowCheetah.Transforms.targets</SlowCheetahTargets>
</PropertyGroup>

<Import Project="$(SlowCheetahTargets)" Condition="Exists('$(SlowCheetahTargets)')" />

Note: If you look at your project file they will not be next to each other. The property will be towards the top of the project file and the import towards the bottom.

So now you might have an idea of why this won’t work on your CI server, the file SlowCheetah.Transforms.targets will not exist in the %localappdata% folder. There is an easy to way to fix this which doesn’t require any changes to your CI server. Check in the files and update your import. Here is what I did:

  1. Create a folder named SlowCheetah in the same folder as my solution
  2. Added SlowCheetah.Tasks.dll and SlowCheetah.Transforms.targets to that folder
  3. Create a solution folder named SlowCheetah in the solution
  4. Drag and drop the files from your SlowCheetah folder into the SolutionFolder
  5. Updated my project file to point to these files

Now your solution should look roughly like the following image.

image

Note: Steps #3/#4 are not strictly required

For #5 you’ll have to edit your project file, don’t worry its an easy modification. Just follow these steps:

  1. Right click on the project and select unload
  2. Right click on the unloaded project and select Edit
  3. Update the value for the SlowCheetahTargets import

In my case the folder that I placed the files into is 1 directory above the project itself. So the new value for the property is as follows.

<SlowCheetahTargets>$(MSBuildProjectDirectory)\..\SlowCheetah\SlowCheetah.Transforms.targets</SlowCheetahTargets>

After that I checked in all the files, kicked off a build and viola the files are transformed. Let me know if you need any more info.

Sayed Ibrahim Hashimi – @SayedIHashimi

Note: I work for Microsoft, but these are my own opinions

msbuild | SlowCheetah | Visual Studio | Visual Studio 2010 Monday, December 12, 2011 5:02:42 AM (GMT Standard Time, UTC+00:00)  #     |