- | rssFeed | My book on MSBuild and Team Build | Archives and Categories
Searched for : transformxml

Updating XML files with MSBuild

Today I just saw a question posted on StackOverflow asking how to update an XML file using MSBuild during a CI build executed from Team City.

There is not correct single answer, there are several different ways that you can update an XML file during a build. Most notably:

  1. Use SlowCheetah to transform the files for you
  2. Use the TransformXml task directly
  3. Use the built in (MSBuild 4.0) XmlPoke task
  4. Use a third party task library

#1 Use SlowCheetah to transform the files for you

Before you start reading too far into this post let me go over option #3 first because I think it’s the easiest approach and the most easily maintained. You can download my SlowCheetah XML Transforms Visual Studio add in. Once you do this for your projects you will see a new menu command to transform a file on build (for web projects on package/publish). If you want to get this working from a CI server, its pretty easy see my post at http://sedodream.com/2011/12/12/SlowCheetahXMLTransformsFromACIServer.aspx.

#2 Use the TransformXml task directly

If you want a technique where you have a “main” XML file and you want to be able to contain transformations to that file inside of a separate XML file then you can use the TransformXml task directly. For more info see my previous blog post at http://sedodream.com/2010/11/18/XDTWebconfigTransformsInNonwebProjects.aspx

#3 Use the built in XmlPoke task

Sometimes it doesn’t make sense to create an XML file with transformations for each XML file. For example if you have an XML file and you want to modify a single value but to create 10 different files the XML transformation approach doesn’t scale well. In this case it might be easier to use the XmlPoke task. Note this does require MSBuild 4.0.

Below are the contents of sample.xml (came from the SO question).

<Provisioning.Lib.Processing.XmlConfig instancetype="XmlConfig, Processing, Version=1.0.0.0, Culture=neutral">
  <item>
    <key>IsTestEnvironment</key>
    <value>True</value>
    <encrypted>False</encrypted>
  </item>
  <item>
    <key>HlrFtpPutDir</key>
    <value>C:\DevPath1</value>
    <encrypted>False</encrypted>
  </item>
  <item>
    <key>HlrFtpPutCopyDir</key>
    <value>C:\DevPath2</value>
    <encrypted>False</encrypted>
  </item>
</Provisioning.Lib.Processing.XmlConfig>

So in this case we want to update the values of the value element. So the first thing that we need to do is to come up with the correct XPath for all the elements which we want to update. In this case we can use the following XPath expressions for each value element.

  • /Provisioning.Lib.Processing.XmlConfig/item[key='HlrFtpPutDir']/value
  • /Provisioning.Lib.Processing.XmlConfig/item[key='HlrFtpPutCopyDir']/value

I’m not going to go over what you need to do to figure out the correct XPath because that’s not the purpose of this post. There are a bunch of XPath related resources on the interwebs. In the resources section I have linked to the online XPath tester which I always use.

Now that we’ve got the required XPath expressions we need to construct our MSBuild elements to get everything updated. Here is the overall technique:

  1. Place all info for all XML updates into an item
  2. Use XmlPoke along with MSBuild batching to perform all the updates

For #2 if you are not that familiar with MSBuild batching then I would recommend buying my book or you can take a look at the resources I have online relating to batching (the link is below in resources section). Below you will find a simple MSBuild file that I created, UpdateXm01.proj.

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" DefaultTargets="UpdateXml" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <SourceXmlFile>$(MSBuildProjectDirectory)\sample.xml</SourceXmlFile>
    <DestXmlFiles>$(MSBuildProjectDirectory)\result.xml</DestXmlFiles>
  </PropertyGroup>

  <ItemGroup>
    <!-- Create an item which we can use to bundle all the transformations which are needed -->
    <XmlConfigUpdates Include="ConfigUpdates-SampleXml">
      <XPath>/Provisioning.Lib.Processing.XmlConfig/item[key='HlrFtpPutDir']/value</XPath>
      <NewValue>H:\ReleasePath1</NewValue>
    </XmlConfigUpdates>
    
    <XmlConfigUpdates Include="ConfigUpdates-SampleXml">
      <XPath>/Provisioning.Lib.Processing.XmlConfig/item[key='HlrFtpPutCopyDir']/value</XPath>
      <NewValue>H:\ReleasePath2</NewValue>
    </XmlConfigUpdates>
  </ItemGroup>
  
  <Target Name="UpdateXml">
    <Message Text="Updating XML file at $(DestXmlFiles)" />
    <Copy SourceFiles="$(SourceXmlFile)"
          DestinationFiles="$(DestXmlFiles)" />
    <!-- Now let's execute all the XML transformations -->
    <XmlPoke XmlInputPath="$(DestXmlFiles)"
             Query="%(XmlConfigUpdates.XPath)"
             Value="%(XmlConfigUpdates.NewValue)"/>
  </Target>
</Project>

The parts to pay close attention to is the XmlConfigUpdates item and the contents of the UpdateXml task itself. Regarding the XmlConfigUpdates, that name is arbitrary you can use whatever name you want, you can see that the Include value (which typically points to a file) is simply left at ConfigUpdates-SampleXml. The value for the Include attribute is not used here. I would place a unique value for the Include attribute for each file that you are updating. This just makes it easier for people to understand what that group of values is for, and you can use it later to batch updates. The XmlConfigUpdates item has these two metadata values:

  • XPath
    • This contains the XPath required to select the element which is going to be updated
  • NewValue
    • This contains the new value for the element which is going to be updated

Inside of the UpdateXml target you can see that we are using the XmlPoke task and passing the XPath as %(XmlConfigUpdate.XPath) and the value as %(XmlConfigUpdates.NewValue). Since we are using the %(…) syntax on an item this start MSBuild batching. Batching is where more than one operation is performed over a “batch” of values. In this case there are two unique batches (1 for each value in XmlConfigUpdates) so the XmlPoke task will be invoked two times. Batching can be confusing so make sure to read up on it if you are not familiar.

Now we can use msbuild.exe to start the process. The resulting XML file is:

<Provisioning.Lib.Processing.XmlConfig instancetype="XmlConfig, Processing, Version=1.0.0.0, Culture=neutral">
  <item>
    <key>IsTestEnvironment</key>
    <value>True</value>
    <encrypted>False</encrypted>
  </item>
  <item>
    <key>HlrFtpPutDir</key>
    <value>H:\ReleasePath1</value>
    <encrypted>False</encrypted>
  </item>
  <item>
    <key>HlrFtpPutCopyDir</key>
    <value>H:\ReleasePath2</value>
    <encrypted>False</encrypted>
  </item>
</Provisioning.Lib.Processing.XmlConfig>

So now we can see how easy it was to use the XmlPoke task. Let’s now take a look at how we can extend this example to manage updates to the same file for an additional environment.

How to manage updates to the same file for multiple different results

Since we’ve created an item which will keep all the needed XPath as well as the new values we have a bit more flexibility in managing multiple environments. In this scenario we have the same file that we want to write out, but we need to write out different values based on the target environment. Doing this is pretty easy. Take a look at the contents of UpdateXml02.proj below.

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" DefaultTargets="UpdateXml" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  
  <PropertyGroup>
    <SourceXmlFile>$(MSBuildProjectDirectory)\sample.xml</SourceXmlFile>
    <DestXmlFiles>$(MSBuildProjectDirectory)\result.xml</DestXmlFiles>
  </PropertyGroup>

  <PropertyGroup>
    <!-- We can set a default value for TargetEnvName -->
    <TargetEnvName>Env01</TargetEnvName>
  </PropertyGroup>
  
  <ItemGroup Condition=" '$(TargetEnvName)' == 'Env01' ">
    <!-- Create an item which we can use to bundle all the transformations which are needed -->
    <XmlConfigUpdates Include="ConfigUpdates">
      <XPath>/Provisioning.Lib.Processing.XmlConfig/item[key='HlrFtpPutDir']/value</XPath>
      <NewValue>H:\ReleasePath1</NewValue>
    </XmlConfigUpdates>
    
    <XmlConfigUpdates Include="ConfigUpdates">
      <XPath>/Provisioning.Lib.Processing.XmlConfig/item[key='HlrFtpPutCopyDir']/value</XPath>
      <NewValue>H:\ReleasePath2</NewValue>
    </XmlConfigUpdates>
  </ItemGroup>

  <ItemGroup Condition=" '$(TargetEnvName)' == 'Env02' ">
    <!-- Create an item which we can use to bundle all the transformations which are needed -->
    <XmlConfigUpdates Include="ConfigUpdates">
      <XPath>/Provisioning.Lib.Processing.XmlConfig/item[key='HlrFtpPutDir']/value</XPath>
      <NewValue>G:\SomeOtherPlace\ReleasePath1</NewValue>
    </XmlConfigUpdates>

    <XmlConfigUpdates Include="ConfigUpdates">
      <XPath>/Provisioning.Lib.Processing.XmlConfig/item[key='HlrFtpPutCopyDir']/value</XPath>
      <NewValue>G:\SomeOtherPlace\ReleasePath2</NewValue>
    </XmlConfigUpdates>
  </ItemGroup>
  
  <Target Name="UpdateXml">
    <Message Text="Updating XML file at $(DestXmlFiles)" />
    <Copy SourceFiles="$(SourceXmlFile)"
          DestinationFiles="$(DestXmlFiles)" />
    <!-- Now let's execute all the XML transformations -->
    <XmlPoke XmlInputPath="$(DestXmlFiles)"
             Query="%(XmlConfigUpdates.XPath)"
             Value="%(XmlConfigUpdates.NewValue)"/>
  </Target>
</Project>

The differences are pretty simple, I introduced a new property, TargetEnvName which lets us know what the target environment is. (note: I just made up that property name, use whatever name you like). Also you can see that there are two ItemGroup elements containing different XmlConfigUpdate items. Each ItemGroup has a condition based on the value of TargetEnvName so only one of the two ItemGroup values will be used. Now we have a single MSBuild file that has the values for both environments. When building just pass in the property TargetEnvName, for example msbuild .\UpdateXml02.proj /p:TargetEnvName=Env02. When I executed this the resulting file contains:

<Provisioning.Lib.Processing.XmlConfig instancetype="XmlConfig, Processing, Version=1.0.0.0, Culture=neutral">
  <item>
    <key>IsTestEnvironment</key>
    <value>True</value>
    <encrypted>False</encrypted>
  </item>
  <item>
    <key>HlrFtpPutDir</key>
    <value>G:\SomeOtherPlace\ReleasePath1</value>
    <encrypted>False</encrypted>
  </item>
  <item>
    <key>HlrFtpPutCopyDir</key>
    <value>G:\SomeOtherPlace\ReleasePath2</value>
    <encrypted>False</encrypted>
  </item>
</Provisioning.Lib.Processing.XmlConfig>

You can see that the file has been updated with different paths in the value element.

#4 Use a third party task library

If you are not using MSBuild 4 then you will need to use a third party task library like the MSBuild Extension Pack (link in resources).

Hope that helps.

Resources

Sayed Ibrahim Hashimi @SayedIHashimi

Thursday, December 29, 2011 3:38:56 AM (GMT Standard Time, UTC+00:00)  #     | 

XDT (web.config) Transforms in non-web projects

One of the really cool features that we shipped for Visual Studio 2010 was web.config (XDT) transformations. Because the transformations are so simple and straightforward one of the first questions that someone asks after using it is “how can I use this in my other projects?” Unfortunately this feature is only built into the Web Application Projects (WAP). But it is very easy to reuse this because we just rely on an MSBuild task to do the heavy lifting for us. I received an email from that basically went like this

“Hi, I would like to use XDT transformations on my WPF project for both the app.config file as well as my unity.xml file. How can I do this?”

So one answer is to modify your project file to use the TransformXml task as I have blogged previously about (link below). But I thought that since this was such a common problem that I should go ahead and create a .targets file which would solve the above problem and could be re-used by anyone.

Let me clarify the scenario a bit before we dive into the details of the solution. I have create a sample Wpf project, named Wpf01. Inside of that project I have created these files:

  • app.confog
    • app.debug.config
    • app.release.config
  • Sample01.xml
    • Sample01.debug.xml
    • Sample01.release.xml
  • Sub\Sub\Sub01.xml
    • Sub\Sub\Sub01.debug.xml
    • Sub\Sub\Sub01.release.xml

Take a look at the image below (note: I manually edited the project file to make the file nest under each other, I will explain that shortly)

image

The files with .debug/.release are transform files. When I build I expect the following to happen:

  1. Transform app.config with app.{Configuration}.config and write file to output folder with the correct name i.e. Wpf01.exe.config instead of just app.config
  2. Transform Sample01.xml with Sample01.{Configuration}.config and write it to output folder with the name Sample01.config
  3. Transform Sub\Sub\Sub01.xml with Sub\Sub\Sub01.{Configuration}.config and write it to the output folder with the name Sub\Sub\Sub01.xml
  4. None of my source files should change

Usage

Before I get into the solution let me explain how to use the solution first because if you are not interested in the MSBuild details you can skip over that Smile

  1. You must have installed Web projects with Visual Studio on the machine (it contains the TransformXmll task).
  2. Create the folder %ProgramFiles (x86)%\MSBuild\Custom. If you want to share this across team members then see my note at the end of this blog.
  3. Download TransformFiles.targets (link below) and place the file into the folder %ProgramFiles (x86)%\MSBuild\Custom.
  4. Edit your project file (right click on the project Unload Project, right click again and pick edit)
  5. At the end of the project file place the element <Import Project="$(MSBuildExtensionsPath)\Custom\TransformFiles.targets" /> immediately above the closing </Project> tag
  6. For files that you want transformed a metadata value of TransformOnBuild to true. See below on what this means.
  7. Build and take a look at files in your output directory

For #5 lets examine the sample that I created. In this sample I had an app.config file. When I first created the project the entry in the project file for app.config looked like the following.

<None Include="app.config" />

So what you need to do is to add a new metadata value as described above for that. So it will turn into the following.

<None Include="app.config">
  <TransformOnBuild>true</TransformOnBuild>
</None>

The transform targets will look for items that have this value declared on them and then during build it will transform them, if the transform file exists in the same folder as the file itself. You will need to add TransfromOnBuild to all the files that you want to transform. So in my case I added it to app.config, Sample01.xml and Sub01.xml. Note you should not add this to the transform files themselves because you will just waste your own time. After you do this you should perform a build then take a look at the output directory for your transformed files. The app.config should write out the the correct file and the others as expected.

Nest transforms under the source file

You might have noticed that in the image above that the transform files are nested under the files themselves. To do this you need to add the DependentUpon metadata value to the child items. For instance for app.config the child items look like the following.

<None Include="app.debug.config">
  <DependentUpon>app.config</DependentUpon>
</None>
<None Include="app.release.config">
  <DependentUpon>app.config</DependentUpon>
</None>

Implementation

If you are wondering how this works then this is the section for you. TransformFile.targets has 2 targets; DiscoverFilesToTransform and TransformAllFiles. DiscoverFilesToTransform looks through a set of items (None, Content, and Resource). Inside of DiscoverFilesToTransform I look for values with the %(TransformOnBuild)==true. After all of those are collected I identify if there is an app.config file being transformed and if so it is placed into a specific item list and all others go into another item list.

Inside of TransformAllFiles the TransformXml task is used to transform all of the files. This target injects itself into the build process by having the attribute AfterTargets="Build;_CopyAppConfigFile". So whenever the Build or _CopyAppConfigFile targets are called the TransformAllFiles target will execute.

Here if the full code for this file.

<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <UsingTask TaskName="TransformXml"
         AssemblyFile="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v10.0\Web\Microsoft.Web.Publishing.Tasks.dll"/>
  
  <ItemDefinitionGroup>
    <!-- Set the default value to false here -->
    <None>
      <TransformOnBuild>false</TransformOnBuild>
    </None>    
    <Content>
      <TransformOnBuild>false</TransformOnBuild>
    </Content>    
    <Resource>
      <TransformOnBuild>false</TransformOnBuild>
    </Resource>
    <EmbeddedResource>
      <TransformOnBuild>false</TransformOnBuild>
    </EmbeddedResource>
    
    <_FilesToTransform>
      <IsAppConfig>false</IsAppConfig>
    </_FilesToTransform>
  </ItemDefinitionGroup>

  <PropertyGroup>
    <TransformAllFilesDependsOn>
      DiscoverFilesToTransform;
    </TransformAllFilesDependsOn>
  </PropertyGroup>
  <Target Name="TransformAllFiles" DependsOnTargets="$(TransformAllFilesDependsOn)" AfterTargets="Build;_CopyAppConfigFile">
    <!-- Now we have the item list _FilesToTransformNotAppConfig and _AppConfigToTransform item lists -->
    <!-- Transform the app.config file -->    
    <ItemGroup>
      <_AppConfigTarget Include="@(AppConfigWithTargetPath->'$(OutDir)%(TargetPath)')" />
    </ItemGroup>
    
    <PropertyGroup>
      <_AppConfigDest>@(_AppConfigTarget->'%(FullPath)')</_AppConfigDest>
    </PropertyGroup>

    <MakeDir Directories="@(_FilesToTransformNotAppConfig->'$(OutDir)%(RelativeDir)')"
             Condition="Exists('%(RelativeDir)%(Filename).$(Configuration)%(Extension)')"/>
    
    <TransformXml Source="@(_AppConfigToTransform->'%(FullPath)')"
                  Transform="%(RelativeDir)%(Filename).$(Configuration)%(Extension)"
                  Destination="$(_AppConfigDest)"
                  Condition=" Exists('%(RelativeDir)%(Filename).$(Configuration)%(Extension)') " />

    
    <TransformXml Source="@(_FilesToTransformNotAppConfig->'%(FullPath)')"
                  Transform="%(RelativeDir)%(Filename).$(Configuration)%(Extension)"
                  Destination="@(_FilesToTransformNotAppConfig->'$(OutDir)%(RelativeDir)%(Filename)%(Extension)')"
                  Condition=" Exists('%(RelativeDir)%(Filename).$(Configuration)%(Extension)') " />
  </Target>
  
  <Target Name="DiscoverFilesToTransform">
    <!-- 
    This will look through items list: None & Content for those
    with Metadata <TransformOnBuild>True</TransformOnBuild>
    -->
    <ItemGroup>
      <_FilesToTransform Include="@(None);@(Content);@(Resource);@(EmbeddedResource)"
                         Condition=" '%(TransformOnBuild)' == 'true' "/>
    </ItemGroup>    

    <PropertyGroup>
      <_AppConfigFullPath>@(AppConfigWithTargetPath->'%(RootDir)%(Directory)%(Filename)%(Extension)')</_AppConfigFullPath>
    </PropertyGroup>

    <!-- Now look to see if any of these are the app.config file -->
    <ItemGroup>
      <_FilesToTransform Condition=" '%(FullPath)'=='$(_AppConfigFullPath)' ">
        <IsAppConfig>true</IsAppConfig>
      </_FilesToTransform>
    </ItemGroup>
          
    <ItemGroup>
      <_FilesToTransformNotAppConfig Include="@(_FilesToTransform)"
                                     Condition=" '%(IsAppConfig)'!='true'"/>
      
      <_AppConfigToTransform  Include="@(_FilesToTransform)"
                              Condition=" '%(IsAppConfig)'=='true'"/>
    </ItemGroup>
  </Target>
</Project>

Gaps

With most things found on blogs there are some gaps Those are described here.

Clean build => It’s a best practice to delete files upon clean, but in this case I am not. This would be pretty easy to add, if you are interested let us know and I will update the sample.

Incremental build => The transforms will run every time you build even if the outputs are up to date, if this is an issue for you let us know and I will update the sample.

Sharing with team members

If you want to share with team members instead of placing this into %ProgramFiles (x86)% just place it into a folder in version control then change the <Import statement to point to that file instead of using MSBuildExtensionPath.

 

If you end up using this please let us know what is your experience with it.

Resources

Sayed Ibrahim Hashimi @sayedihashimi

msbuild | MSBuild 4.0 | MSDeploy | Web Publishing Pipeline Thursday, November 18, 2010 5:41:09 AM (GMT Standard Time, UTC+00:00)  #     | 

ASP.NET Web Application: Publish/Package Tokenizing Parameters

Today I just saw a question posted on stackoverflow.com asking Why are some Web.config transforms tokenised into SetParameters.xml and others are not? Let me give some background on this topic for those who are not aware of what the question is.

With Visual Studio 2010 when you package your application using the Build Deployment Package context menu option, see image below.

image

When build the package by default the package will be created in obj\{Configuration}\Package\{ProjectName}.zip where {Configuration} is the current build configuration, and {ProjectName} is the name of the project. So in this case I since I’m building with Debug and the project name is MvcApplication1 the package will be placed at obj\Debug\Package\MvcApplication1.zip. If you take this package and then import into IIS 7 with the “Import Application” option shown below. Note: The machine must have the Web Deployment Tool (aka MSDeploy) installed.

image

Once you click on Import Application then browse out to the package you will be shown a screen which prompts your for parameters. Its shown below.

SNAGHTML2d2664

On this screen you can see that we are prompting for a couple parameter values here. One is an IIS setting, Application Path, and the other is a connection string which will be placed inside the web.config file. If your Web Application Project (WAP)  had 5 different connection strings then they would automatically show up here on this page. Since connection strings are replaced so often we create parameters for all connection strings by default. You can define new parameters on your own, quite easily actually, but that is the topic for another blog post.

Now back to the question. He is asking why do we “tokenize” the connection strings in web.config. To clarify take a look at my web.config file below.

<configuration>
  <appSettings>
    <add key="setting01" value="value01"/>
  </appSettings>
  
  <connectionStrings>
    <add name="ApplicationServices"
         connectionString="data source=.\SQLEXPRESS;Integrated Security=SSPI;AttachDBFilename=|DataDirectory|aspnetdb.mdf;User Instance=true"
         providerName="System.Data.SqlClient" />
  </connectionStrings>
  
</configuration>

After I perform a package this will get changed. Take a look @ the web.config file which resides in the package (you can get to the file at obj\{CofigurationName}\Package\PackageTmp\web.config). You will see what is shown below.

<configuration>
  <appSettings>
    <add key="setting01" value="value01"/>
  </appSettings>
  <connectionStrings>
    <add name="ApplicationServices"
         connectionString="$(ReplacableToken_ApplicationServices-Web.config Connection String_0)"
         providerName="System.Data.SqlClient" />
  </connectionStrings>

</configuration>

So his question is why is the connection string replaced with $(ReplacableToken_ApplicationServices-Web.config Connection String_0) and nothing else is? We do this because we do not want you to accidently copy your web to a location and have it executing SQL statements against a SQL server which you did not intend. The idea is that you will create a package that you can deploy to many different environments. So the value that was in your web.config (or web.debug.config/web.release.config if you are using a web.config transformation) will not be placed inside the web.config in the package. Instead those values will be used as defaults in the package itself. We also create a SetParameters.xml file for you so that you can tweak the values. For my app see the MvcApplication1.SetParameters.xml file below.

<?xml version="1.0" encoding="utf-8"?>
<parameters>
  <setParameter name="IIS Web Application Name" 
                value="Default Web Site/MvcApplication1_deploy" />
  <setParameter name="ApplicationServices-Web.config Connection String" 
                value="data source=.\SQLEXPRESS;Integrated Security=SSPI;AttachDBFilename=|DataDirectory|aspnetdb.mdf;User Instance=true" />
</parameters>

The idea is that you can deploy your package in 2 ways. Through the IIS Manager which will prompt you for the parameters or you can deploy using msdeploy.exe with the –setParamFile switch to specify the path to the SetParameters.xml file. In this case I could create a QA01.SetParameters.xml file along with a QA02.SetParameters.xml file to deploy my web to my two QA servers. How do we do this?

How connection strings are tokenized

You might be wondering how the connection strings are tokenized to begin with. With Visual Studio 2010 we released web.config transformations, which all you to write terse web.config transformations inside of files like web.debug.config/web.release.config. When you package/publish your web these transform files are used to transform your web.config based on what you expressed in the appropriate transform file. We have an MSBuild task TransformXml which performs the transformation. We use that same task to tokenize the connection strings. If you are interested in the details take a look at %ProgramFiles32%\MSBuild\Microsoft\VisualStudio\v10.0\Web\Microsoft.Web.Publishing.targets in the AutoParameterizationWebConfigConnectionStringsCore target.

Now what if you do not want the connection string tokenized?

Prevent tokenizing connection strings

If you want to prevent your web.config connection strings from being tokenized it’s pretty easy. All we need to do is the add a property to the build/package/publish process. We can do that in 2 ways. Edit the project file itself or create a file with the name {ProjectName}.wpp.targets where {ProjectName} is the name of your project. The second approach is easier so I use that. In my case it would be MvcApplication1.wpp.targets. The contents of the file are shown below.

<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

  <PropertyGroup>
    <AutoParameterizationWebConfigConnectionStrings>false</AutoParameterizationWebConfigConnectionStrings>
  </PropertyGroup>
  
</Project>

Note: You may need to reload the project in Visual Studio for this to take effect.

Inside of this file I have declared the property, AutoParameterizationWebConfigConnectionStrings, to be false. This is telling the Web Publishing Pipeline (WPP) that it should not replace replace the connection strings with tokens, instead leave them as they are.

Questions/Comments???

Other Resources

asp.net | Deployment | msbuild | MSBuild 4.0 | MSDeploy Thursday, November 11, 2010 5:41:09 AM (GMT Standard Time, UTC+00:00)  #     | 

ASP.NET Web Projects: web.debug.config & web.release.config

Warning: What you see below feels hacky to me, but if you find it useful then use it

I have heard a lot of questions and confusion regarding web.debug.config and web.release.config. For example here is just one question on StackOverflow. The question states:

Hello, I want to use the web.config transformation that works fine for publish also for debugging.

When i publish a web app, visual studio automatically transforms the web.config based on my 
current build configuration. How can i tell visual studio 
to do the same when i start debugging. On debug start it simply 
uses the default web.config without transformation.

Any idea?

First let me explain, as I did to that question, the purpose of the files: web.config/web.debug.config/web.release.config.

web.config

This is the config file which developers should use locally. Ideally you should get this to be standardized. For instance you could use localhost for DB strings, and what not. You should strive for this to work on dev machines without changes.

web.debug.config

This is the transform that is applied when you publish your application to the development staging environment. This would make changes to the web.config which are required for the target environment.

web.release.config

This is the transform that is applied when you publish your application to the "production" environment. Obviously you'll have to be careful with passwords depending on your application/team.

The problem with transforming the web.config that you are currently running is that a transform can perform destructive actions to the web.config. For example it may delete a attributes, delete elements, etc.

Resolution

Ok, with that out the way not let’s see how we can enable what the question asker wants to do. To recap, when he builds on a particular configuration he wants a specific transform to be applied to web.config. So obviously you do not want to maintain a web.config file, because it is going to be overwritten. So what we need to do is to create a new file web.template.config, which is just a copy of web.config. Then just delete web.config by using Windows Explorer (don’t delete using Visual Studio because we do not want to delete it from the project). Note: If you are using a source control provider which is integrated into Visual Studio then you probably want to delete web.config from source control. Also with this we do not want to use web.debug.config or web.release.config because these already have a well defined role in the Web Publishing Pipeline so we do not want to disturb that. So instead we will create two new files, in the same folder as the project and web.template.config, web.dev.debug.config and web.dev.release.config. The ideas is that these will be the transforms applied when you debug, or run, your application from Visual Studio. Now we need to hook into the build/package/publish process to get this all wired up. With Web Application Projects (WAP) there is an extensibility point that you can create a project file in the same folder with the name {ProjectName}.wpp.targets where {ProjectName} is the name of the project. If this file is on disk in the same folder as the WAP then it will automatically be imported into the project file. So I have created this file. And I have placed the following content:

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

  <!-- Make sure web.config will be there even for package/publish -->
  <Target Name="CopyWebTemplateConfig" BeforeTargets="Build">
    <Copy SourceFiles="web.template.config"
          DestinationFiles="web.config"/>
  </Target>
  
  <PropertyGroup>
    <PrepareForRunDependsOn>
      $(PrepareForRunDependsOn);
      UpdateWebConfigBeforeRun;
    </PrepareForRunDependsOn>
  </PropertyGroup>

  <!-- This target will run right before you run your app in Visual Studio -->
  <Target Name="UpdateWebConfigBeforeRun">
    <Message Text="Configuration: $(Configuration): web.dev.$(Configuration).config"/>
    <TransformXml Source="web.template.config"
              Transform="web.dev.$(Configuration).config"
              Destination="web.config" />
  </Target>

  <!-- Exclude the config template files from the created package -->
  <Target Name="ExcludeCustomConfigTransformFiles" BeforeTargets="ExcludeFilesFromPackage">
    <ItemGroup>
      <ExcludeFromPackageFiles Include="web.template.config;web.dev.*.config"/>
    </ItemGroup>
    <Message Text="ExcludeFromPackageFiles: @(ExcludeFromPackageFiles)" Importance="high"/>
  </Target>
</Project>

Let me explain this a bit. I have created the CopyWebTemplateConfig target which will always copy web.template.config to web.config on build, even if you are not debugging your application in Visual Studio. This is needed because we still need to support the package/publish process of Visual Studio. Then I extended the property PrepareForRunDependsOn to include the UpdateWebConfigBeforeRun target. This property is used to identify the list of targets which needs to be executed before any managed project is run from Visual Studio. In this target I am using the TransformXml task to transform web.template.config, using the correct web.dev.***.config file. After that your app starts up using the correct web.config based on  your build configuration.

After that I have another target ExcludeCustomConfigTransformsFiles, which I inject into the package/publish process via the attribute BeforeTargets=”ExcludeFilesFromPackage”. This is needed because we do not want these files to be included when the application is packaged or published.

So that is really all there is to it. To explain the package/publish process a bit more for this scenario. When you package/publish web.debug.config or web.release.config, depending on build configuration, will still be used. But ultimately the file that it is transforming is web.template.config, so you may have to adjust depending on what you have in that file. Questions/Comments?

Sayed Ibrahim Hashimi - @sayedihashimi

msbuild | MSDeploy | Web Publishing Pipeline Thursday, October 21, 2010 7:13:26 AM (GMT Daylight Time, UTC+01:00)  #     | 

Extending XML (web.config) Config transformation

Last week on StackOverflow I answered a question, Make web.config transformations working locally and in a response to my answer the question asker asked me if I would be able to a question he posed earlier Advanced tasks using web.config transformation. Evidently he is really interested in config transformations! I don’t blame him, I’m really into them as well.

In his question he asks (summarizing) can we replace portions of attribute values instead of this entire attribute? So for instance you have the following in your web.config. Below is two sets of appSettings one from Dev and the other from Prod (taken from the original question).

<!-- DEV ENTRY -->
<appSettings>
 <add key="serviceName1_WebsService_Url" value="http://wsServiceName1.dev.domain.com/v1.2.3.4/entryPoint.asmx" />
 <add key="serviceName2_WebsService_Url" value="http://ma1-lab.lab1.domain.com/v1.2.3.4/entryPoint.asmx" />
</appSettings>

<!-- PROD ENTRY -->
<appSettings>
 <add key="serviceName1_WebsService_Url" value="http://wsServiceName1.prod.domain.com/v1.2.3.4/entryPoint.asmx" />
 <add key="serviceName2_WebsService_Url" value="http://ws.ServiceName2.domain.com/v1.2.3.4/entryPoint.asmx" />
</appSettings>

In the above we just want to replace dev with prod and ma1-lab.lab1.domain with ws.ServiceName2.domain. For those wondering currently we have the following transformations out of the box.

  • Replace – Replaces the entire element
  • Remove – Removes the entire element
  • RemoveAll – Removes all matching elements
  • Insert – Inserts an element
  • SetAttributes – Sets the value of the specified attributes
  • RemoveAttributes – Removes attributes
  • InsertAfter – Inserts an element after another
  • InsertBefore – Inserts an element before another

At the end of this article I’ve linked to another blog which has more info about these transformations. So it sounds like SetAttributes is almost what we want, but not quite what there. A little known fact is that you can create your own config transformations and use those. In fact all of the out of the box transformations follow the same patterns that custom transformations would. To solve this issue we need to create our own config transformation, AttributeRegexReplace. This transformation will take an attribute value and do a regular expression replace on its value. In order to create a new transformation you first reference the Microsoft.Web.Publishing.Tasks.dll which can be found in the %Program Files (x86)%MSBuild\Microsoft\VisualStudio\v10.0\Web folder. If you are working with a team it is best if you copy that assembly, place it in a shared folder in source control, and make the reference from that location. After you create the reference to that assembly you will need to create a class which extends the Transform class. The class diagram for this abstract class is shown below.

image

The only thing that you will need to implement is the Apply method. You don’t even need to fully understand all of the properties and methods just the portions that you are interested in. Here we will not cover all the details of this class, or other related classes which exist, but there will be future posts which will shed more light on this area.

In the sample class library that I created, I called the project CustomTransformType. Inside of that project I created the class AttributeRegexReplace. The entire contents of that class are shown below, we will go over the details after that.

namespace CustomTransformType
{
    using System;
    using System.Text.RegularExpressions;
    using System.Xml;
    using Microsoft.Web.Publishing.Tasks;

    public class AttributeRegexReplace : Transform
    {
        private string pattern;
        private string replacement;
        private string attributeName;

        protected string AttributeName
        {
            get
            {
                if (this.attributeName == null)
                {
                    this.attributeName = this.GetArgumentValue("Attribute");
                }
                return this.attributeName;
            }
        }
        protected string Pattern
        {
            get
            {
                if (this.pattern == null)
                {
                    this.pattern = this.GetArgumentValue("Pattern");
                }

                return pattern;
            }
        }

        protected string Replacement
        {
            get
            {
                if (this.replacement == null)
                {
                    this.replacement = this.GetArgumentValue("Replacement");
                }

                return replacement;
            }
        }

        protected string GetArgumentValue(string name)
        {
            // this extracts a value from the arguments provided
            if (string.IsNullOrWhiteSpace(name)) 
            { throw new ArgumentNullException("name"); }

            string result = null;
            if (this.Arguments != null && this.Arguments.Count > 0)
            {
                foreach (string arg in this.Arguments)
                {
                    if (!string.IsNullOrWhiteSpace(arg))
                    {
                        string trimmedArg = arg.Trim();
                        if (trimmedArg.ToUpperInvariant().StartsWith(name.ToUpperInvariant()))
                        {
                            int start = arg.IndexOf('\'');
                            int last = arg.LastIndexOf('\'');
                            if (start <= 0 || last <= 0 || last <= 0)
                            {
                                throw new ArgumentException("Expected two ['] characters");
                            }

                            string value = trimmedArg.Substring(start, last - start);
                            if (value != null)
                            {
                                // remove any leading or trailing '
                                value = value.Trim().TrimStart('\'').TrimStart('\'');
                            }
                            result = value;
                        }
                    }
                }
            }
            return result;
        }

        protected override void Apply()
        {
            foreach (XmlAttribute att in this.TargetNode.Attributes)
            {
                if (string.Compare(att.Name, this.AttributeName, StringComparison.InvariantCultureIgnoreCase) == 0)
                {
                    // get current value, perform the Regex
                    att.Value = Regex.Replace(att.Value, this.Pattern, this.Replacement);
                }
            }
        }
    }
}

In this class we have 3 properties; Pattern, Replacement, and AttributeName. All of these values will be provided via an argument in the config transformation. For example take a look at the element below which contains a transform attribute may look like the following.

<add key="two" value="two-replaced" 
         xdt:Transform="AttributeRegexReplace(Attribute='value', Pattern='here',Replacement='REPLACED')" 
         xdt:Locator="Match(key)"/>

In this example I declare that I am using AttributeRegexReplace and then specify the values for the attributes within the (). In the class above I have a method, GetArgumentValue, which is used to parse values from that argument string. When your transform is invoked the string inside of () is passed in as the ArgumentString value. If you are using a , as the argument separator, as I am, then you can use the Arguments list. Which will split up the arguments by the , character. Surprisingly in the 101 lines of code in the sample there are only a few interesting lines. Those are what’s contained inside the Apply method. Inside that method I search the TargetNode’s attributes (TargetNode is the node which was matched in the xml file being transformed) for an attribute with the same name as the one specified in the AttributeName property. Once I find it I just make a call to Regex.Replace to get the new value, and assign it. Pretty simple! Now lets see how can we use this.

Let’s say you have the following very simple web.config

<?xml version="1.0"?>
<configuration>
  <appSettings>
    <add key="one" value="one"/>
    <add key="two" value="partial-replace-here-end"/>
    <add key="three" value="three here"/>
  </appSettings>
</configuration>

If we want to be able to use our own transform then we will have to use the xdt:Import element. You can place that element inside the xml document anywhere immediately under the root element. This element will allow us to utilize our own transform class. It only has 3 possible attributes.

  • namespace – This is the namespace which the transform is contained in
  • path – This is the full path to the assembly
  • assembly – This is the assembly name which contains the transform

You can only use one of the two; path and assembly. Basically it boils down to how the assembly is loaded. If you use path the assembly will be loaded with Assembly.LoadFrom and if you chose to use assembly passing in the AssemblyName, for instance if the assembly in in the GAC, then it will be loaded using Assembly.Load.

I chose to use path, because I just placed the file inside of the MSBuild Extensions directory (%Program Files (x86)%MSBuild) in a folder named Custom. Then I created my config transform file to be the following.

<?xml version="1.0"?>

<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
                        
  <xdt:Import path="C:\Program Files (x86)\MSBuild\Custom\CustomTransformType.dll"
              namespace="CustomTransformType" />

  <appSettings>
    <add key="one" value="one-replaced" xdt:Transform="Replace" xdt:Locator="Match(key)" />
    <add key="two" value="two-replaced" xdt:Transform="AttributeRegexReplace(Attribute='value', Pattern='here',Replacement='REPLACED')" xdt:Locator="Match(key)"/>
  </appSettings>
</configuration>

Then to run this I created an MSBuild file, PerformTransform.proj, which is shown below.

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" DefaultTargets="Demo" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <UsingTask TaskName="TransformXml"
           AssemblyFile="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v10.0\Web\Microsoft.Web.Publishing.Tasks.dll"/>

  <PropertyGroup>
    <TransformDest>web.tranzed.config</TransformDest>
  </PropertyGroup>
  
  <Target Name="Demo">
    <Delete Files="$(TransformDest)" />
    <TransformXml Source="web.config"
                  Transform="web.dev.config"
                  Destination="$(TransformDest)" />                  
  </Target>
  
</Project>

This file uses the TransformXml task as I outlined in a previous post Config transformations outside of web app builds. Once you execute the Demo target with the command msbuild PerformTransform.proj /t:Demo you will see the file web.tranzed.config with the following contents.

<?xml version="1.0"?>
<configuration>
  <appSettings>
    <add key="one" value="one-replaced"/>
    <add key="two" value="partial-replace-REPLACED-end"/>
    <add key="three" value="three here"/>
  </appSettings>
</configuration>

So you can see that the replacement did occur as we intended. Below you will find the download link for the samples as well as another blog entry for more info on the out of the box transformations.

Resources

Sayed Ibrahim Hashimi

Config-Transformation | msbuild | MSDeploy Thursday, September 09, 2010 6:51:43 AM (GMT Daylight Time, UTC+01:00)  #     | 

Config transformations outside of web app builds

If you are using Visual Studio 2010 then you may already be familiar with the Web.config transformations that are now available. What you might not know is that you can use that same technology to transform config files outside of the build process. You will need Visual Studio 2010 installed on the machine where you perform these transformations. It is very easy to perform these transformation as well. Let’s say that we start with the app.config file shown below.

<configuration>
    <connectionStrings>
        <clear/>
        <add name="Default" connectionString="Data Source=localhost;Initial Catalog=Sample01;Integrated Security=True;" />
    </connectionStrings>
    
    <appSettings>
        <add key="contactEmail" value="contact@demo.example.com"/>
        <add key="siteUrl" value="http://demo.example.com"/>
    </appSettings>
    
</configuration>

Then we create another file, transform.xml, which contains our transformations. That file is shown below.

<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
    <connectionStrings>
        <clear/>
        <add name="Default" connectionString="Data Source=NOT-localhost;Initial Catalog=Sample01;Integrated Security=True;" 
             xdt:Locator="Match(name)" xdt:Transform="Replace"/>
    </connectionStrings>

    <appSettings>
        <add key="contactEmail" value="contact@example.com" xdt:Locator="Match(key)" xdt:Transform="Replace"/>
        <add key="siteUrl" value="http://example.com" xdt:Locator="Match(key)" xdt:Transform="Replace"/>
    </appSettings>

</configuration>

Then we can easily execute the transformations by using MSBuild. So I created a file named trans.proj and it is shown below.

<Project ToolsVersion="4.0" DefaultTargets="Demo" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    <UsingTask TaskName="TransformXml"
             AssemblyFile="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v10.0\Web\Microsoft.Web.Publishing.Tasks.dll"/>

    <Target Name="Demo">
        <TransformXml Source="app.config"
                      Transform="Transform.xml"
                      Destination="app.prod.config"/>
    </Target>
</Project>

This MSBuild file uses the TransformXml task which is shipped with Visual Studio 2010. We specify the source file, transform file and the destination. Pretty straight forward.

In order to execute this I open a Visual Studio 2010 command prompt, browse to the directory containing both files, and enter the following command

msbuild trans.proj /t:Demo

Once you do this then you will find the file app.prod.config with the following contents.

<configuration>
    <connectionStrings>
        <clear/>
        <add name="Default" connectionString="Data Source=NOT-localhost;Initial Catalog=Sample01;Integrated Security=True;"/>
    </connectionStrings>
    
    <appSettings>
        <add key="contactEmail" value="contact@example.com"/>
        <add key="siteUrl" value="http://example.com"/>
    </appSettings>
    
</configuration>

Sayed Ibrahim Hashimi

Config-Transformation | msbuild | MSBuild 4.0 | MSDeploy | Visual Studio | Visual Studio 2010 Monday, April 26, 2010 5:22:06 AM (GMT Daylight Time, UTC+01:00)  #     |