- | rssFeed | My book on MSBuild and Team Build | Archives and Categories Thursday, September 23, 2010

MSBuild: You’ve heard of incremental building but have you heard of partial building?

MSBuild supports a concept known as Incremental Building which you may already be familiar with but I will cover it here briefly but that is not the focus of this post. Incremental building is the concept that you should only build what is out of date. To support this MSBuild has the attributes, inputs and outputs on the Target element. With these attributes you can specify the files that go into a target (via inputs attribute), and the files that you are expecting to come out of a target (via outputs attribute). Once you do this MSBuild will compare the timestamp of the inputs to the outputs and if all outputs are up-to-date (i.e. the inputs are older) then the target will be skipped. Take a look at the very simple project file below.



  
    
  

  
    dest\
  
    
  
    
    
    
  

  
    
    
  

In this project file we have two targets; CopyFiles and DeleteTwoFiles. Ignore DeleteTwoFiles for now. Also take a note that the directory where I’m executing this build has a folder, src, with the files listed in the Files item. On the CopyFiles target I have specified the inputs and outputs. The inputs is just @(Files), this are the files that the target is acting upon. The outputs contains the expression @(Files->'$(Dest)%(Filename)%(Extension)'). Which is the same expression from the Copy statement. If the Dest folder is empty and I execute the CopyFiles target the result is shown below.

image

So just as expected the files were copied over, so its all good. Now what happens if I execute it again? The output is shown below.

image

So as you can see the target was skipped, the message statement “CopyFiles” was not executed nor was the copy as a result. So this, in a nutshell, is incremental building.

Now, with the dest folder containing all files, what do you think would happen I execute the command msbuild.exe PartialBuilding01.proj /t:DeleteTwoFiles;CopyFiles? This command will first delete two files from the output directory and then call the CopyFiles target again. Let’s see the result below.

image

When the CopyFiles target was executed you see that statement “Building target ‘CopyFiles’ partially, …”. When the time came to execute the target MSBuild examined the inputs and outputs, it determined that the files 01.txt & 02.txt were out of date (because they didn’t exist in the target) but 03.txt, 04.txt and 05.txt were up to date. So MSBuild feed the CopyFiles target a value for the Files item that only contained the 01.txt and 02.txt and let it do its thing. So if you are performing builds which are taking a lot of time then incremental (and therefore partial) building is your friend!

Sayed Ibrahim Hashimi

msbuild Thursday, September 23, 2010 7:05:59 AM (GMT Daylight Time, UTC+01:00)  #     | 
Thursday, September 09, 2010

Extending XML (web.config) Config transformation

Last week on StackOverflow I answered a question, Make web.config transformations working locally and in a response to my answer the question asker asked me if I would be able to a question he posed earlier Advanced tasks using web.config transformation. Evidently he is really interested in config transformations! I don’t blame him, I’m really into them as well.

In his question he asks (summarizing) can we replace portions of attribute values instead of this entire attribute? So for instance you have the following in your web.config. Below is two sets of appSettings one from Dev and the other from Prod (taken from the original question).

<!-- DEV ENTRY -->
<appSettings>
 <add key="serviceName1_WebsService_Url" value="http://wsServiceName1.dev.domain.com/v1.2.3.4/entryPoint.asmx" />
 <add key="serviceName2_WebsService_Url" value="http://ma1-lab.lab1.domain.com/v1.2.3.4/entryPoint.asmx" />
</appSettings>

<!-- PROD ENTRY -->
<appSettings>
 <add key="serviceName1_WebsService_Url" value="http://wsServiceName1.prod.domain.com/v1.2.3.4/entryPoint.asmx" />
 <add key="serviceName2_WebsService_Url" value="http://ws.ServiceName2.domain.com/v1.2.3.4/entryPoint.asmx" />
</appSettings>

In the above we just want to replace dev with prod and ma1-lab.lab1.domain with ws.ServiceName2.domain. For those wondering currently we have the following transformations out of the box.

At the end of this article I’ve linked to another blog which has more info about these transformations. So it sounds like SetAttributes is almost what we want, but not quite what there. A little known fact is that you can create your own config transformations and use those. In fact all of the out of the box transformations follow the same patterns that custom transformations would. To solve this issue we need to create our own config transformation, AttributeRegexReplace. This transformation will take an attribute value and do a regular expression replace on its value. In order to create a new transformation you first reference the Microsoft.Web.Publishing.Tasks.dll which can be found in the %Program Files (x86)%MSBuild\Microsoft\VisualStudio\v10.0\Web folder. If you are working with a team it is best if you copy that assembly, place it in a shared folder in source control, and make the reference from that location. After you create the reference to that assembly you will need to create a class which extends the Transform class. The class diagram for this abstract class is shown below.

image

The only thing that you will need to implement is the Apply method. You don’t even need to fully understand all of the properties and methods just the portions that you are interested in. Here we will not cover all the details of this class, or other related classes which exist, but there will be future posts which will shed more light on this area.

In the sample class library that I created, I called the project CustomTransformType. Inside of that project I created the class AttributeRegexReplace. The entire contents of that class are shown below, we will go over the details after that.

namespace CustomTransformType
{
    using System;
    using System.Text.RegularExpressions;
    using System.Xml;
    using Microsoft.Web.Publishing.Tasks;

    public class AttributeRegexReplace : Transform
    {
        private string pattern;
        private string replacement;
        private string attributeName;

        protected string AttributeName
        {
            get
            {
                if (this.attributeName == null)
                {
                    this.attributeName = this.GetArgumentValue("Attribute");
                }
                return this.attributeName;
            }
        }
        protected string Pattern
        {
            get
            {
                if (this.pattern == null)
                {
                    this.pattern = this.GetArgumentValue("Pattern");
                }

                return pattern;
            }
        }

        protected string Replacement
        {
            get
            {
                if (this.replacement == null)
                {
                    this.replacement = this.GetArgumentValue("Replacement");
                }

                return replacement;
            }
        }

        protected string GetArgumentValue(string name)
        {
            // this extracts a value from the arguments provided
            if (string.IsNullOrWhiteSpace(name)) 
            { throw new ArgumentNullException("name"); }

            string result = null;
            if (this.Arguments != null && this.Arguments.Count > 0)
            {
                foreach (string arg in this.Arguments)
                {
                    if (!string.IsNullOrWhiteSpace(arg))
                    {
                        string trimmedArg = arg.Trim();
                        if (trimmedArg.ToUpperInvariant().StartsWith(name.ToUpperInvariant()))
                        {
                            int start = arg.IndexOf('\'');
                            int last = arg.LastIndexOf('\'');
                            if (start <= 0 || last <= 0 || last <= 0)
                            {
                                throw new ArgumentException("Expected two ['] characters");
                            }

                            string value = trimmedArg.Substring(start, last - start);
                            if (value != null)
                            {
                                // remove any leading or trailing '
                                value = value.Trim().TrimStart('\'').TrimStart('\'');
                            }
                            result = value;
                        }
                    }
                }
            }
            return result;
        }

        protected override void Apply()
        {
            foreach (XmlAttribute att in this.TargetNode.Attributes)
            {
                if (string.Compare(att.Name, this.AttributeName, StringComparison.InvariantCultureIgnoreCase) == 0)
                {
                    // get current value, perform the Regex
                    att.Value = Regex.Replace(att.Value, this.Pattern, this.Replacement);
                }
            }
        }
    }
}

In this class we have 3 properties; Pattern, Replacement, and AttributeName. All of these values will be provided via an argument in the config transformation. For example take a look at the element below which contains a transform attribute may look like the following.

<add key="two" value="two-replaced" 
         xdt:Transform="AttributeRegexReplace(Attribute='value', Pattern='here',Replacement='REPLACED')" 
         xdt:Locator="Match(key)"/>

In this example I declare that I am using AttributeRegexReplace and then specify the values for the attributes within the (). In the class above I have a method, GetArgumentValue, which is used to parse values from that argument string. When your transform is invoked the string inside of () is passed in as the ArgumentString value. If you are using a , as the argument separator, as I am, then you can use the Arguments list. Which will split up the arguments by the , character. Surprisingly in the 101 lines of code in the sample there are only a few interesting lines. Those are what’s contained inside the Apply method. Inside that method I search the TargetNode’s attributes (TargetNode is the node which was matched in the xml file being transformed) for an attribute with the same name as the one specified in the AttributeName property. Once I find it I just make a call to Regex.Replace to get the new value, and assign it. Pretty simple! Now lets see how can we use this.

Let’s say you have the following very simple web.config

<?xml version="1.0"?>
<configuration>
  <appSettings>
    <add key="one" value="one"/>
    <add key="two" value="partial-replace-here-end"/>
    <add key="three" value="three here"/>
  </appSettings>
</configuration>

If we want to be able to use our own transform then we will have to use the xdt:Import element. You can place that element inside the xml document anywhere immediately under the root element. This element will allow us to utilize our own transform class. It only has 3 possible attributes.

You can only use one of the two; path and assembly. Basically it boils down to how the assembly is loaded. If you use path the assembly will be loaded with Assembly.LoadFrom and if you chose to use assembly passing in the AssemblyName, for instance if the assembly in in the GAC, then it will be loaded using Assembly.Load.

I chose to use path, because I just placed the file inside of the MSBuild Extensions directory (%Program Files (x86)%MSBuild) in a folder named Custom. Then I created my config transform file to be the following.

<?xml version="1.0"?>

<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
                        
  <xdt:Import path="C:\Program Files (x86)\MSBuild\Custom\CustomTransformType.dll"
              namespace="CustomTransformType" />

  <appSettings>
    <add key="one" value="one-replaced" xdt:Transform="Replace" xdt:Locator="Match(key)" />
    <add key="two" value="two-replaced" xdt:Transform="AttributeRegexReplace(Attribute='value', Pattern='here',Replacement='REPLACED')" xdt:Locator="Match(key)"/>
  </appSettings>
</configuration>

Then to run this I created an MSBuild file, PerformTransform.proj, which is shown below.

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" DefaultTargets="Demo" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <UsingTask TaskName="TransformXml"
           AssemblyFile="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v10.0\Web\Microsoft.Web.Publishing.Tasks.dll"/>

  <PropertyGroup>
    <TransformDest>web.tranzed.config</TransformDest>
  </PropertyGroup>
  
  <Target Name="Demo">
    <Delete Files="$(TransformDest)" />
    <TransformXml Source="web.config"
                  Transform="web.dev.config"
                  Destination="$(TransformDest)" />                  
  </Target>
  
</Project>

This file uses the TransformXml task as I outlined in a previous post Config transformations outside of web app builds. Once you execute the Demo target with the command msbuild PerformTransform.proj /t:Demo you will see the file web.tranzed.config with the following contents.

<?xml version="1.0"?>
<configuration>
  <appSettings>
    <add key="one" value="one-replaced"/>
    <add key="two" value="partial-replace-REPLACED-end"/>
    <add key="three" value="three here"/>
  </appSettings>
</configuration>

So you can see that the replacement did occur as we intended. Below you will find the download link for the samples as well as another blog entry for more info on the out of the box transformations.

Resources

Sayed Ibrahim Hashimi

Config-Transformation | msbuild | MSDeploy Thursday, September 09, 2010 6:51:43 AM (GMT Daylight Time, UTC+01:00)  #     |