- | rssFeed | My book on MSBuild and Team Build | Archives and Categories Tuesday, 22 July 2014

Stop checking-in binaries, instead create self-bootstrapping scripts

A few weeks ago Mads Kristensen and I created a few site extensions for Azure Web Sites which the Azure Image Optimizer and Azure Minifier. These extensions can be used to automatically optimize all images on a site, and minify all .js/.css files respectively. These are shipped as nuget packages in nuget.org as well as site extensions in siteextensions.net.

After creating those utilities we also update the image optimizer to support being called in on the command line via a .exe. We have not yet had a chance to update the minifier to be callable directly but we have an open issue on it. If you can help that would be great.

The exe for the image optimizer that can be used from the command line can be found in the nuget package as well. You can also download it from here, but to get the latest version you nuget.org is the way to go.

After releasing that exe I wanted an easy way to use it on a variety of machines, and to make it simple for others to try it out. What I ended up with is what I’m calling a “self-bootstrapping script” which you can find at optimize-images.ps1. Below you’ll see the entire contents of the script.

[cmdletbinding()]
param(
    $folderToOptimize = ($pwd),

    $toolsDir = ("$env:LOCALAPPDATA\LigerShark\tools\"),

    $nugetDownloadUrl = 'http://nuget.org/nuget.exe'
)

<#
.SYNOPSIS
    If nuget is in the tools
    folder then it will be downloaded there.
#>
function Get-Nuget(){
    [cmdletbinding()]
    param(
        $toolsDir = ("$env:LOCALAPPDATA\LigerShark\tools\"),

        $nugetDownloadUrl = 'http://nuget.org/nuget.exe'
    )
    process{
        $nugetDestPath = Join-Path -Path $toolsDir -ChildPath nuget.exe
        
        if(!(Test-Path $nugetDestPath)){
            'Downloading nuget.exe' | Write-Verbose
            (New-Object System.Net.WebClient).DownloadFile($nugetDownloadUrl, $nugetDestPath)

            # double check that is was written to disk
            if(!(Test-Path $nugetDestPath)){
                throw 'unable to download nuget'
            }
        }

        # return the path of the file
        $nugetDestPath
    }
}

<#
.SYNOPSIS
    If the image optimizer in the .ools
    folder then it will be downloaded there.
#>
function GetImageOptimizer(){
    [cmdletbinding()]
    param(
        $toolsDir = ("$env:LOCALAPPDATA\LigerShark\tools\"),
        $nugetDownloadUrl = 'http://nuget.org/nuget.exe'
    )
    process{
        
        if(!(Test-Path $toolsDir)){
            New-Item $toolsDir -ItemType Directory | Out-Null
        }

        $imgOptimizer = (Get-ChildItem -Path $toolsDir -Include 'ImageCompressor.Job.exe' -Recurse)

        if(!$imgOptimizer){
            'Downloading image optimizer to the .tools folder' | Write-Verbose
            # nuget install AzureImageOptimizer -Prerelease -OutputDirectory C:\temp\nuget\out\
            $cmdArgs = @('install','AzureImageOptimizer','-Prerelease','-OutputDirectory',(Resolve-Path $toolsDir).ToString())

            'Calling nuget to install image optimzer with the following args. [{0}]' -f ($cmdArgs -join ' ') | Write-Verbose
            &(Get-Nuget -toolsDir $toolsDir -nugetDownloadUrl $nugetDownloadUrl) $cmdArgs | Out-Null
        }

        $imgOptimizer = Get-ChildItem -Path $toolsDir -Include 'ImageCompressor.Job.exe' -Recurse | select -first 1
        if(!$imgOptimizer){ throw 'Image optimizer not found' }       

        $imgOptimizer
    }
}

function OptimizeImages(){
    [cmdletbinding()]
    param(
        [Parameter(Mandatory=$true,Position=0)]
        $folder,
        $toolsDir = ("$env:LOCALAPPDATA\LigerShark\tools\"),
        $nugetDownloadUrl = 'http://nuget.org/nuget.exe'
    )
    process{        
        [string]$imgOptExe = (GetImageOptimizer -toolsDir $toolsDir -nugetDownloadUrl $nugetDownloadUrl)

        [string]$folderToOptimize = (Resolve-path $folder)

        'Starting image optimizer on folder [{0}]' -f $folder | Write-Host
        # .\.tools\AzureImageOptimizer.0.0.10-beta\tools\ImageCompressor.Job.exe --folder M:\temp\images\opt\to-optimize
        $cmdArgs = @('--folder', $folderToOptimize)

        'Calling img optimizer with the following args [{0} {1}]' -f $imgOptExe, ($cmdArgs -join ' ') | Write-Host
        &$imgOptExe $cmdArgs

        'Images optimized' | Write-Host
    }
}

OptimizeImages -folder $folderToOptimize -toolsDir $toolsDir -nugetDownloadUrl $nugetDownloadUrl

The script is setup to where you call functions like Get-NuGet and GetImageOptimzer to get the path to the .exe to call. If the .exe is not in the expected location, under %localappdata% by default, it will be downloaded and then the path will be returned. In the case of this script I use nuget.org as my primary distribution mechanism for this so the script will first download nuget.exe and then use that to get the actual binaries. WIth this approach, you can avoid checking in binaries and have scripts which are still pretty concise.

After creating optimize-images.ps1 I thought it would be really useful to have a similar script to execute XDT transforms on xml files. So I created transform-xml.ps1. That script first downloads nuget.exe and then uses that to download the nuget packages which are required to invoke XDT transforms.

A self-bootstrapping script doesn’t need to be a PowerShell script, you can apply the same techniques to any scripting language. I’ve recently created an MSBuild script, inspired by Get-Nuget above, which can be used in a similar way. You can find that script in a gist here. It’s below as well.

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="GetNuget">

  <!--
  This is an MSBuild snippet that can be used to download nuget to the path
  in the property NuGetExePath property.
  
  Usage:
   1. Import this file or copy and paste this into your build script
   2. Call the GetNuGet target before you use nuget.exe from $(NuGetExePath)
  -->
  
  <PropertyGroup>
    <NuGetExePath Condition=" '$(NuGetExePath)'=='' ">$(localappdata)\LigerShark\AzureJobs\tools\nuget.exe</NuGetExePath>
    <NuGetDownloadUrl Condition=" '$(NuGetDownloadUrl)'=='' ">http://nuget.org/nuget.exe</NuGetDownloadUrl>
  </PropertyGroup>
  
  <Target Name="GetNuget" Condition="!Exists('$(NuGetExePath)')">
    <Message Text="Downloading nuget from [$(NuGetDownloadUrl)] to [$(NuGetExePath)]" Importance="high"/>
    <ItemGroup>
      <_nugetexeitem Include="$(NuGetExePath)" />
    </ItemGroup>
    <MakeDir Directories="@(_nugetexeitem->'%(RootDir)%(Directory)')"/>
    <DownloadFile
        Address="$(NuGetDownloadUrl)"
        FileName="$(NuGetExePath)" />    
  </Target>

  <PropertyGroup Condition=" '$(ls-msbuildtasks-path)'=='' ">
    <ls-msbuildtasks-path>$(MSBuildToolsPath)\Microsoft.Build.Tasks.v4.0.dll</ls-msbuildtasks-path>
    <ls-msbuildtasks-path Condition=" !Exists('$(ls-msbuildtasks-path)')">$(MSBuildFrameworkToolsPath)\Microsoft.Build.Tasks.v4.0.dll</ls-msbuildtasks-path>
    <ls-msbuildtasks-path Condition=" !Exists('$(ls-msbuildtasks-path)')">$(windir)\Microsoft.NET\Framework\v4.0.30319\Microsoft.Build.Tasks.v4.0.dll</ls-msbuildtasks-path>
  </PropertyGroup>
  
  <UsingTask TaskName="DownloadFile" TaskFactory="CodeTaskFactory" AssemblyFile="$(ls-msbuildtasks-path)">
    <!-- http://stackoverflow.com/a/12739168/105999 -->
    <ParameterGroup>
      <Address ParameterType="System.String" Required="true"/>
      <FileName ParameterType="System.String" Required="true" />
    </ParameterGroup>
    <Task>
      <Reference Include="System" />
      <Code Type="Fragment" Language="cs">
        <![CDATA[
            new System.Net.WebClient().DownloadFile(Address, FileName);
        ]]>
      </Code>
    </Task>
  </UsingTask>
  
</Project>

This script has a single target, GetNuGet, which you can call to download nuget.exe to the expected location. After that you can use the path to nuget.exe from the NuGetExePath property. I’ve already removed nuget.ext from SideWaffle and AzureJobs repository using this technique. It’s a great way to avoid checking in nuget.exe.

 

Thanks,
Sayed Ibrahim Hashimi | http://msbuildbook.com | @SayedIHashimi

msbuild | powershell | scripting Tuesday, 22 July 2014 07:36:11 (GMT Daylight Time, UTC+01:00)  #     | 
Saturday, 19 July 2014

Introducing PSBuild – an improved interface for msbuild.exe in PowerShell

For the past few months I’ve been working on a project I’m calling PSBuild. It’s an open source project on GitHub which makes the experience of calling MSBuild from PowerShell better. Getting started with PSBuild is really easy. You can install it in one line.

(new-object Net.WebClient).DownloadString("https://raw.github.com/ligershark/psbuild/master/src/GetPSBuild.ps1") | iex

You can find this info on the project home page as well.

 

When you install PSBuild one of the functions that you get is Invoke-MSBuild. When you call Invoke-MSBuild it will end up calling msbuild.exe. Some advantages of using Invoke-MSBuild are.

Calling Invoke-MSBuild

A most basic usage of Invoke-MSBuild.

Invoke-MSBuild .\App.csproj

This will build the project using the default targets. The call to msbuild.exe on my computer is below.

C:\Program Files (x86)\MSBuild\12.0\bin\amd64\msbuild.exe 
    .\App.csproj 
    /m 
    /clp:v=m
    /flp1:v=d;logfile=C:\Users\Sayed\AppData\Local\PSBuild\logs\App.csproj-log\msbuild.detailed.log 
    /flp2:v=diag;logfile=C:\Users\Sayed\AppData\Local\PSBuild\logs\App.csproj-log\msbuild.diagnostic.log

From the call to msbuild.exe you can see that the /m is passed as well as a couple file loggers in %localappdata%. We will get to the logs later. More on Invoke-MSBuild,

To get a sense for how you can use Invoke-MSBuild take a look at the examples below.

# VisualStudioVersion and Configuration MSBuild properties have easy to use parameters
Invoke-MSBuild .\App.csproj -visualStudioVersion 12.0 -configuration Release

# How to pass properties
Invoke-MSBuild .\App.csproj -visualStudioVersion 12.0 -properties @{'DeployOnBuild'='true';'PublishProfile'='toazure';'Password'='mypwd-really'}

# How to build a single target
Invoke-MSBuild .\App.csproj -visualStudioVersion 12.0 -targets Clean

# How to bulid multiple targets
Invoke-MSBuild .\App.csproj -visualStudioVersion 12.0 -targets @('Clean','Build')

 

Getting log files

When you call Invoke-MSBuild on a project or solution the output will look something like the following.

image

Notice that line at the end. You can access your last log using the command.

Open-PSBuildLog

This will open the detailed log of the previous build in the program that’s associated with the .log extension. You can also use the Get-PSBuildLastLogs function to view the path for both log files written. If you want to view the folder where the log files are written to you can execute start (Get-PSBuildLogDirectory).

Helper functions

There are a couple of things that I’m constantly having to look up when I’m authoring MSBuild files; Reserved property names and escape characters. PSBuild has a helper function for each of these Get-MSBuildEscapeCharacters and Get-MSBuildReservedProperties. In the screenshot below you’ll see the result of executing each of these.

image

 

Default Properties

The Invoke-MSBuild cmdlet has a property –defaultProperties. You can pass in a hashtable just like the –properties parameter. These properties are applied as environment variables before the call to msbuild.exe and then reverted afterwards. The effect here is that you can have a property value which will be used if no other value for that property is specified in MSBuild.

 

There is so much more to PSBuild. This is just the tip of the iceberg. Keep an eye on the project page for more info. I’d love your help on this project. Please consider contributing to the project https://github.com/ligershark/psbuild.

 

Thanks,
Sayed Ibrahim Hashimi | http://msbuildbook.com/ | @SayedIHashimi

msbuild | psbuild Saturday, 19 July 2014 06:20:43 (GMT Daylight Time, UTC+01:00)  #     |