Richard's Braindump

Agile KPI's

2008-05-20T09:01:00.001+10:00

I was recently asked by Aaron to do a blog post about KPI's when using Scrum (thanks for the suggestion!). The basic idea being that many companies want to have role level KPI's, but how do you do that with Scrum, especially when Scrum seems to be about the performance of the team. I say "seems" because it's often how people who are new to Scrum will look at things, which is unfortunate as they miss the point that scrum is concerned with not just the team's results but with the individuals in that team as well.

So, what KPI's should be considered for members of a scrum team? At a team level it's easy enough - delivery of working software that provides business value. But for individuals, what do we do?

The first instinct might be do look for something that's easy to measure such as taking on a certain number of hours worth of tasks per sprint, or being present on time at the daily stand up meetings, but these would be poor KPI's as they measure attendance and allocation of tasks, and not delivery of business value by individuals. Instead I'd suggest that what we really want to gauge is how much overall value people are adding to the business and the team - and that value isn't always tied directly to their work artifacts/outputs.

Here's some suggestions for KPI's you could apply to individuals in a scrum team:

Positive Team Involvement
Mentoring
Knowledge Sharing
Proactive Alerting of Impediments
Working According to Team Standards (e.g. TDD, style & naming, code comments & documentation, regular check-ins, etc)

In case it's not implicitly obvious, I'd suggest that individuals in the team would have different expectations on their KPI measures. For example, the more experienced team members would be expected to spend more time mentoring than the juniors. The team members with greater domain knowledge would be expected to spend more time sharing that knowledge than those who are new to the problem domain, etc.

So then how do we then measure these factors? One method might be through peer feedback, others might use independent observation, others might try something completely different. My suggestion would be a mix of observation and feedback - the scrum master needs to observe the team dynamics and behaviours on an ongoing basis, however they often miss things that the quiet achievers do during the day, so having a peer review process is a good thing to add to the mix. You might want to get feedback from the team at the end of each sprint or you might choose to have a 6 monthly anonymous feedback mechanism and use that with a mix of judgement to work things out. Whatever you choose it will end up being a judgement call as to wether people met their KPI's or not.

The most important thing to remember is that in the same way that the team is measured on results, individual KPI's should be trying to measure the individuals value, not their effort or their time spent in the office.

Unity vs Castle Windsor

2008-05-13T21:19:00.001+10:00

Recently I did a post on breaking dependencies which used the Castle Windsor container as the Inversion of Control container. I thought I'd also have a look at doing the same thing with the Unity container from Microsoft's Patterns & Practices team, which was version 1.0'ed last month.

The good news? It's pretty much just some slight syntax changes and that's about it. The only problem I found was that after installing the Unity help file for VS2008 my Visual Studio help is now broken and now I have to go and repair my Visual Studio installation. I'd suggest sticking to the CHM file instead.

Here's the only differences between the two applications:

1. References (duh!)

Castle Windsor

using Castle.Windsor;
using Castle.Core.Resource;
using Castle.Windsor.Configuration.Interpreters;

Unity

using System.Configuration;
using Microsoft.Practices.Unity;
using Microsoft.Practices.Unity.Configuration;

2. Instantiating the Container

Castle Windsor

            IWindsorContainer container = new WindsorContainer(new XmlInterpreter());

Unity

            IUnityContainer container = new UnityContainer();

            UnityConfigurationSection section = (UnityConfigurationSection)ConfigurationManager.GetSection("unity");
            section.Containers.Default.Configure(container);

3. The .config files

Castle Windsor

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <configSections>
        <section name="castle" type="Castle.Windsor.Configuration.AppDomain.CastleSectionHandler, Castle.Windsor" />
    </configSections>

    <castle>
        <components>
            <component id="prompt.input" service="SalesPrompter.IInputControl, SalesPrompter"
                type="SalesPrompter.PromptInputControl, SalesPrompter" />
            <component id="console.display" service="SalesTax.IMessageDisplay, SalesTax"
                type="SalesTax.ConsoleMessageDisplay, SalesTax" />
            <component id="input.parser" service="SalesTax.IInputParser, SalesTax"
                type="SalesTax.InputParser, SalesTax" />
            <component id="salelinefactory" service="SalesTax.ISaleLineFactory, SalesTax"
                type="SalesTax.SaleLineFactory, SalesTax" />
        </components>
    </castle>

</configuration>

Unity

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <configSections>
        <section name="unity"  type="Microsoft.Practices.Unity.Configuration.UnityConfigurationSection, Microsoft.Practices.Unity.Configuration" />
    </configSections>

    <unity>
        <typeAliases></typeAliases>
        <containers>
            <container>
                <types>
                    <type type="SalesPrompter.IInputControl, SalesPrompter" 
                        mapTo="SalesPrompter.PromptInputControl, SalesPrompter"/>
                    <type type="SalesTax.IMessageDisplay, SalesTax" 
                        mapTo="SalesTax.ConsoleMessageDisplay, SalesTax"/>
                    <type type="SalesTax.IInputParser, SalesTax" 
                        mapTo="SalesTax.InputParser, SalesTax"/>
                    <type type="SalesTax.ISaleLineFactory, SalesTax" 
                        mapTo="SalesTax.SaleLineFactory, SalesTax"/>
                </types>
            </container>
        </containers>
    </unity>

</configuration>

Note that there was nothing different in the resolving of references. Both containers have a Resolve<T>() method that does exactly the same thing.

So, how easy is that? :-) The P&P team have done a pretty good job with this container. Admittedly I haven't really looked into some of the more advanced options, but it looks quite good and being from the P&P team it's likely to get more traction that the Castle Windsor container in the corporate development world.

Breaking Dependencies using Dependency Injection

2008-05-07T13:05:00.001+10:00

Let's assume you've done some reading recently and heard talk of this Loosely Coupled Architecture thing and how it's meant to help keep an application maintainable, make it more readily changeable and will assist in the creation of unit tests . You know that it's the right way to go and you can see all the benefits of it. You even think "I should do this on that application at work", but every time you look at the code at work and wonder what space monkeys were hired to write the code in the first place and you think about how tightly bound everything is it all starts to look too hard, so you give up and get back to working with things as they because the problem seems insurmountable.

Now a loosely coupled architecture is at it's heart just another way of saying object oriented design. I'm sure you'll remember from when you learnt all that Object Oriented Programming stuff that there are some basic principle for OO Programming, namely:

Abstraction, Encapsulation, Polymorphism and Inheritance

And there are also some OO Design principles that sit alongside the OOP principles:

Encapsulate what changes

Favour composition over inheritance

Program to interfaces not implementations

Depend on abstractions, not concrete classes (dependency inversion)

A class should have only one reason to change (single responsibility)

Your work code and the "this problem is too big" feeling doesn't have to be the case. It's just an indication that you've got legacy code and that it isn't well designed. The good news is that by following some basic steps you can refactor a legacy, tightly-bound application into one with a loosely coupled architecture, though it will take some time (think many months, not weeks).

There are two basic methods for doing this. The first is Extract & Override (see Roy Osherove's blog entry) and the other is Dependency Injection. This is what I'll focus on here.

First, let's have a look at some code from a simple sales application:

    public class Sale
    {
        private List<SaleLine> saleLines = new List<SaleLine>();
        private decimal totalTax;
        private decimal totalValue;
        ...
        public bool Add(string inputLine)
        {
            SaleLine saleLine;

            saleLine = InputParser.ProcessInput(inputLine);
            if (saleLine == null)
                return false;
            saleLines.Add(saleLine); 
            ...

Here we see that a sale has multiple sale lines associated with it. Pretty normal, and something you'll see in many code bases the world over, but we're breaking some of the OOD rules - we're programming to implementations not interfaces and we're using a static method (on the InputParser) which means we're tightly coupled between the Sale object and the SaleLine object.

We're also doing something else that's a no-no. The Add method is coordinating a whole lot of work to convert an input string into a SaleLine object before adding it to the saleLines collection, whichmeans the class does more than one thing.

Let's have a quick look at the ProcessInput method on the InputParser class as well

        public static SaleLine ProcessInput(string input)
        {
            SaleLine saleLine;
            ...
            // create the sale line
            saleLine = new SaleLine(quantity, name, price);
            return saleLine;
        }

What we have here is another tight coupling. We're creating saleline objects from within the method (the new SaleLine() call) which means we have a tight dependency between the saleline class and the InputParser class. If we wanted to change the SaleLine to some other class, we'd have some refactoring to do.

So let's make some changes...

Lets start by creating interfaces for all of our concrete classes, for example:

    public class Sale : ISale
    {
        private List<ISaleLine> saleLines = new List<ISaleLine>();

Now, if our definition of what a sale line is changes we don't have to make any changes in the Sale object. Nice.

Let's also change the input parser so it doesn't create Sale objects directly. We should do that through the use of a factory pattern implementation, such as the following:

    public class SaleLineFactory : ISaleLineFactory
    {
        public ISaleLine GetNewSaleLine(int lineQuantity, string name, decimal unitPrice)
        {
            return new SaleLine(lineQuantity, name, unitPrice);
        }
    }

And in our InputParser we change the code as follows:

            saleLine = saleLineFactory.GetNewSaleLine(quantity, productName, price);

Good. So now we're not actually creating the SaleLine object directly in our code. This is a good thing as it breaks the tight coupling between the classes, but doesn't it just move the dependency from the SaleLine class to the SaleLineFactory class?

It would if we just added saleLineFactory = new SaleLineFactory() to the InputParser, but not if we pass an instance of the saleLineFactory class to the InputParser. This is the Dependency Injection pattern at work.

Now, while we're working in the InputParser class we also have the opportunity to fix another tight coupling. That between the Sale class and the InputParser. Let's change the input parser so that we're no longer a static class, but rather a normal class (in a real world situation this could be complex) and let's pass to the ProcessInput method the Sale we want the new line to be added to. We can then change the sale object so that it's Add method just adds ISaleLine objects instead of being passed a string and then calling to the parser. Something like this:

    public class Sale : ISale
    {
        private List<ISaleLine> saleLines = new List<ISaleLine>();
        ...
        public bool Add(ISaleLine saleLine)
        {
            if (saleLine == null)
                return false;
            saleLines.Add(saleLine);

    public class InputParser : IInputParser 
    {
        private ISaleLineFactory saleLineFactory;
        public InputParser(ISaleLineFactory saleLineFactory)
        {
            this.saleLineFactory = saleLineFactory;
        }

        public void ProcessInput(string input, ISale sale)
        {
            ISaleLine saleLine;
            ...
            saleLine = saleLineFactory.GetNewSaleLine(quantity, name, price);
            sale.Add(saleLine);
            return;
        }

Much cleaner. We now have no hard dependencies between our classes and we are programming to interfaces not concrete classes.

But we're not done yet. We still have to create a saleLineFactory class, an InputParser class, a Sale object, etc for us to be able to do any real work. Where/when/how do we do all of this?

This is where an Inversion of Control (IoC) container can come in handy. We could just create these objects manually at application startup, but using an IoC container is much easier. An IoC container lets you map interfaces to classes, i.e. to say that InterfaceX is implemented by ClassY and the container will resolve these dependencies for you. Most IoC containers let you do a whole lot more as well (Aspects for example). For this example we'll use the Castle Windsor container to control our object creation for us:

        static void Main(string[] args)
        {
            IWindsorContainer container = new WindsorContainer(new XmlInterpreter());

            IInputControl inputter = container.Resolve<IInputControl>();
            IInputParser parser = container.Resolve<IInputParser>();

            ISale sale;
            string input;

            sale = new Sale();
            input = inputter.GetInput();
            while (!inputter.EOF())
            {
                parser.ProcessInput(input, sale);
                input = inputter.GetInput();
            }

And here's the configuration file:

    <configSections>
        <section name="castle" type="Castle.Windsor.Configuration.AppDomain.CastleSectionHandler, Castle.Windsor" />
    </configSections>

    <castle>
        <components>
            <component id="prompt.input" service="Sales.IInputControl, Sales"
                type="Sales.PromptInputControl, Sales" />
            <component id="input.parser" service="Sales.IInputParser, Sales"
                type="Sales.InputParser, Sales" />
            <component id="salelinefactory" service="Sales.ISaleLineFactory, Sales"
                type="Sales.SaleLineFactory, Sales" />
        </components>
    </castle>

Now the astute among you will immediately ask why I didn't put the Sale or SaleLine objects into the container? The answer is fairly simple - they're not good candidates for objects to place in the container. You want to put things in the container that are service classes, not data classes. The definition of a sale and saleline will be fairly static, plus the use of the ISaleLineFactory lets us control which ISaleLine implementation we want to use. it's also why the factory is one of the classes the IoC container is aware of - a factory class is a service that the application uses.

You might also ask, why am I using sale = new Sale() in my sample code? Well, I could have written an ISaleFactory and used that instead (and in a real world system it's what I'd recommend) but this is a sample app, so I beg your forgiveness :-)

Finally, the really astute amongst you will notice that I never call Resolve for the ISaleLineFactory interface and I don't provide parameters for the IInputParser class when it's resolved. So how does it get into the input parser? Well, the cool thing about the Windsor container is that it will automatically scan the constructors for classes it's aware of, and if that constructor has a dependency (i.e. the ISaleLineFactory interface) then the container will automatically instantiate an object for that interface and pass it through.

This makes managing dependencies and instantiation of classes so much simpler, and in large applications this is invaluable.

Now there is one other side effect to these changes. By having loosely coupled objects writing unit tests is a lot easier now, and they are genuine unit tests, in other words they test one thing and one thing only. Previously I would have had an input parser test as follows:

        [Test]
        public void ParseACorrectLine()
        {
            saleLine = InputParser.ProcessInput("1 book at 12.49");
            Assert.IsNotNull(saleLine);
            Assert.AreEqual("book",saleLine.ProductName);
            Assert.AreEqual(1, saleLine.Quantity);
            Assert.AreEqual(12.49m, saleLine.Price);
        }

But this is effectively an integration test. It's testing that the parser and the saleline classes are both working and working together properly. Not an ideal situation.

But with a loosely coupled design I can now use mocking and write a test as follows:

        [TestMethod]
        public void ParseACorrectLine()
        {
            mocksRepository = new MockRepository();
            saleLineFactory = mocksRepository.DynamicMock<ISaleLineFactory>();
            sale = mocksRepository.DynamicMock<ISale>();
            parser = new InputParser(display, saleLineFactory);
            Expect.Call(saleLineFactory.GetNewSaleLine(1,"book",12.49m)).Return(new SaleLine(1,"book",12.49m));
            mocksRepository.ReplayAll();
            parser.ProcessInput("1 book at 12.49", sale);
            mocksRepository.VerifyAll();
        }

Now my test is checking that the interactions between the InputParser and the other classes are correct, not that the other classes are functioning correctly as well (which is what the first test did). Note that the return of a new SaleLine object is just there to provide a return value for the call to the saleline factory. Not having it would cause the test to fail.

Hopefully this is enough to get you started on improving that crappy code you have to work with every day and making it a more loosely coupled application, and as always I'd love some feedback :-)

Agile and Remuneration

2008-05-07T10:09:00.001+10:00

A thread has sprung up on the scrumdevelopment list about seniority and scrum, and how some people don't like scrum because if it's all about the team then how do individuals get recognized and get treated as more senior than others.

Well firstly, there's a misunderstanding that needs to be overcome here. The agile manifesto states that one of the principles is "Individuals and interactions over processes and tools." Individuals! If you're a member of a scrum team you won't suddenly be seen the same as everyone else yet at the same time having a job title like "super senior developer uber architect" doesn't mean the team will respect you either. As in all team situations individuals need to prove themselves to their team mates through valuable contributions. Respect and leadership will come naturally as a result.

This is all well and good, but how do you then tie this to remuneration, salaries, bonuses, etc. How do you ensure that a team gets rewarded but that individuals who contribute more to the team get rewarded further?

In Rugby Union (the sport when the scrum name came from) they have the concept of a man-of-the-match. After each match a team will succeed or fail, but regardless of the result an individual is voted for as the player who contributed most to the performance of their team. At the end of the season those votes are tallied and a "best and fairest" award is given to the person who contributed most to the overall performance of the team. For software development why not apply the exact same principle to work out who has contributed the most?

The process is straightforward:

At the start of the sprint retrospective (you are doing one, aren't you?) get the team to vote using a 3-2-1 system for the 3 people who contributed most to the performance of the team. Obviously alter this slightly if you have a smaller team. This also has the handy side-effect of getting the team thinking about what happened over the duration of the sprint, which is useful for the retrospective itself.
Don't work out the results straight away. Conduct the sprint retrospective as per usual.
After the retrospective is done, work out the results and tell the team who the MVP or person-of-the-sprint is. If you can, give that person a prize or some other form of recognition on the spot in front of their peers.
At the end of a release cycle, or some other time period of your choosing (maybe based on pay review cycles) tally up the votes over the sprints in that period and you'll have a clear understanding of who the main contributors are. Being peer voted there's no "teachers pet" syndrome to worry about either.

Once you know who the best people on the team are you can provide rewards, recognition, bonuses and/or pay increases accordingly.

Be aware that individuals who do not rate highly in the voting may feel excluded which is why it's important to reward the team as a whole as well. In general try and ensure that team rewards are better than individual rewards otherwise you'll end up with a fractured team.

An Agile Agile Presentation for SBTUG Tomorrow Night

2008-04-29T20:59:00.001+10:00

Unfortunately Mitch has come down ill (get better, mate!) and won't be able to present at SBTUG - The Sydney Business & Technology User Group - tomorrow night. Craig has kindly let me step in and cover for him and I'll be doing my Agile Agile presentation so feel free to come along and be a part of the action.

I hope to see you there, and bring your questions!

P.S. Thanks Craig for the kind words.

The Whitest Beach In Australia

2008-04-26T18:02:00.001+10:00

Another quick holiday update - Wednesday we spent the day in the Cape Le Grande National Park near Esperance in Western Australia, which is apparently home to the whitest beach in Australia at Lucky Bay. After visiting it I can understand why it's so highly rated; and the other beaches in the park aren't that far behind it.

The water is crystal clear, the ocean is a brilliant blue and the sand is so soft and clean it's almost unbelievable. It actually crunches and squeaks underfoot and clumps up like icing sugar, just fantastic! Plus, if you have a 4WD you can drive out along Luck Bay's 3km of shoreline. Oh, and there's almost no-one around - being 750km from Perth certainly helps. Check out these shots for an idea of what I'm talking about.

Here's my foot seen through thigh deep water - you can't do that at a city beach!

And here's a view towards the ocean. Check out the range of blues - simple amazing!

And finally a shot along the shore - as you can see the waves are small enough for me to take the camera into the water without fear of getting wet, and being in a national park there's no development along the coastline. Just Australia at it's best!

In Scarborough Beach

2008-04-14T19:16:00.001+10:00

Holidays are great when you're in a fantastic location. The family and I are currently in Scarborough Beach, Western Australia have a great, relaxing time.

Check out this wonderful sunset. You don't see sunsets like this on the east coast (for obvious reasons)

Away for a Few Weeks

2008-04-11T18:48:00.001+10:00

Just a quick note to let you all know I'll be on leave for a few weeks in Western Australia driving around the southwest corner of this great country of mine. I _might_ do an occasional blog post while I'm away but it's more likely that I'll be relaxing and sleeping instead - you can always keep up with what I'm doing via twitter.

How To Fix Problems With Locked Files During a Web Application Build

2008-04-11T18:47:00.001+10:00

I had a client recently that was having a big problem with builds on dev machines failing due to files being locked and unable to be deleted. The problem irregular and it was causing the developers to get frustrated and waste a lot of time in stopping and restarting Visual Studio to try and release the locks.

The errors the team were seeing were all similar to this:

C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Microsoft.Common.targets : warning MSB3061: Unable to delete file "C:\WebApplication\bin\Web.dll". Access to the path 'C:\WebApplication\bin\Web.dll' is denied.

The application itself was a web application, build on XP machines running VS2005 SP1 and set to use IIS as the default web server (i.e. the application virtual directory was pointed at the source code folder).

The team had already checked that the indexing service was off and that antivirus wasn't locking files. When using the sysinternals utilities to see what other processes were locking those files nothing appeared apart from devenv.exe itself (i.e. visual studio).

The problem as it turned out was in the settings of the virtual directory in IIS6. The virtual directory still had the "index this location" flag turned on. It turned out that IIS was placing a temporary lock on these files, even though the web application wasn't being started (ie it was just a compile, not a debug run). Why this didn't appear in the sysinternals utilities I'm not sure.

Anyway, after turning the "index this location" setting off the file locking problem disappeared and the devs started smiling again. Happy days :-)

Deploying Web Sites using TFS Deployer, PowerShell and FTP

2008-04-07T18:17:00.001+10:00

I had a situation recently where I needed to deploy a web site into a production environment from TFS where the only way we could transfer files was via FTP. In case it's not obvious I didn't want to deploy every time the build ran, but rather, only when the build was declared good.

To do this, I used TFS Deployer (developed by Mitch, Chris, Darren and Geoff of Readify) available from CodePlex. TFS Deployer is a small utility that monitors changes in the build quality of a TFS Team Build and initiates a PowerShell script based on what the change in build quality was. It runs something like this:

A TeamBuild is executed and after completion the build has a quality of "Unexamined"

1. Someone runs some tests, checks the build is OK and decides the build quality can be progressed

2. TFS updates the build quality and raises an alert that the build quality has changed

3. On startup TFS Deployer registers itself as a listener for build quality events and picks up the alert. It examines the old/new qualities and determines if a PowerShell script should be run. If a script needs to be run then PowerShell is started and the script executed.

4. Results from the script are emailed to a specified email address of distribution list.

TFS Deployer Setup

I'll assume that TFS Deployer is installed and set up as per the instructions. To initiate the PowerShell script we set up a deployment mapping as follows:

    <Mapping xmlns=""
                 Computer="MyBuildServer"
                 OriginalQuality="Unexamined"
                 NewQuality="Released to Production"
                 Script="DeployToProd.ps1"
                 NotificationAddress="you.are@here.com" />

That's about it. Pretty tough :-) Now for something a little harder.

Getting Ready for Deployment via FTP

OK, this is where it could get painful. PowerShell doesn't have any native FTP support. So the first option you have would be to use the native .NET FTP classes from but that's a real pain because you effectively have to implement your own FTP client in PowerShell. No thanks!

But don't despair - the open source Indy Project is a project that provides a wrapper and helper functions around for all the FTP calls you might want to make (plus a whole bunch of other low level networking goodness) and they have a .NET version of their library. Grab a copy of that (and the Mono.Security DLL that goes with it) and you'll have all you really need to get going.

I should also mention that because I'm deploying to a production box I'm assuming that IIS is already set up - after all it's very unlikely that we'd be dropping the IIS virtual directory on every deployment. We'll just be dropping our files in over the top of whatever is already there.

The PowerShell Script

Time for the script itself - I'll just take this section at a time. First up, the script is called from TFS Deployer which means we have access to the TFS build data for the build that just had it's quality changed. So we're going to set two variables for the folders where things live. $loc is the root of the build drop location, and $sourcefiles is where the published web site live (ie the stuff we need to deploy).

Set-Location $TfsDeployerBuildData.DropLocation;
$loc = get-location;
set-location "Mixed Platforms\Release\_PublishedWebSites\";
$sourceFiles = get-location;

Opening and Closing FTP Connections

Next we're going to create a few PowerShell script functions to support the opening and closing of our FTP connections.

When we open the connection we're first going to use the .NET assembly loader to bring in the Indy.Sockets library then use the methods in that to make the connection and then return the FTP connection object from the function. The close-method is a bit simpler in that we simply wrap the call to the FTP-disconnect. (yes, it's not really required as a function, but you never know when you might want to add logging, error handling, etc).

function Open-FTPConnection($ftphost, $username, $password) {

[void][Reflection.Assembly]::LoadFrom("C:\path\to\Indy.Sockets.dll")
  $ftp = new-object Indy.Sockets.FTP  
  $ftp.Disconnect()
  $ftp.Host = $ftphost
  $ftp.Username = $username
  $ftp.Password = $password
  $ftp.Connect()
  $ftp.Passive=$true;
  return $ftp
}

function Close-FTPConnection($ftp) {
    $ftp.Disconnect();
}

Oh - For those not familiar with PowerShell syntax the :: operator lets us call static methods on a type and the [] wrap a type identifier.

FTP Miscellaneous Functions

Next we define a few more functions just to wrap up some of the basic FTP calls. Note: Download-FTPFile isn't used - it's just there for your reference.

function Get-FTPCurrentLocation($ftp) {
  return $ftp.RetrieveCurrentDir();
}

function Download-FTPFile($ftp, $sourceFileName, $targetDir) {
    $ftp.Get($sourceFileName, ($targetDir + $sourceFileName), $true, $false);
}

function Upload-FTPFile($ftp, $sourceFileName, $targetFileName) {
    $ftp.Put($sourceFileName, $targetFileName, $false);
}

Nothing overly complex in that.

Getting the Contents of an FTP Folder

Now we get to something a bit more interesting. Here we're creating a function that iterates over the contents of the FTP location and optionally deleting files as it goes. It will return a list of sub-folders in the folder for later use.

The function works as follows:

1. Get the directory listing for the current FTP location.

2. Checks if the current directory is the root folder or not - appends a trailing slash if it isn't.

3. For each item in the folder...

3a. Parses the string to get the file name

3b. Checks if the item is a directory or a file (directories have a "d" in their attributes)

3c. For folders we return the full path to the sub-folder.

3d. For files we check the delete flag and nuke the file if it is set.

You'll notice that the $result.add($name) call is cast to a [void]. If we don't do this then when we return from the function we get 2 sets of files. One from the $result object and one for each file name that was written to the output stream by the $result.add() method call.

function Get-FTPFolders($ftp, [bool]$removeFiles) {
    $ls = new-object System.Collections.Specialized.StringCollection;    
    $result = new-object System.Collections.Specialized.StringCollection;    
    $ftp.List($ls, "", $true);
    $currdir = Get-FTPCurrentLocation($ftp);
    if($currdir -ne "/") {
        $currdir = $currdir + "/";
    }
    foreach ($item in $ls)
    {
        [string[]]$fields=[Regex]::Split($item, " +");
        $startField=8;   #the file/directory name starts after 8 fields
        [string]$name=$currdir;

        #make sure we join up file names that were split (ie ones with spaces)
        for ($field=$startField; $field -lt $fields.Length; $field++)
        {
            if ($field -eq $startField)
            {
              $temp = ""
            } else
            {
              $temp = " "
            }
            $name += $temp + $fields[$field];
        }
        
        if ($item.StartsWith("d")) 
        { #directory
            [void]$result.Add($name);
        } 
        else 
        {
        if ($item.StartsWith("-")) { #files have '-' as first character
            if ($removeFiles)
                {
                $ftp.Delete($name);
                }
            }
        }
    }
    return $result
}

Iterating/Deleting the FTP Folder Structure

Next I have two more methods. One to read through the contents of the FTP folder hierarchy, and one to clear it out. Both methods are roughly the same with only variations for the call to Get-FTPFolders.

function Get-FTPTree($ftp)
{
    $thisdir = Get-FTPCurrentLocation($ftp);
    $thisdir;
    $subfolders = (Get-FTPFolders $ftp $false);
    if ($subfolders -ne $null) {
        foreach ($xitem in $subfolders)
        {
            $ftp.ChangeDir($xitem);
            Get-FTPTree($ftp);
        }
    }
    $ftp.ChangeDir($thisdir);
    return;
}

function Clean-FTPTree($ftp)
{
    $thisdir = Get-FTPCurrentLocation($ftp);
    $thisdir;
    $subfolders = (Get-FTPFolders $ftp $true);
    if ($subfolders -ne $null) {
        foreach ($xitem in $subfolders)
        {
            $ftp.ChangeDir($xitem);
            Clean-FTPTree($ftp);
            $foldername = $xitem.split("/");
            $ftp.ChangeDir($thisdir);
            $ftp.RemoveDir($foldername[$foldername.Count - 1]);
        }
    }
    $ftp.ChangeDir($thisdir);
    return;
}

Note that in the Clean method we delete folders by stripping the folder name off the end of the full path, stepping back up a level and then calling the removedir method. It's not pretty, but it works.

Putting it all Together

Now we have everything in place let's actually do what we need to do. Here's what happens:

1. We make the connection to the FTP server.

2. We clean out the existing FTP tree

3. We get a list of all the files we're going to upload.

4. We copy each file individually to the ftp server, creating folders where required (note the filename processing).

5. Just to make sure things are right - we get the contents of the site so we can check the upload worked.

6. Close the connection - we're done!

#Make a connection

$f = Open-FTPConnection "my.ftp.server.com" "ftp_user_account" "ftp_account_password";

write-output "----CLEANING----";
Clean-FTPTree($f);

write-output "----UPLOADING----";

$localfiles = (get-childitem $sourcefiles -r)
foreach ($localfile in $localfiles) {
  $remfilename = $localfile.FullName.Replace($sourcefiles.ProviderPath, "");
  $remfilename = $remfilename.Replace("\", "/");
  if ($localfile.Attributes -eq "Directory") {
    Write-Output ("Creating " + $localfile.FullName + ":" + $remfilename);
    $f.MakeDir($remfilename);
  }
  else {
   Write-Output ("   Uploading " + $localfile.FullName + ":" + $remfilename);
   upload-ftpfile $f $localfile.FullName $remfilename
   $remfilename;
 }
}

write-output "----VERIFYING----";
Get-FTPTree($f);

Close-FTPConnection $f

One thing you might notice is that we use the ProviderPath property of the $sourcefiles variable. This is because $sourcefiles is a PathInfo object and when TFS Deployer runs the PathInfo will contain a UNC path pointing to a network share - if we just use the Path property then we will get the PowerShell Provider identifier in the string causing the FTP upload to fail.

Hopefully this is a good starting point for you if you are trying to do the same thing.

Deming's Rules of Management and Agile Methods

2008-04-01T08:54:00.001+11:00

I saw a post this morning by David Anderson this morning on how a fault tolerant culture helps create great teams. Admittedly he was talking about the British cycling team's recent gold medal haul but the point is valid regardless. In his post he referred to W. Edwards Deming whom I'd never heard of as a source for the principle of removing fear from a culture. Curious, I flicked over to the wikipedia entry and had a look at what the esteemed but unheard of Deming had to say.

Here's an excerpt from the wikipedia entry:

Deming's 14 points

Deming offered fourteen key principles for management for transforming business effectiveness. The points were first presented in his book Out of the Crisis (p. 23-24)^[19].

Create constancy of purpose toward improvement of product and service, with the aim to become competitive and stay in business, and to provide jobs.
Adopt the new philosophy. We are in a new economic age. Western management must awaken to the challenge, must learn their responsibilities, and take on leadership for change.
Cease dependence on inspection to achieve quality. Eliminate the need for inspection on a mass basis by building quality into the product in the first place.
End the practice of awarding business on the basis of price tag. Instead, minimize total cost. Move towards a single supplier for any one item, on a long-term relationship of loyalty and trust.
Improve constantly and forever the system of production and service, to improve quality and productivity, an thus constantly decrease cost.
Institute training on the job.
Institute leadership (see Point 12 and Ch. 8 of "Out of the Crisis"). The aim of supervision should be to help people and machines and gadgets to do a better job. Supervision of management is in need of overhaul, as well as supervision of production workers.
Drive out fear, so that everyone may work effectively for the company. (See Ch. 3 of "Out of the Crisis")
Break down barriers between departments. People in research, design, sales, and production must work as a team, to foresee problems of production and in use that may be encountered with the product or service.
Eliminate slogans, exhortations, and targets for the work force asking for zero defects and new levels of productivity. Such exhortations only create adversarial relationships, as the bulk of the causes of low quality and low productivity belong to the system and thus lie beyond the power of the work force.
a. Eliminate work standards (quotas) on the factory floor. Substitute leadership.
b. Eliminate management by objective. Eliminate management by numbers, numerical goals. Substitute leadership.
a. Remove barriers that rob the hourly worker of his right to pride of workmanship. The responsibility of supervisors must be changed from sheer numbers to quality.
b. Remove barriers that rob people in management and in engineering of their right to pride of workmanship. This means, inter alia, abolishment of the annual or merit rating and of management by objective (See CH. 3 of "Out of the Crisis").
Institute a vigorous program of education and self-improvement.
Put everybody in the company to work to accomplish the transformation. The transformation is everybody's work.

What struck me as I read this was how much lean and agile processes try to utilise the principles.

Here's my quick take on alignment between Deming's principles and agile methods:

Point 1 - Constancy of purpose. Best described by the agile manifesto. It's a clear and unchanging vision of what agile methods are all about and what they are trying to achieve.

Point 2 - Take on leadership for change. Agile understands and accepts change. Change happens - let's make the most of it and lead the way towards change. Change for the better.

Point 3 - Build quality into the product from the start. Test Driven Development (see video) is a perfect example of this as is the desire to minimise code debt.

Point 4 - Minimize cost. Time is the number one cost in software development. Minimizing wasted effort is the single best way to reduce cost. Minimal code debt also goes a very long way to achieving this

Point 5 - Improve constantly. Agile calls this principle "inspect and adapt". It's embodied in scrum retrospectives and other review processes.

Point 6 - Training on the job. For developers this is encapsulated by pair programming.

Point 7 - Institute leadership. Leadership in an agile project arises from within a team - it is not imposed through management hierarchies (and imposed "leadership" is typically psuedo-leadership - leaders still naturally arise in teams).

Point 8 - Drive out fear. I don't know that this is explicitly addressed by any agile methodology per se but it is something that all good leaders will endeavour to build into their teams.

Point 9 - Break down barriers. Agile methods cry out for openness and collaboration. It's at the heart of what agile is all about. Barriers are impediments and slow down teams.

Point 10 Eliminate slogans, targets, etc. The essence of this is that management can't drive people to work harder to improve because of organisational impediments (the system). In software development "the system" is the organisation and the red-tape and processes that organisation imposes on their staff. Scrum in particular addresses this (indirectly) through the role of Scrum Master - a person responsible for removing impediments to the team. If the organisation is slow the Scrum Master should be addressing that.

Point 11 - Substitute Leadership. See point 7.

Point 12 - Remove barriers. Covered in points 9 & 10.

Point 13 - Self improvement. Once again the inspect and adapt principle at play - good teams should be applying this personally as well as at a team level.

Point 14 - Everyone works to achieve transformation. The is the very essence of an organisation with an agile heart. Everyone working together to achieve an objective and improve what they do.

I think I might have to go and hunt for Deming's book and have a read. I'd be interested in your thoughts as well - especially if you've read Deming's work.

Syntax Highlighting for Powershell in Notepad++

2008-03-29T18:13:00.001+11:00

If you ever find yourself doing any Powershell work (maybe when using TFSDeployer for example) then you'll probably be editing .ps1 files in a text editor (I hope you're not still using notepad!!). My personal tool of choice is Notepad++.

Unfortunately most notepad replacements still aren't aware of the syntax rules for .ps1 files and treat them like plain text. But fret no more - you can now get powershell syntax highlighting in Notepad++. Have a look at http://weblogs.asp.net/jgalloway/archive/2006/11/25/powershell-language-definitions-for-notepad.aspx and download the language definition. It works a treat!

Here’s what it looks like in action:

Nice! Thanks Jon!

As a footnote Steven Nagy had the following to say in a recent conversation we had:

I had a nice play with PowerGui today and think I will keep both handy. PowerGui gives you intellisense and richer syntax highlighting than Notepad++
However you can open 5 ps1 files in NP++ in one click and they all open in ONE window, unlike PowerGui which creates a new instance with each file. Other than that, PowerGui owns. And it costs the same: $0.00

A Tale of Two Companies

2008-03-26T22:30:00.001+11:00

There were once 2 companies who both wanted to develop some custom software to help their businesses enter an evolving market and gain a competitive edge. Both companies made an initial approximation of the costs involved in getting the software developed and both companies set aside a similar sum of money for the project. Neither company had any real appetite or skill for developing the software internally and decided to get external companies to help.

Company A

Company A decided to take the traditional, recommended, "best practice" approach for getting the software developed. It's what they'd always done in the past, it's what their project managers recommended and even though they'd never had a project come in on time before they hoped that if they did things right this time then it would all be OK. They also decided to set aside a portion of their budget just in case there was some overrun in the project.

Because Company A wanted to get the best value for their money they decided to put out a tender. They knew that a tender would give them a result they wanted for the lowest fixed price and they'd get everything they asked for.

They also wanted to get the tender right. They knew how many software projects failed but they figures that that was mainly because requirements in tenders had never been quite right before. If they could get their tender requirements right this time then it would give them their best shot at a successful project and they'd be able to enter this new market with confidence.

Company A decided that the best way to get requirements right was to get in an external expert to help draft and clarify the requirements and to word the tender in the best way possible so that anyone could to clearly understand what it is that they were after.

It takes a week to find the right expert and during the initial meeting the expert listens to the high level overview of the project and estimates about 6 weeks to get the tender ready. The expert then starts talking to the stakeholders in the company and begins to draw up an initial draft of the tender.

The expert uncovered more requirements than were initially explained and, as this was a perfect opportunity, a number of stakeholders added wish list items to the tender to be included as low priority items. After two months the tender draft is ready and placed before the stakeholders for review. Everyone in the company understood why the extra 2 weeks were needed to get the draft completed and thought that it was time well spent, especially as some critical requirements were discovered that would otherwise have been missed. Unfortunately the review also determined that some of the requirements weren't well described and others some seemed to have been misunderstood.

2 weeks later the revised draft was again presented to the stakeholders for review. There were still a few things that weren't quite right and a couple of the stakeholders were unable to attend due to scheduling conflicts but as the tender had now taken a month longer than estimated the remaining stakeholders signed off on it and got the project moving.

The published tender had over 400 requirements and suppliers were given a 4 week window in which to respond. Responses when they arrived were larger in both number and size than expected and the review of the responses took 4 weeks to score and complete before a group of 3 potential suppliers were short listed and allocated a day each to present their proposals, 2 weeks after being notified.

Now after 20 weeks the company was at the point where they were ready for vendor presentations. And they still had no software to show for it. Plus they'd spent 15% of their project budget already.

Company B

Company B took a rather different approach. They'd been burned by tenders and fixed price, fixed scope contracts before and their CIO had read about agile development and some of the success stories that were talked about.

While it was considered a risky approach, the CIO knew that repeating what had failed before in the hope that the result would be different was the very definition of insanity, and recommended that an agile approach be taken and immediately set about locating a number of potential vendors who could deliver a project using an agile methodology. After 2 weeks the potential vendors had been contacted and a preferred vendor chosen. The CIO and the vendor also took an unusual approach to the contract. They agreed that cost would be limited, but that scope would not. A minimum set of functionality would be delivered but beyond that anything could be changed by the company. Company B could terminate the contract at any time of their choosing, as long as they paid out a portion of the remaining contract. The vendor would deliver the highest value software they could using a prioritised list of requirements as determined by Company B. The vendor also stipulated that they wanted to have a single representative of Company B to work with, not a committee. Changes to functionality, priorities and requirements were allowed at any time with no extra cost so long as those things changed were not currently being worked on. If Company B wanted to add new features or change existing ones then that was their right. The vendor also agreed to deliver regular releases of the product so that the company could see what was being built and could optionally put that code into production should they deem it appropriate.

It took a few more weeks to get the legals and the resourcing of people sorted out but even so, the project kicked off 4 weeks after the initial "go" decision.

During the initial week of the project stakeholders from Company B and members of the vendor's project team sat together and worked out a prioritised list of high level requirements. Many of the requirements were still somewhat vague but the vendor wasn't concerned - they would work with Company B's people to clarify those requirements better once they got to them. Each requirement was given a relative effort estimate based on how much effort there might be compared to other requirements in the list, and was also given a simple set of criteria to help determine when a requirement could be considered complete. After some consultation with the vendor the prioritised list of requirements was altered slightly to take into consideration any cross-requirement dependencies and pre-requisites but the end result was an order list of things that Company B wanted in their software.

The very next week development work began.

The vendors team started by taking the 4 first items from the requirement list making a commitment to complete them over the next 3 weeks, saying they would have something finished that they could show the people of Company B when the time was up. However the CIO was a little concerned - 4 items didn't seem like that much to work on. However after the team talked through how much code and database set up needed to be done in order to complete a single item the CIO was willing to go with it. At worst they would waste 3 weeks and if they didn't like what they saw they could always terminate the contract.

Over the next 3 weeks the team set to work. However the teams definition of work was a little unusual, they didn't go and lock themselves away and then turn up 3 weeks later with something to show, they were very visible, spent a lot of time asking people questions, clarifying what they were doing and getting stakeholders together to make sure they understood what needed to be done.Plus there were all these wall charts and sticky notes being stuck up on windows and walls so that anyone could see what was being done by the team and how they were tracking. For the first time people in Company B could see what those mysterious developers were actually doing.

At the end of the third week of development the team got the stakeholders together and showed them what they had done. All 4 items they took on were there and working. No smoke, no mirrors. Plus the people from company B were allowed to play with that version of the software whenever they wanted. Company B was quietly pleased.

The next week the vendors team took on 12 items to deliver in 3 weeks. Now the CIO was thinking that they'd taken on too much. But the same thing happened - lots of communication, lots of conversations and at the end of the next 3 weeks more product was delivered and things were coming together. Company B was starting to take notice.

Shortly after this some of the Company B people were playing with the software and realised that with what they had already it would be a great idea and would really improve things if they could make a few changes to what they had planned. They hadn't thought of them before, but now that they had seen the software and how it was shaping up it just seemed so logical. The vendor was quite happy for the company to add requirements to their list, just as long as the list was re-prioritised and anything they didn't want any more was removed.

This pattern of work for another 4 iterations and at the end of the fourth iteration the CIO and the stakeholders looked at the product and realised that all of their main functionality was already in place. There were still a lot of items on the requirements list, but they were low priority items and probably not worth the development cost, at least not for an initial release.

After 23 weeks the software was put into production.

As everything that was needed was already delivered Company B decided to terminate the contract and saved around 40% of their budget. The money saved then went into further efforts to open up a lead in the emerging market giving the company even more of a competitive edge. As for the vendor they became the sole supplier of development services to Company B from that point on.

Company B was already up and running with their system and starting to make inroads into the new market before Company A even got the development work started on their project. Over the next few years Company B continued to push hard providing their customers with regular updates to their offerings, adapting to the changing state of the market and managed to do so at a pace that no-one else could get close to matching. When Company A finally made their venture into that market it was with a poorly conceived, me-too product that cost way more than budgeted and didn't work anywhere near as well as that of the established players. Company A withdrew from that market not long after.

Before you ask - no, this isn't a story based on any specific companies. But it is a story I'm sure you can relate to as this sort of thing happens all over the world every single day. There are still far too many Company A's in this world and we should do all we can to help turn them into Company B's.

I'd love to hear your comments!

Firefox 3 Beta 4 - A Mini Review

2008-03-12T10:56:00.001+11:00

With IE8 Beta 1 finally getting out the door at Mix08 last week I figured I’d check on the competition and spark up FF3 Beta 4.

Here’s a quick summary of what’s new (from a visible perspective)

1. Speed – it’s sooo much quicker to use. You’ll notice the difference in ajax heavy apps (google reader, etc) as well as in general rendering.

2. Downloads now get virus checked. Nice.

3. New address bar – very funky. Not only address search, but titles as well

4. Most FF2 add-ons don’t work. You’ll need to find updates. Here’s Firebug 1.1 beta (FireFtp has a beta you can get as well)

You can see the parallel loading has increased to 6-7 items instead of 2, which explains some of the speed boost.

5. Bookmarking is a little different – note the tagging. Very cool.

6. IE8 Activities work and are available from http://www.kaply.com/weblog/2008/03/07/microsoft-activities-for-firefox-new-version/
Adding activities from the IE8 activity gallery works, but Mitch’s Wikipedia activity doesn’t due to the use of IE8 specific javascript. Even so...

results in

7. IE8 style Webslices can be got from http://www.glazman.org/weblog/dotclear/index.php?post/2008/03/10/WebSlices-in-Firefox-who-wants-to-try- but they’re still halfbaked (it’s obviously a quick hack and more work is needed)

When you try to mouse over the icon so you can add it, it disappears because you've left the webslice (doh!). I’m sure there’ll be an update soon enough, but in the meantime you can right-click to add a “webchunk” which then looks like this:

Oh- before you make any comments on the content this is all legit ebay search results and all stemmed from the selection on the SMH web site.

8. Oh - it does occasionally crash L But at least the crash reporting is OK, and when you restart you can try and restore your sessions which is OK (unless the offending page always causes the crash).

Overall– a really good experience, but being beta I’d expect that there are still issues with it I haven’t seen in the short time I’ve used it. Is it better than IE8? Some people think so - http://blogs.computerworld.com/battle_of_the_betas_firefox_3_beats_ie8

Improve Your Technical Presentations

2008-03-11T14:05:00.001+11:00

I was watching Scott Hansleman’s ASP.NET MVC presentation this morning and noticed him zooming in on various screen areas and drawing around parts of the screen he was talking about during the demo. “What the hell is he using to do that?” thought I.

Initially I thought it must’ve been a tablet thingy and that if I mentioned it Mitch would get all “nah-ne-nah-ne-nah-nah” on me. But I did a quick bit of digging and learned that it’s actually a utility from the SysInternals team called ZoomIt - http://www.microsoft.com/technet/sysinternals/Miscellaneous/ZoomIt.mspx

I’ve downloaded it and played about with it and think it’s fantastic – it makes showing small fonts and display areas (such as the solution panel in Visual Studio) so much easier and avoids the usual presenters question of "Can you guys up the back see that?". I wish I’d known about this in the past. If you haven’t already seen it, I’d recommend getting it and trying it out.

P.S. Don’t worry about the size – it’s all of 74KB to download.

Castle Windsor

2008-03-10T19:19:00.001+11:00

Spent a couple of hours this afternoon getting to know the Castle Windsor IoC container again.

I ran through the sample as a refresher and then converted a small app I had lying around to use loosely coupled services/components.

Talk about simple, easy and powerful. I've read enough about this thing to know how it works and played with it briefly in the past, but the latest release adds support for generics and makes it so much easier to use than before (not that it was hard).

The effort involved in converting an existing "hard wired" app to be one using loosely coupled services connected at runtime via the plumbing magic of an IoC container is obviously dependent on the current state of the app, but even so switching my app across was only a matter of adding a few lines of code and doing a bit of refactoring.

For instance let's say I have an app that parses some input, I could instantiate parsing and messaging services like this:

IWindsorContainer container = new WindsorContainer();
container.AddComponent<IMessageDisplay, ConsoleMessageDisplay>("console.display");
container.AddComponent<IInputParser, InputParser>("inputparser");

If I wanted my parser to display messages to the user when there was a problem I'd normally have to wire up the connection between the classes myself, but now I don't need to. All I need to do is have a constructor for the parser like so:

private IMessageDisplay display;
public InputParser(IMessageDisplay display)
{
  this.display = display;
}

and the Windsor container takes care of injecting the appropriate IMessageDisplay class for me.

The container looks after the lifestyle (singleton, pooling, instance-per-request, etc) of each service as well and the lifecycle (instantiation and disposal) of each object.

All that nasty plumbing that we typically have to do is removed and our application becomes so much easier to maintain.

Why do we still insist on writing code in such a tightly coupled manner?