Tips on How to Contribute to an Open Source Project

At the Sydney Alt.Net group last month we ran an Open Source Hack Night. The idea was to move from being a consumer of open source to being a contributor, to demystify what’s involved in an open source project, talk through some of the dos and don’ts, and the practical steps of getting started. Then we jumped in and did just exactly that over pizza and drinks. It was a fun night.

This is a recap of what we talked about on the night. I’m not going to walk through the mechanics of how to make a contribution, but will list some tips on how to find and get involved in a project and how to be successful at it.

Find a project. Duh!

More and more projects these days have issues that make for good “on-ramps” into the project. The web site http://up-for-grabs.net is an aggregator site for issues that meet this criteria. Have a browse of the projects on the site and see what interests you. Or just look at a project you use in your normal day to day work and see where/how you might be able to help.

Maintainers are people, just like you.

They’re developers with the same time pressures and insecurities that you have. The project they’re working on was likely started to scratch an itch and then made public in the hope it would help others. Remember to interact and treat these people with the same respect that you yourself want to be treated.

Check the social dynamic

If you see a project you might want to contribute to, look through the issues and pull requests to see how open the maintainers communicate and how open they are to contributions and different ideas. Lurk in the chat room (if they have one) and see what the banter is like. If you don’t like what you see, move along.

Contribution guidelines

Read the contribution guidelines if they exist. And follow them! They might be in a readme.md or a contributors.md or on a wiki. If you ignore them don’t be surprised if your contribution is rejected. Pretty obvious, right

Contribute small changes to start with

It reduces the time you spend before getting feedback from the maintainer(s) and they’re easier for maintainers to absorb. A large contribution that you have put a lot of time into only to have it rejected is extremely discouraging and can you off from ever making future contributions. Large contributions also carry a lot more risk for the project making maintainers leery of spending the time to look through if this is your first contribution. Remember that maintainers are just like you. What would you want to see from a new developer making changes in your work projects? Small, incremental changes or massive, cross cutting changes. It’s the same for open source maintainers. They need time to get to know you.

P.S. Check with your employer about making contributions on work time. Unsurprisingly employers expect you to be building the thing they’re paying you to build. You should also check what the rules are for contributing back code to projects you’re using for work. Some employers still have backwater policies about open source and IP protection.

You don’t need to write code to contribute

Documentation is the one thing open source projects struggle with more than anything else. Maintainers would love it if you made documentation contributions. It’s just as critical to adoption as the code is, if not more so. After all, consider how you feel about using an open source project with no documentation versus one that has great documentation. A lot of projects now use https://readthedocs.org/ to host their documentation with the sources in git repositories. It’s easy to help write documentation.

You can also contribute by responding to questions/issues on the project site, on StackOverflow, or on other forum(s). You could also create screencasts, talk about the project at user groups, blog about it or do anything else you can to raise awareness. These are all useful and valuable contributions. Don’t be constrained by thinking it’s all about code.

A counter-example

Let’s learn from a an example of what not to do, shall we? Have a look at https://github.com/aaronpowell/db.js/pull/112 and you’ll see a number of things that have gone wrong here. To make it easier, let me give you a list:

1. Assuming the maintainer is able to respond immediately to all your requests (12 hours between the posting of the PR and re-asking for it to be merged and then 24 hours until the second request for it to be merged).
2. Making a large contribution that deals with more than just the issue that needs fixing.
3. Changing the development tool chain used by the project.
4. Claiming not to have time to adjust the pull request, but expecting that the maintainer does.
5. Getting angry with the maintainer for asking for changes to make the contribution easier to absorb.

Yeah. Don’t do this, OK? Great! We’re good to go, then.

 

Want more information?

Need to go a little deeper? Want specific “How To” information or other tips? Check out this selection of posts and articles others have written:

• Don’t “Push” Your Pull Requests
• The Social Coding Contract (video)
• The Beginner’s Guide to Contributing to Open Source Projects
• Contributing to Open Source on GitHub
• Getting Started Contributing to Open Source (jQuery foundation)

Most of all, dive in! Have a go! We’d love to see you involved. Good luck!

Side Project: A Distributed Test Runner

I’m working with a customer who, for historical reasons, has a test lab with 20-something test machines of various speed and capacity and automated through code they’ve written themselves over the years. As part of the regular build process they distribute all their automated functional tests across these machines and a test run typically takes somewhere between 2.5 to 3 hours. They want this to be as fast as possible, so to figure out the fastest way they can complete a test run they have been pre-allocating tests to machines based on historical run times.  They also have some tests that can only be run by some machines because of how the machines are configured, and they tie these tests to machines via a test category attribute.

In theory, this approach seems OK; you look at the previous execution times to work out the best distribution for the next test run, but in practice it doesn’t work out like that. Machines aren’t all the same capacity so test times can vary significantly between one machine and another, and there are new tests with no history being added all the time. The end result is some machines in the lab end up being idle for around 30 minutes, meaning test runs take longer than they should.

Now the obvious solution here is to move away from all the pre-allocated, predictive approach to distributing the tests and instead simply put all the tests in a queue. Test machines then grab the next available test from the queue and execute it. This way all machines will be busy up until the point the queue is empty and the final tests are awaiting completion by other machines in the lab.

Microsoft does exactly this with TFS. Machines in a test lab have a test agent installed on them and those agents communicate with a test controller that holds a queue of tests to be executed and farms them out to the agents based on certain criteria.

Borrowing from that concept, I decided to produce a proof of concept that my customer could borrow from and incorporate into their hand coded test lab environment.

That code is now available at https://github.com/rbanks54/DistributedTestRunner and I thought I’d share it in case you were interested.

The architecture is pretty simple.

1. Test Controller

The test controller is a set of REST endpoints (built in ASP.NET Web API) and a rudimentary UI that shows the status of a test run. The controller is a console app, running as a self-hosted OWIN server. No need for IIS here.

To start a test run the end user provides a path to an MSTest based assembly either via the API or the UI. The assembly is then parsed and all the tests placed in queues based on the category attributes, waiting for agents to start requesting tests to run.

2. Test Agents

An agent simply polls the controller for a test to run. Mulitple agents can run at once.

The controller will determine what test is next from the queue and return it to the agent. The agent then kicks off MSTest, passing the test name in as an argument, gathers the test results and sends a success/fail status back to the controller to indicate the test has completed, before then asking for another test to run.

Additionally, since we’re spinning up an instance of MSTest for each test we execute (I know, it’s not efficient) we do a little extra work to merge the individual test result files from each MSTest run into a single TRX file so that when all tests for a test run are completed, we can see the results in a single file.

 

It all works pretty well. Not bad for a small amount of effort!

So, feel free to have a look at the code if you like, and borrow from it as you will. If you like to experiment, feel free to take the code and extend it to make it more interesting and useful. It’s open source! I’d love to see what you do with it!

Just remember it’s a proof of concept at the moment. I made an assumption that the test assembly is in the same folder as the test agent/controller, I didn’t secure the API calls. I didn’t write unit tests (I know; practice what I preach, right?). I don’t send the TRX files back to the controller after the test run completes. I could’ve used SignalR for polling instead of the simple timer loop I used. These are all things you could improve on if you wanted to try your hand at something.

Personally, I found it fun and interesting to go through the process of putting it all together and then walking through the customer through how it works and the approaches I took. Maybe you’ll find it useful too. If not, don’t worry. It was fun to write and, after all, isn’t that why we do the job we do?

Happy coding!

GitViz: Live Git Repository Visualisation

Tatham Oddie, Dietloff Giliomee and I are currently delivering developer uplift training for a large number of developers over multiple courses. Part of this training involves teaching the developers, who are very familiar with centralised version control, how to use Git.

What we have found is that explaining git concepts by drawing pictures on whiteboards and waving our hands around in the air works, but only to a point.

To try and alleviate this problem Tatham put together a great little tool over the past two days that provides a way for people to see what is happening in their git repository as they work, in real time. Now we students can see what git is doing simply by using the commands and visualising the repository changes with GitViz  How cool is that!

Even better, it’s all open source which means you can head over to the GitHub project to grab the code yourself, submit pull requests or simply make suggestions for improvements. We hope you enjoy it!

Underscores in Test Names are a Pain to Type, Right? Not Anymore!

It might be a large assumption given some of the customers I deal with, but I’m going to assume you write unit tests.  I’m also going to assume that when you write tests your test names have underscores separating all the words and making your test names human readable. Something like This_is_a_really_long_test_name_but_thats_ok_because_its_easy_to_read_when_I_use_underscores.

The only (minor) problem I have with this approach is that underscore isn’t the easiest key to type.  I’d rather just hit the space bar and have Visual Studio change that space to an underscore for me – after all, that’s exactly what computers are good for :-)

Here’s the good news, there’s a great little ReSharper macro that has been developed that does just that – it’s available from https://github.com/joaroyen/ReSharperExtensions and all you need to do is create an R# live template that wires the macro up to your test name field in a live template.  For example:

Now when you use the live template to create a test you can simply type the test name with spaces and they’ll be converted into underscores for you automagically.  Fanstastic!

Kudos to Joar Oyen for his great work, and feel free to check out his blog post for how this works under the hood.

Git & TFS Working Together – Version 2

UPDATE: Recent changes mean that the shelveset approach to checking in changes is no longer required. See Git-TFS Recent Improvements for more information

---

I posted a while ago on how to get Git and TFS version control working together, however there are some limitations with that approach that reduces it’s usefulness for some people, specifically that checking in doesn’t deal with check-in policies or being able to associate work items with a changeset.

Well, recently on the Australian Alt.Net mailing list the discussion came up again in reference to getting Mercurial working with TFS, and during the conversation we were pointed to the git-tfs project which I was unaware of before then.  Curiosity appropriately piqued, I went and grabbed the code to see how it worked, and I’m pleased to say that the approach is much better that the previous approach I blogged about, though the issues with solution files still exist, though it’s somewhat alleviated with this approach.

What is This git-tfs Thingy Anyway?

OK, introduction time, the git-tfs project is a git plug-in, much like git-svn, and it defines some git commands that let git work against TFS source control.  The nice thing about the approach it takes is that instead of trying to commit directly to TFS and having to deal with check in policies and all that jazz it creates a shelveset instead. You can then switch to Team Explorer and do an unshelve and check in using the normal TFS practices.  The normal development workflow follows the path shown in the diagram on the left.

Getting Started

First up, you’ll need a version of git installed.  I’ve tested using msysgit and it works well, so grab a copy of that if you haven’t already. Then either download git-tfs or go grab yourself a copy of the source either from Matt Burke’s (@spraints) repository or from my own fork. My fork includes changes to make it work with TFS2010 and alternate user accounts, and hopefully Matt will take those onboard for his next release.

Remember, if you download the source you’ll need the Visual Studio SDK in order to build it.

Also you’ll need to add the folder where the compiled output to your path, so that git knows where to look for the “git tfs” commands you are going to use.

Clone TFS

You should now be able to get the source from TFS via git using the “git tfs clone” command, as shown here

git tfs clone http://tfs2008-vm:8080 $/TeamProject0 local_git_folder

Just pass in the source control folder you want to clone and the local folder you want the repository created in and wait for the code to download.  If you need to supply non-default credentials as I do when working with my virtual machines, then you’ll need to supply the –username option, for example:

git tfs clone --username tfs2008-vm\tfssetup http://tfs2008-vm:8080 $/TeamProject0 local_git_folder

Note that the –username option only works if you grab my fork (at least until the changes are merged back into spraints version).  When you use that option you’ll be prompted with the usual TFS credentials dialog box so you can enter your password every time you do a git tfs command.

Oh! Be aware that the folder you check out your git source to should be different than the folder used for your normal TFS workspace controlled work.

Yes, this means you’ll have 2 copies of the source.  One workspace/tfs controlled copy and one git controlled copy.

What About Those Solution Files?

Yes, I know, those pesky solution files with their inbuilt source control links that force you to connect to TFS when you open the solution.  They’re enough to drive a person to drink!

So, here’s the easy way to deal with it.  After cloning your repository, pull out the network cable from your PC or turn off wireless and then open the solution.  When prompted to go into offline mode, simply say “Yes! Of course I want to go offline you great lump!" and watch the solution open normally.  You can then plug your network cable back in and carry on as if nothing had ever happened.  Visual Studio will keep the solution in offline mode until such time as you tell it otherwise, and considering we’re using git for source control, that means we never will.

Option Two? Create a local branch of your code and remove the source control bindings from the solution in your local branch.  It’s a little more work, and you’ll have some more merging to do between your master and local branches, but it is just that tiny bit safer.

The choice is up to you, though I’d probably lean towards the first option for now.

Shelve Your Changes

OK, so now go make some changes in your git version of the code and commit them to the repository.  Once you’ve finished we can think about pushing them up to the TFS server.

The way this works is that all local changes you make are aggregated into a shelveset and placed on the server.  This way you don’t need to worry about rewriting local history or having multiple shelvesets on the server that you would have to process in chronological order.

Push your changes using “git tfs shelve” as follows:

git tfs shelve "my shelveset name"

Now switch to the TFS controlled version of the project, go to the pending changes window and Unshelve your newly created shelveset.  Check that everything is OK and do your normal TFS check in, with all the normal policies and association with work items applied that you need to.

Fetching Updates

Nice! OK, so now we have our local changes in TFS.  What about the reverse? What if we now want to get updates from TFS down to update our local git repository?

All we need to do is a “git tfs fetch”.  When we do, git-tfs will go off and grab the latest changesets from the server and bring them down as tagged objects.  They won’t get directly applied to the master copy, so it’s up to you to bring those changesets in yourself.

This is where you need to do a local “git merge” to take those tagged objects and merge them into your working copy.  You can do this via the command line or using the Git Gui as shown here

Once the merge is done you should be able to see all the changes from TFS, and visualise it using the git gui tools, for example:

And that’s it, we have a workable approach to using git with TFS.  Obviously there’s room for improvement with the possibility of committing directly to TFS, but for now this approach works well and I think it’s much better than the svn-bridge based approach I was using earlier.

 

Kudos to Matt Burke (@spraints) for the great work in getting this together.  It’s a great solution!

Twitual Studio

On Episode 2 of the Talking Shop Down Under podcast (go subscribe!) I was talking with Matt Hamilton and saying how it would be nice to have a Twitter client inside Visual Studio, given that’s where we spend a large amount of our time.

Well, I decided to not just talk about it, and actually do something about it instead.  So I hereby humbly announce the birth of yet another new twitter client, with the difference being that this one you should be able to use from within Visual Studio 2010.  Welcome Twitual Studio!

I’m rather time poor and my WPF skills are really bad, so I shamelessly borrowed code from Rich Stern’s WPF Tutorial and I’m learning as I do this.  That said, if this is something you want to use for yourself and you want to see something genuinely usable sooner rather than later, then I’d love you to help out.  The project is up on Codeplex and uses Mercurial for source control, so feel free to clone the repository, hack the thing to pieces and send your patches through to me.  I’d love to have them.

For the curious, here’s an early screen shot of what it looks like:

How to get Git and TFS Working Together

Update: There's now a second approach for doing this that you may want to look at

TFS has a lot going for it and from a team perspective it’s a fantastic tool and something I wouldn’t pass up willingly.  That said it is a little lacking in some areas and one of those is the source control story.  For some time now I’ve been toying with writing a Git-TFS bridge but just haven’t had the time to really get stuck into it.  For that reason I decided to see if I can get Git talking to TFS via the TFS SvnBridge utility.  I know what you’re thinking and yes, it’s a 3-headed monster if ever I’ve seen one, but I’m a brave (or foolhardy) adventurer and I don’t mind the occasional challenge.

As a note I’m not going to go into a lot of detail on some of the Git commands.  This is more about how to get Git and TFS working together, not how to use Git.

Getting started

Download the SvnBridge client from http://svnbridge.codeplex.com/.  Extract the SvnBridge.exe from the zip file and run it.  It will appear in your system tray and will show you the base URL to use for your SVN commands when you mouse over it.

To make sure it’s all working try browsing the TFS server via TortoiseSVN (repo-browser). I found this to be a little slow, but it worked fine.  The URL I used in SVN was something like this http://localhost:8080/tfs.server.name/projectName/Trunk

Assuming that’s OK you can then create an empty folder, open a command prompt and set the new folder as your current directory. Now you can clone (i.e. copy) the TFS repository into Git using the command below (as a note I use msysgit for my git work and have the git bin folder in my path).

git svn clone http://localhost:8080/tfs.server.name/projectName/Trunk/SubFolder SubFolder

This admittedly did take quite a while to do.  It was a large source tree and the TFS server is in a remote location but even so, it’s a lot slower than just doing a full get latest from TFS.  Also, be aware that cloning a repository is making a copy of the whole repository, including history. If you just want the latest revisions and you’re not worried about history (it’s in TFS after all) then I’d recommend you use the –r option and supply a recent changeset number (or range).

I also noticed that failing to put the target folder name at the end (even when it’s the same) can result in errors like this so make sure you include it.

Invalid filesystem path syntax: REPORT request failed on '/tfs.server/!svn/vcc/default': Target path does not exist at c:\Program Files (x86)\Git/libexec/git-core/git-svn line 4567

Now go get a coffee and come back.  Once the download completes you should be able to browse your source folder and see everything is there.

We’re all set now, right?

OK, so now we have all the files from TFS locally, and because we used the SvnBridge we don’t have the annoying TFS read-only flag set on our files.  Nice.  We can start working on our project again, right?

Well not quite.  Visual Studio has this nasty “feature” where it puts source control binding information into the solution and project files.  It’s a major pain in the butt and is something you’ll have to work around if you’re in a team environment where not everyone is using this Git/Svn/TFS combo.  We’ll need to manually remove all that binding information from the solution, however we have to think of our team members.  We can’t just edit the solution file directly as other TFS users will/may be relying on it.

Also, before we make this change we’re going to do one other thing.  We’re going to create a local branch and use that as our working folder.

You can either use the Git Gui (see the branch menu) or you can do it via the command line using the git branch or git checkout –b NewBranch commands.  I’ll leave it up to you to choose the method you prefer.

So now we should have both a master and a local branch. Before we proceed just confirm that you are in the correct branch by doing a git branch command (no arguments) at the command line – the current branch should be highlighted.  If you’re using the GUI then the branch name is shown near the top.

Now, to remove the bindings open the solution file in notepad++ (or the text editor of your choice) and find the section that looks like this and simply remove it.

GlobalSection(TeamFoundationVersionControl) = preSolution
    SccNumberOfProjects = 15
    SccEnterpriseProvider = {4CA534B2-18FA-4F9D-95D4-32DDF27D184C}
    SccTeamFoundationServer = http://tfs.server:8080/
    SccLocalPath0 = .
EndGlobalSection

Save the file and then open it with Visual Studio.  When you do, you’ll be told that the source control provider can’t be installed and then presented with a choice.

Since you’re in your local branch you can take either option.  I prefer to choose to work uncontrolled so that visual studio doesn’t remove all the source bindings from the project files (which would break things for others on the team) and it means there’s one less thing to worry about when pushing changes back to TFS.  Regardless, once you make a choice the solution will open as expected and you can make the changes you need to.

P.S. If you haven’t already done so, you may want to create a .gitignore file to ignore those pesky artefacts that get created during development.  There’s an example of one here - http://gist.github.com/233903.  Just put it in the root of your source tree (the folder with the .git subfolder).  You may also want to commit it to your repository so you can track any changes.

Now we’re cooking! Let’s update TFS!

OK so now we’re all set.  Sweet! Go ahead and do your normal development work.  It’s OK.  You’re allowed to :-)

When you’ve made changes you can commit them to Git using either the command line or the Git Gui (which I tend to use) and do any of the other normal things you would do when using Git.

When you’re ready we can now push your changes up to the TFS server.  First we’ll want to push our local branch changes to the master branch (remember to be aware of changes in the solution file).  You may also want to do a git rebase (example) on your local branch to consolidate all your small changes into a single commit before merging to master in preparation for pushing to TFS.

The workflow is basically: git rebase (optional) –> git checkout master  –> git merge –> push to TFS

And the command to commit to TFS is simply

git svn dcommit

Git will push all your local changes to the server and if everything is OK you should be able to switch to team explorer and see the changes in there.  Nice :-)

How do I do a “Get Latest”?

I almost forgot, you will no doubt want to do a “get latest” at some point.  From the command prompt simply do this:

git svn fetch

And it will get the latest changes for you and update your local git repository.  Note that git won’t just get the latest change, it will actually get all the individual changesets from TFS so that the local Git history is up to date.  This is slightly different from TFS where the local workspace is just being brought up to the current changeset.

Pros and Cons?

So, what are the benefits and where do things fall over?

Well, obviously the biggest benefit is that you are now able to use git and work in a truly offline manner from TFS.  You can branch locally and make changes in isolation of others without messing things up for anyone other than yourself, and the speed when working locally is blindingly fast (no over the wire communication with TFS)

On the downside SvnBridge is quite slow and the process to get code in and out of TFS involves more steps than clicking a few buttons in Visual Studio. 

Also, with Git holding all the history, you’ll chew up quite a bit more disk space locally.  I know disk is cheap but in many organisations developers are forced to work with puny boxes and miniscule amounts of space so it may be an issue.

Caveat Emptor:  I haven’t done a lot of work this way as yet and there are many things I haven’t come across as yet (check in policies blocking a commit for instance) so your mileage may vary. You have been warned :-)

If you do try this, I’d be keen to hear how you get along with it.  Good luck!

Raising Events With Rhino Mocks AAA Syntax

There’s various posts on the web showing how to raise events in Rhino Mocks but they typically show you how to do it using the Record/Replay syntax, which I personally find quite awkward.

I was just helping out someone today and showing them how to do it using the Arrange, Act, Assert (AAA) syntax in Rhino Mocks 3.5 and I thought you might be interested in know how to do this as well.

Let’s start with a basic scenario… let’s assume that in our class under test we wanting to check that when we call a method on another object, that it will raise a CancelEvent and we can listen to that event and take action as appropriate.  I like to test behaviour in my classes, so in my test I want to be sure that my class under test correctly subscribes to the event and that it will set the cancel flag correctly..

Here’s some code.

public interface IProcessor
{
 string AMethodThatRaisesAnEvent(int value);
 event EventHandler<CancelEventArgs> AboutToProcess;
}

public class EventListener
{
    IProcessor processor;
    public EventListener(IProcessor processor)
    {
        this.processor = processor;
        this.processor.AboutToProcess += HandleTheEvent;
    }

    public string MakeItHappen()
    {
        return processor.AMethodThatRaisesAnEvent(0);
    }

    public void HandleTheEvent(object sender, CancelEventArgs args)
    {
        if (sender is IProcessor)
        {
            args.Cancel = true;
        }
        return;
    }
}

So as we can see, the constructor takes in the processor object and starts listening for the AboutToProcess event.  When it fires it checks the senders type and and sets the Cancel flag.  It’s a silly thing to do in reality since normally you would check property values on the sender and decide wether to cancel or not, but for the purposes of the post, it will do.

Now let’s write our behaviour test.

[Test]
public void TheEventListenerShouldCauseProcessingToBeCancelled()
{
    var processor = MockRepository.GenerateStub<IProcessor>();
    var args = new CancelEventArgs();
    var listener = new EventListener(processor);
    processor.Stub(p => p.AMethodThatRaisesAnEvent(0)).IgnoreArguments()
        .Do(new Func<int , string>(value =>
                {
                    processor.Raise(x => x.AnEvent += null, processor, args);
                    Assert.IsTrue(args.Cancel);
                    return string.Empty;
                }));
    listener.MakeItHappen();
}

So what’s happening here?

In the first few lines we arrange the objects we want for the test, then we get into the juicy bit.

It may look a little scary but if you break it down into it’s components, it’s actually prettty easy to grok.  First we set up a stub for when the AMethodThatRaisesAnEvent() is called.  We ignore any argument values that may be passed into it and we then supply a method implementation using Rhino’s Do() method.

Inside Do we supply a new function that takes an int as a parameter and returns a string – the same signature as the AMethodThatRaisesAnEvent() method that we are stubbing.  Inside it we Raise() the event we want to fire and have an Assert that will check if the Cancel flag has been set to true.  Note that we also need to supply a return value otherwise the code won’t compile.

Look a little closer at the Raise() method.  You’ll see that you don’t just say what event to raise, you actually supply a lambda that subscribes to the event you want to raise.  It looks a little wierd but it’s a workaround for the language limitations.  The second two parameters are the event sender and eventargs parameters.

Finally we make a call on our class under tests MakeItHappen() method.  There’s no asserts aftrer that because we are only wanting to check the event handling behaviour, not the return value from the mock object.  Also, if you set a breakpoint inside the Do method, you’ll see that the event raising happens after the MakeItHappen() call, even though it appears before it in the source.

And there you have it – raising events from mock objects using Rhino Mocks.

P.S. Some of you will have noticed that I’m using Stubs instead of Mocks.  This is because I’m not asserting anything on the fake object itself and I’m simply stubbing out code.  Please don’t get too hung up on this.  There’s a whole world of pointless arguments about what to call fake objects; stubs, mocks, fakes, whatever.  I certainly don’t care and the code works just the same if you use GenerateMock and processor.Expect() so do what you’re comfortable with and ignore the storm in a teacup argument over what to call them.

How to Make Linq to NHibernate’s Expand() Type Safe

Linq to NHibernate is great.  It makes your queries so much easier to understand since it expresses the intent of what you are doing better than the CreateCriteria API does.

For example compare this statement:

var query = session.CreateCriteria<UserProfile>()
  .Add(Restrictions.Eq("Id", identifier))
  .SetFetchMode("Sites"), FetchMode.Join);
var result = query.UniqueResult<userprofile>();

with this:

var query = from p in session.Linq<UserProfile>().Expand("Sites")
  where p.Identifier.Equals(identifier)
  select p;
var result = query.SingleOrDefault();

To me, the second is much easier to read, especially if you have people who aren’t familiar with NHibernate trying to read your code.  The only problem with both of these queries is the use of those nasty magic strings.  What happens if I refactor my UserProfile class and rename the Sites collection to something like PublicSites?  Obviously my query will no longer work as expected but I’d only find that out when I ran my integration tests (or at run time if I happened to be lazy and don’t write tests).

I want to avoid all that magic string nastiness and lean on the compiler to help me out, so I planned to write some extension methods to improve the situation, but do that and reinvent the wheel? After all, Marcin Budny has done all that wheel-inventing already and made the extension methods available from his blog.  Sweet!

Now I can get rid of that “Sites” magic string and write strongly typed code that looks like this:

var query = from p in session.Linq<UserProfile>().Expand(u => u.Sites)
  where p.Identifier.Equals(identifier)
  select p;
var result = query.FirstOrDefault();

Isn’t that so much nicer :-)  Thanks Marcin for making your code available!

NUnit for Team Build Gets an Update

For those of you using NUnit with TFS you may want to go and visit the NUnit4TeamBuild project site on CodePlex and get the latest release.

It fixes some issues with support NUnit 2.5 and test names longer than 255 characters, but most importantly it removes the need to have MSTest available for publishing the results back to the TFS server through the inclusion of a new utility that makes the appropriate WCF calls directly instead of doing so through the MSTest executable.

Many thanks to Kev Watkins for the contribution! I love seeing open source working so well :-)

For more info on how the whole thing works go and see Kev Watkins blog post or visit the project site.

My Visual Studio Environment (R# and more)

At the most recent Oz Virtual Alt.Net meeting I gave a run through of the various things I have installed on my box for making Visual Studio work the way I want it to, and making myself much more productive in the process.  The recording of that session can be seen either down below or over on the OzAlt.Net blog.

For reference here are the links to the various things I was showing off:

• ReSharper 4.5
• Power Commands (I accidentally used the old name Cool Commands in the session)
• Snippet Designer
• TestDriven.Net
• StyleCop

There’s actually more stuff I have installed, but since I didn’t show them off I won’t list them now.  Maybe in another blog post I’ll go through what those things do.

 

Some Thoughts on Object Databases

I spent a decent amount of time working with PostgreSQL some years ago and thought at the time that the concept of databases that could store objects was fantastic.  None of that object relational impedance mismatch hoo-hah – just save an object, retrieve an object, etc.  Unfortunately reality wasn’t so great - the tooling around reporting and support by other applications was lacking, and that plus various other influences at the time meant I didn’t really get to explore it much further and it eventually dropped off my radar.

This morning I listened to the Alt.Net podcast where Rob Conery is talkng about Object Databases (or post-relational databases as they’re sometimes called).  It’s an interesting podcast and well worth the 30 minutes you’ll need to listen to the whole thing, and it was great to hear that Rob’s sentiments reflected most of mine, though I’m sure he expressed them in a lots gooder way.

It sparked me to have a quick look at where things are at these days in the ODBMS world and to see if the critiques the SQL fans have of them are still valid.  So let me present a very paraphrased version of the wikipedia entry on the Pro’s and Con’s of ODBMS’s

Pros:

• Performance for certain operations smashes relational databases (mainly around CRUD operations)
• You can still add indexes for object properties to improve performance
• Some ODBMS tools now feature SQL language support
• No need to do object to database mapping

Cons:

• General queries can be slower (i.e. selecting data by an entities property values)
• Interoperability
• May require a change in design philosophy.  Need to stop being data centric and start being domain centric in your application design.

OK, so speed can be an issue as well as interoperability.  It’s the second one that is the biggest barrier to adoption and is after all the main reason I didn’t do more with it in the past – however Rob made a comment on the podcast that really stuck; why not run both an ODBMS and and RDBMS side by side?  The two types of database engines really serve different purposes, and people already run multiple databases today with different purposes – e.g. transactional vs reporting databases – so why not do the same with ODBMS and RDBMS engines?

Here’s what the ODBMS.org web site has to say on this idea:

“Thus object databases are increasingly established as a complement to (not a replacement for) relational databases for efficient resolution of the OR mismatch. ODBMSs are flourishing as embeddable persistence solutions in devices, on clients, in packaged software, in real-time control systems, and to power websites.”

Interesting stuff and it certainly gets the mind thinking of possibilities.  I think it might be time to have a revisit of the whole ODBMS scene and see where things are at now.  I’ll probably start with the db4o database engine as it’s an open source native .NET engine with a strong developer community, it supports linq for querying (codeproject sample) and weighs in at a very light 13.5 MB.  That’s good enough for me :-)

Reactions to Oxite

So you might have heard that Oxite was released recently.  You know the one – it’s the all new, all shiny, singing and dancing blog engine that is meant to show off how great ASP.NET MVC apps should be built.  The system used to run the MIX web site.  Yeah, that’s the one.  And it’s meant to be a great example of good architecture, appropriate use of design patterns and clean code as well, isn’t it?

Hmmm.  It appears not.  Check out this review by Karl…

http://codebetter.com/blogs/karlseguin/archive/2008/12/15/oxite-oh-dear-lord-why.aspx

It’s quotes like this that worry me: “Actually, I seriously think I'm missing a project, because there's no way the team built all this functionality and only 51 tests - apologies if I'm missing a part of the project” or this comment from a review on the project’s Codeplex site - “You have three total test classes, and one test class that is 1400 lines long.”  Whoa!  Shouldn’t testing and quality be a prime concern of a well written project?  I would have thought so.

All I can say at the moment is “Ouch!”  Now in all fairness I haven’t seen the code myself yet so I’m going to download it and have a look at the code myself and make up my own mind.  You should too, and when you do see if you can spot why some people are ripping into it as (yet another) really bad Microsoft example of how to develop applications.

TFS Build Monitor v0.3

For those of you who are interested in having your very own continuous integration build light for TFS or want to be able to tweet build status information automatically, a new version of the TFS Build Monitor (formerly the Twitter Build Publisher) is now available from Codeplex at http://www.codeplex.com/BuildMonitor

The changes in v0.3 are that the application is now a WPF application instead of a console app and it can be minimized to the system tray, thus keeping it out of your way when you're doing real work.

There's also a few minor changes related to handling errors when the Delcom device isn't attached or when the TFS server can't be reached, etc.

If you've got any problems - submit an issue on the codeplex site.

Enjoy!

A Quick Play With IBatis.NET

You may remember a few posts back that I was working with a team trying to use a pure stored procedure approach to access a database, and trying to do so using an OR/M. One of the commenters on the post mentioned iBATIS. Now I always thought iBATIS was a competitor to NHibernate, Linq2SQL, et al. but it's actually a different approach where instead of being a fully blown OR/M it just does simple data mapping of business objects to and from SQL statements and nothing more than that.

So I decided to have a bit of a play with it and I must say it looks pretty good. Here's a quick bit of sample code I knocked up to see how it works.

Assume I have the following sproc:

CREATE PROCEDURE InsertSaleHeader
   @Tax money,
   @TotalValue money,
   @SaleNumber int output
AS
BEGIN
    SET NOCOUNT ON;

   INSERT INTO [dbo].[SaleHeader]
          ([Tax]
          ,[TotalValue]
          ,[ModifiedDate])
    VALUES
           (@Tax,@TotalValue,GETDATE())
    SET @SaleNumber = scope_identity()
   
END

It's just a basic insert sproc that updates a modified date on insert and also returns a new identity value for the record we just added using an out parameter. This is the sort of thing that was a real pain to so with many of the full featured OR/M's.

I then wrote an integration test (p.s. please forgive my incorrect casing in places):

using IBatisNet.DataMapper;
using SalesDAL.ibatis;

namespace SalesIntegrationTests
{
   [TestClass]
   public class ibatisSaleRepositoryTests
    {
       [TestMethod]
       public void CreateSaleAndGetSalesUsingIBatis()
       {
           ISqlMapper mapper = Mapper.Instance();
           ISaleRepository repository = new ibatisSaleRepository(mapper);
           ISale sale = new SalesTax.Sale();
           sale.Add(new SalesTax.SaleLine(1, "imported box of chocolates", 10.00m, true));
           bool result = repository.CreateSale(sale);
           Assert.IsTrue(result);
       }
   }
}

This just creates an instance of a sale object and sends it to a sale repository class. The code for the ibatisSaleRepository is shown below.

namespace SalesDAL.ibatis
{
   public class ibatisSaleRepository : ISaleRepository
    {
       private ISqlMapper mapper;

       public ibatisSaleRepository(ISqlMapper mapper)
       {
           this.mapper = mapper;
       }

       public bool CreateSale(ISale sale)
       {
           mapper.Insert("InsertSaleHeader", sale);
           LastSaleId = sale.SaleNumber;
           return true;
       }

       public int LastSaleId
       {
           get;
           private set;
       }

As you can see the only actual work that happens is in the CreateSale() method where the IBatis Insert() method is called, passing in the object to be saved (note that I’m only saving the sale header here – we could extend this to save the sale lines easily enough).

And that’s it for the code. Nothing too complex at all. The rest of the work is done through the config files that IBatis loads during the Mapper.Instance() method call in the unit test. When it's initialised iBATIS loads up the SqlMap.config file and then processes it any other config files that are reference by it to create the mapping behaviours it needs to know.

The IBatis SqlMap.config file I used is shown here:

<?xml version="1.0" encoding="utf-8" ?>
<sqlMapConfig xmlns="http://ibatis.apache.org/dataMapper" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >
   <settings>
       <setting useStatementNamespaces="false"/>
   </settings>
   <providers embedded="providers.config, SalesIntegrationTests" />
   <database>
       <provider name="sqlServer2.0"/>
       <dataSource name="SalesData" connectionString="Data Source=.;Initial Catalog=SalesDatabase;Integrated Security=True"/>
   </database>
   <sqlMaps>
       <sqlMap embedded="ibatis.SalesMap.xml, SalesDAL"/>
   </sqlMaps>
</sqlMapConfig>

And finally the SalesMap.xml file is as follows:

<?xml version="1.0" encoding="UTF-8" ?>
<sqlMap namespace="SalesDAL" xmlns="http://ibatis.apache.org/mapping" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >
   <alias>
       <typeAlias alias="SaleHeader" type="SalesInterfaces.ISale, SalesInterfaces" />
   </alias>
   <statements>
       <procedure id="InsertSaleHeader" parameterMap="InsertSaleHeader-Params">
            dbo.InsertSaleHeader
        </procedure>
   </statements>
   <parameterMaps>
       <parameterMap id="InsertSaleHeader-Params" class="SaleHeader">
           <parameter property="Tax" />
           <parameter property="TotalValue" />
           <parameter property="SaleNumber" direction="Output" column="SaleNumber" />
       </parameterMap>
   </parameterMaps>
</sqlMap>

I won't go into all the details of the configuration, but as you can see it's not onerous in any way. In fact I found that overall iBATIS it’s a lot simpler to use than NHibernate for this type of operation and it's much easier and faster using it than doing the alternative and writing ADO.NET by hand.

Oh, I should also mention that there is a Castle Windsor facility that loads iBatis up for you so you don’t have to maintain dependencies throughout your application.

So when is using IBatis a good choice? I'll let the IBatis team say it in their own words (taken from the documentation):

So, how do you decide whether to OR/M or to DataMap? As always, the best advice is to implement a representative part of your project using either approach, and then decide. But, in general, OR/M is a good thing when you

1. Have complete control over your database implementation
2. Do not have a Database Administrator or SQL guru on the team
3. Need to model the problem domain outside the database as an object graph.

Likewise, the best time to use a Data Mapper, like iBATIS, is when:

1. You do not have complete control over the database implementation, or want to continue to access a legacy database as it is being refactored.
2. You have database administrators or SQL gurus on the team.
3. The database is being used to model the problem domain, and the application's primary role is to help the client use the database model.

In the end, you have to decide what's best for your project. If a OR/M tool works better for you, that's great! If your next project has different needs, then we hope you give iBATIS another look. If iBATIS works for you now: Excellent!

Using Mind Mapping to Capture User Stories

I've been doing a fair amount of requirements gathering lately across a number of projects that I've been working on. At the start of each project I needed some way to get a handle on the size of the project and I wanted to do so in a way that improved customer collaboration while helping me structure what the project was all about..

Enter user stories.  For those who don't know - user stories are requirements written in the form of "as a <role> I want <something> so I can <reason>".  They're short one sentence structures that act as a mechanism for drawing out not only what a system needs to do, but also why.  The why often being the thing that is overlooked in traditional requirements documentation.  Stories are also of great use when trying to compile a product backlog and prioritise the order in which work should be carried out.

Now in times past I would've sat down with the customer, opened an Excel sheet or One Note section, and started writing stories right there and then. On the screen.  In front of them.  In one great big long list.

And this works really well.

The customer gets involved in the story writing, in the flow of developing stories and describing roles.  It's engaging and fun and the customers love the involvement.  Now here's the wrinkle - we like to write stories in groups of related items so it's easier to keep track of what our thinking is, but unfortunately people aren't linear creatures.  We tend to jump around, go off on sidetracks and tangents, work our way around a problem, attack it from different angles and generally approach things in a somewhat random but related manner.

When you try recording stories in an excel sheet and you get more than a few hundred of them it starts to get really hard to figure out what stories relate to which parts of a system.  The customer asks "can we go back to where we were talking about thing X? I want to add a few related stories because I've forgotten some stuff".  So we look at the list, realise we can't remember where those other stories were and start searching for keywords until we find the area we want.  Over time it gets kind of confusing.  Also, unless you tag your stories with a functional area they relate to, when you come to collate stories later on you can waste a lot of time and gets things mixed up very easily.  Didn't we cover this already? What should this be related to? etc.

Enter Mind Mapping

Mind mapping, for those who don't know, is a technique where you start by drawing up a few basic ideas and then expanding on those ideas as you think further about them, creating other sub-ideas and so forth.  When you find related ideas you link them together or branch off ideas based on a core theme or concept.  Over time you end up with a whole raft of ideas that are all interrelated and linked in some way.  Mind maps are a great tool for visualising this approach.

So these days when I'm gathering requirements I use this technique but instead of starting with a core idea I start with a placeholder ("the system") and then branch off into the things that people want the system to do. 

Normally people start off with some very high level concepts, and then you start to flesh out those ideas eventually working down into the details. When there is enough detail to write stories in then I start adding the stories directly to the mind map.

Here's an example:

Where this gets useful is in the way people think - you can now visually jump around the mind map and easily navigate to where concepts and functionality are located.  Also, you end up talking about things from the customers view - it's their mind map you are helping to draw up.

If you keep at this for a while you can create some fairly large and complex requirements maps - for example the one pictured below has over 1,200 nodes on it yet it's still easy to navigate and find items plus the customer is able to understand what it is that they are asking from the system without getting lost in the details.  I find it hard to do that effectively in an excel sheet, or with story cards alone, so I shudder to think what a struggle it is for my customers.

 

Don't Forget

Don't confuse things here. A mind map is not the ONLY thing I use for requirements.  I still work with user stories, I still do sizing (though often on the mind map itself), I'll still record stories in excel or TFS or <tool of choice> and I still prioritise them with the customer.

The mind map is just a tool to assist in the effective gathering of requirements, and it helps my customers think through what they want more clearly.  This helps reduce the number of missed requirements and improves understanding between us and I think that's a good thing.

Mind mapping is just one more tool for your agile development toolbox. A useful tool, but not at the exclusion of anything else.

P.S. The software I've used here is FreeMind.  It's a little more structured than some of the other mind mapping tools out there, which I find useful when working through requirements and wish list items.

Rhino Mocks 3.5 Presentation at ALT.NET

Last night I did a presentation on Rhino Mocks 3.5 at the Sydney ALT.NET user group.  I demoed the new AAA syntax (arrange, act, assert) as well as doing a run through of some of the more advanced usages of Rhino such as using mocks to test events and event handlers, the ability to throw exceptions to see how your class handles them and the use of Rhino from VB.NET.

A big thank you to everyone who came and made it such a great evening.

P.S. The demo project I used last night is available for download here.

The Twitter Build Publisher is now up on CodePlex

I posted previously about the Twitter Build Publisher (aka the BuildTweeter) - a little utility that lets you publish build results and quality change events from Team Foundation Server and Team Build to Twitter.

Well I've finally managed to find enough time to put the project up on codeplex for public consumption.  If you want to grab a copy of it, mosey on over and download a copy now from http://www.codeplex.com/BuildTweeter/ and feel free to contribute.

Clone Detective for Visual Studio 2008

I just ran across a new project on CodePlex today called Clone Detective for Visual Studio from Immo Landwerth (also responsible for NQuery) and I've got to say that it's really quite cool.  I've blogged previously about detecting code clones with CCFinder but CCFinder is an external tool and a bit cumbersome to set up and use.

Clone Detective for VS may not be quite as powerful but it is incorporated into VS and is so much easier to use that it will probably become my tool of choice as a result.  It's so much better if you can detect your copy & paste development efforts in the same tool that you can fix them in.

Here's a quick screenshot of what it looks like:

We get 3 new windows we can look at (available from the View menu).

The Clone Explorer is where you initiate clone detection, and see results.  E.g. the AlternateTaxRules.cs file has 2 cloned sections.

The Clone Intersections looks at a file and shows you all the other files that share clones with it.  The different colours represent the different clones sections.

The Clone Results shows you an individual cloned piece of code and where it resides throughout the application (i.e. all the different files where  that same code occurs).

When you look at a file with clones you'll see a purple line next to the cloned lines (shown in the image).  By default visual studio just shows a single source file in the main window, however by dragging the tab with the filename in it to the right you can get to a side-by-side view which makes it so much easier to compare the clones before deciding it you should refactor or not.

From first impressions it looks to be a handy little tool and definitely one to keep in my VS toolbox.

Announcing NUnit for Team Build (TFS 2008)

I'm pleased to say that I've just added some more noise to cyberspace and published NUnit for Team Build on CodePlex.  This is the project where the scripts I used for merging NUnit results into Team Build will live.

I hope you find it useful.