First, Why?

We practice continuous integration (CI) at my office. This means we have a server, specifically TeamCity, monitor our source control repository and perform various builds on a clean system (thus avoiding the “it works on my machine!” syndrome). In addition to performing continuous builds (whenever someone checks code in), we also perform full builds at regular intervals throughout the day. We also perform a more exhaustive nightly build, where in addition to compiling our suite, we also deploy the built suite to our lab server for more processing. This additional processing entails running a program to create database schemas on the 3 database platform we support (SQL Server, Oracle, & Access), perform conversions from one version of the suite to another, and validate the results.

Sounds straight-forward enough, right? Well, there’s one really big wrinkle here: our build servers are physical servers located in Texas, our lab servers (app server and two database servers) are virtual machines located in Florida, and between them is, shall I say, a really crappy network connection. How crappy? Well, if we run the database creation and validation app on our lab app server, the process takes about 45 minutes. If we run the creation and validation app on our build servers, the process takes significantly longer (I’ve been told 4-6 hours). That extended length is not acceptable, as it interferes with our ability to create the MSIs that we use the next day for integration testing and QA.

To get around this, our nightly build process stopped at copying a zip archive of the compiled suite to our lab server. We then relied on a couple of scheduled tasks on the lab server to run batch files to unzip the suite and run the database creation and validation app. This worked, but had a major draw back. If the creation and validation script failed, it sent an email. However, only a very small number of the team received this email. Since the CI server had no knowledge of this process, it could not fail the build, and the majority of the team would go on not knowing that the suite was effectively broken in a way that would only be apparent at run-time, and only if we happened to exercise a certain bit of functionality that relied on the database schema being valid.

In other words, the risk of massive wasted team productivity was very high.

Now, How

As I mentioned before, we use TeamCity as our CI server. We build a suite of applications for Windows clients and servers, so we use MSBuild as our build system. MSBuild has a built-in task called Exec, which allows you to execute any command, and if the command returns a non-zero exit code, it is considered to have failed. This is great, except that it executes the command on the current machine (recall that the build server is in Texas, but we need the database app to run on the server in Florida.

PsExec: A False Start

My first thought for remotely executing commands on a Windows server was to use PsExec from the SysInternals Suite. This worked great, as long as I ran the MSBuild script from my local machine. However, as I discovered after combing the SysInternals forums, PsExec does some funky things with standard in (stdin) and standard error (stderr), such that Java processes (which TeamCity is), seem to hang when calling PsExec. That is, PsExec does not appear to return control to the spawning Java process in the manner that it expects. So, even though the remote processes would execute properly, they would not return control to TeamCity, nor would it provide the exit code of the remote process. As a result, not only would the build fail (we configured TeamCity to fail the build if it ran more than 2 hours), but it would fail for the wrong reason, and continue to mask the real reason from the majority of the team.

This would not do, no sir. So I set out in search of an alternative.

RemCom: A New Hope

After much searching, I came across RemCom, “the open source psexec”. RemCom performed the same function as PsExec, but it did so without funky stdin/stderr redirection. So, I added RemCom to the build process, adjusted the build scripts accordingly, and kicked off a new nightly build in TeamCity. The build finished in about 90 minutes, and it succeeded! Except for one problem: I knew there was still a problem with the databases, so I was expecting it to fail. Sure enough, I checked the logs and found that RemCom had provided all the output of the remote commands (something PsExec never did, btw), including printing the exit code returned. However, it seems that exit code was not being bubbled up to RemCom. So, we had a situation where RemCom was telling us the remote command failed, but RemCom itself was returning a successful exit code, and so MSBuild saw this as successful.

Exec Task: The Empire Strikes Back

It was great that I was getting the output of the remote commands, but now I needed to parse it. Unfortunately, the Exec task that comes with MSBuild simply does not provide you with the ability to capture the output and do anything meaningful with it. So, it was time to look elsewhere.

ExecParse: The Return of the Exit Code

After a little bit of Googling, I came across ExecParse:

A custom MSBuild task that inherits from the Exec MSBuild task. The task adds a parameter to allow parsing of the output using regular expressions, and reporting to the MSBuild logger (and consequently the VS.Net IDE).

Bingo!

I then added a reference to the task library in our build script and followed the simple directions to create a configuration with a simple regular expression to grab the exit code of the remote command from RemCom.

<UsingTask AssemblyFile="ExecParse.dll" TaskName="ExecParse.ExecParse" />
<PropertyGroup>
<DeployServerHost>nasrvfl7004</DeployServerHost>
<DeployServerLocalPath />
  <ExecParseConfiguration>
    <Error>
      <Search>Remote command returned (-?[1-9]\d*)</Search>
      <Message>Remote command failed with error code $1</Message>
    </Error>
  </ExecParseConfiguration>
</PropertyGroup>

<Target Name="CreateDbsOnServer">
<Message Text="Creating databases on Server $(DeployServerHost)..." />
<Error Text="Build and Deploy selected, but the DeployServerHost was not not specified" Condition="'$(DeployServerHost)'==''"></Error>
<Error Text="Build and Deploy selected, but the DeployServerLocalPath was not specified" Condition="'$(DeployServerLocalPath)'==''"></Error>
  <Copy SourceFiles="RemCom.exe;CreateDbsOnServer.bat" DestinationFolder="$(DeployLocation)" />
  <ExecParse
      Command="RemCom.exe \\$(DeployServerHost) /d:$(DeployServerLocalPath) $(DeployServerLocalPath)\CreateDbsOnServer.bat $(DeployServerLocalPath)"
      Configuration="$(ExecParseConfiguration)"
      ErrorCausesFail="true" />
</Target>

That, right there, was the secret sauce. Now, we can execute remote tasks from our CI build, include their output in the build log, and fail the build if those commands fail. NICE!

Going forward, we’re going to investigate including the remote commands into the full builds, as they run reasonably quickly enough. We will probably want to investigate adding another build agent, as this would tie up the build server for 90 minutes, thus preventing the continuous builds from running in a timely manner.

Epilogue

Pavel Sher, presumably from JetBrains, contacted me via Twitter and suggested that I try the latest TeamCity EAP. Pavel tells me that PsExec is working now with this version (would be a 5.0 EAP). Though I am not in a position to go upgrading TeamCity right now, I thought I should include this bit of info, in case someone reading this is.

I gave a presentation to my team yesterday morning about Continuous Integration and how I was implementing it with TeamCity. It was intended to be a quick, 5-minute job. It ended up be a half hour discussion, a back-and-forth. My team was engaged, asking questions, and enthusiastic about what I was doing; they understood why this is a good thing. Rock on!

So, I’ve really been able to step into a leadership role this week. My primary focus has been attempting to understanding a troublesome application that the client had never been satisfied with, yet the organization is now dependant upon (Policies and Procedures… BIG deal in Hospital), and guiding a contractor on the changes that need to be made (I guess I’ve been wearing my architect hat). It’s nice that I’ve been able to think of things at a higher level of abstraction and describe this to the programmer to that we write better code. I guess you can say I’m doing some BDD, but without the tests to verify the behavior. I know, I know. Baby steps.

Another major project I’ve been working on involves work by an outside vendor. They keep their code in a Subversion repository. To support my team’s part of the development, I set up a VisualSVN server on our local source-control server. Working with the vendor, we set up a bunch of svn:externals in my repo to pull their stuff in. Why is this important? Well, it was this combination of externals, combined with the fact that they require different authentication than my local workstation credentials, that made setting this project up in TeamCity, um… challenging. I was able to finally get it working with some guidance from Kirill Maximov (I think of the JetBrains team). The confusion was mostly mine. Once I understood that the external files were pushed in from the external VCS, instead of the local VCS root pulling in the externals, I was able to get things working. Yay!

Day 4

Today started off with some fire extinguishing. At my organization, we have an SSL VPN that allows us to customize the login screens with a set of templates. Since the login screens look like they are a part of our public web site, In order to reduce the maintenance overhead, I opted to share the style sheets with the login templates. Because we have to support IE 6 (the organization is still predominantly IE 6), I use conditional comments to server an additional stylesheet to IE 6 only, that overrides the main styles. Well, a software update over the weekend caused the VPN appliance to strip all HTML comments from the templates. This, of course, broke the login pages (in that some elements overlapped the form fields, preventing people, like the Center Director, from logging remotely). We were getting nowhere fast with the vendor, so I changed the markup of the pages to use tables to mimic a 2-column grid. Fire extinguished.

Next, it was off to a meeting with a client. She was overall very happy with the progress made so far, but she had some bugs she wanted to demonstrate to me. I also wanted to talk to her about her expectations for the administration part. I figured that this would be a great opportunity to try out the BDD style of describing a feature that cucumber promotes:

As a  [Role]
I want [Feature]
So that [Benefit]

(or some variant thereof)

Once the client took me through the bugs she found, I walked her through the process of documenting the administration feature. It was great! There was this continual back and forth, and in the end, we both had a better understanding of how the feature should work. I know the process could be improved, but this was once heck of a start! I highly recommend this method to everyone.

During lunch, I decided to try and capitalize on the enthusiasm my team showed for my work so far. One of the topics that came up during the team meeting was a concern that we were still using Visual SourceSafe 6 (yeah, I know), which, as someone pointed out, is about 10 years old. Everyone in attendance was in agreement that we should upgrade. As I was researching upgrading to VSS 2005, I found out that this version keeps the database compatibility with VSS 6, while introducing a new workflow. One of the biggest gripes I have with VSS 6 is the Lock-Modify-Unlock (LMU) mode of operation. Very outdated, and very counter-productive, especially in the following scenario:

Developer A is tasked with a non-trivial implementation of a new feature
Developer B is tasked with fixing a bug, and traces it down to one of the files Developer A is working on.

With the LMU model, either Developer A has to jump through hoops to save their work elsewhere and undo their checkout so Developer B can work, or Developer B has to wait to fix the bug until Developer A finishes their feature some time in the future. In either case, some form of merging will need to take place.

VSS 2005 introduces the Copy-Modify-Merge (CMM) mode of operation more familiar to Subversion users. So, I opened a discussion with my team about enabling this option during the upgrade. There was some initial resistance, based on a perceived notion that forcing developers to talk to each other about the content of the changes to a file during a merge would somehow introduce more bugs into the code. I countered that this communication was desirable, as it forced more discussion of the code and increased the number of developers with knowledge of a given module of code. This, in turn, reduced the risk to the organization of the dreaded “hit by a bus” scenario. Plus, I proposed that we timebox the CMM mode of operation to a month, and analyze the results afterwards. This was enough to get the dissenters on board, and the blessing of my Team Lead and my Manager. FTW!

I’m sure some of you can see where I’m going with this; Ultimately, I aim to get us out of VSS altogether, and into Subversion. Why subversion? Once we’re on the CMM model in VSS, moving to subversion will be far less of a shock than, say, git. Not that git is out of the question in the future, but git hasn’t clicked for me yet (I haven’t used it much yet), and I would need to train the rest of the team on git. That’s more than I can handle right now. Baby steps.

Finally today, I followed the suggestion of TeamCity and migrated from the local HSQL database to a full-blown database server. In my case, this meant SQL Server 2005. Not exactly the smoothest ride with Integrated Windows (formerly NTLM) authentication, by which I mean no one explicitly documented the process end-to end. The key, it turns out, is to add the ntlmauth.dll to the TeamCity bin directory. Of course, after that, be sure that you change the the account that your TeamCity web service runs as to be the same user that you’re going to connect to the database with. Remember, we’re talking Windows Authentication here, NOT SQL Server Authentication.

So, in summary:

• My team is wicked receptive to all the concepts I’ve introduced so far
• My team includes skeptics, but open-minded skeptics willing to compromise
• TeamCity ROCKS because its not tied to a specific VCS, and there are plugins for just about any VCS
• TeamCity also ROCKS because it is helping expose flaws in the structure of our projects that is making things more difficult than it needs to be

My plans for world domination improving my development team are progressing nicely.

Day two on my journey of Continuous Improvement saw me getting one of my team’s projects set up to automatically build through TeamCity. After fixing a couple project reference issues, I was able to get our public website project building. We use the Visual Studio 2008 Web Site model; we do not precompile the project in any way before deploying to production. This caused a problem in TeamCity. This is because the Build target for Web Site Projects doesn’t do anything. Instead I had to set the build target to Publish. This causes the web site to be precompiled. I did this all on my lunch break I’ve recommended to my manager that he include the cost of an enterprise license in his budget for the next fiscal year.

Once I got the automated build working, I was wicked excited. I spoke with a fellow Senior Developer and our Lead Analyst about what I had accomplished. Both were equally excited. they basically had the same thing to say:

We know our current process is broken. I’m glad you’re doing something about it.

Rock on!

I gave my manager a quick update on what I’ve accomplished so far. He was equally excited. Then he asked the million dollar question:

Does what you have done require your team members to change their workflow?

To which I happily replied: No. This was the right answer at this stage of the game. We’re shooting for low-to-no friction to get the ball rolling.

What I meant was: Not Yet!

I’m presenting my work so far to the team tomorrow at our weekly meeting (note to self: include replacement of weekly, hour-long meeting with daily, 15-minute stand ups in future plans).

Wow, is that post title an oxymoron? What was the tag line on my blog again?

Today was the start of my journey on the road of Continuous Improvement. One of the big things I took away from acts_as_conference was “Just F***in Do It”. Or, put a little more eloquently by Corey Haines (who I later found out was paraphrasing Ghandi):

Be the trouble you want to see.

So, as it turns out, I’ve garnered a certain level of respect with my manager. Basically, my manager provided me with an opportunity to plant the seed of the ideas (which, admittedly, are still germinating in me) that I’ve been able to gave name to recently. Guess what. He was totally receptive! Awesome! I mentioned Continuous Integration, gave him a brief schpeil on the potential benefits, and, perhaps the real selling point, a free tool (recommended by Cory Foy) to do it with. “Great!”

Awesome!

Between helping the new contractor get adjusted, dealing with a SharePoint migration issue, and warding off other departments from dumping their vendor issues on my shoulders, I installed Team City on our source control server (did I mention that I hate VSS? Note to self: migrating it to SVN should be in my Continuous Improvement plan). WOW. I mean, WOW! That was dead freaking simple! I struggled with CruiseControl.NET for what must have been a week, and I had no working build system (recall the part about VSS). FAIL! I spent a whopping half an hour setting up a build, with system tray monitor AND Visual Studio integration. PASS! However, the build failed, exposing a flaw in the project file. Developer FAIL!

Poet Mattie J.T. Stepanek once wrote:

Every journey begins with but a small step and every day is a chance for a new, small step in the right direction.

Baby steps, folks, baby steps.