Continuous Integration: Executing Remote Tasks with TeamCity, MSBuild, RemCom, and ExecParse
Posted in Continuous Integration on October 17th, 2009 by Will – 3 CommentsI spent most of last week at work improving our build process, so I thought I’d share with you, and my future self, how I accomplished this.
First, Why?
We practice continuous integration (CI) at my office. This means we have a server, specifically TeamCity, monitor our source control repository and perform various builds on a clean system (thus avoiding the “it works on my machine!” syndrome). In addition to performing continuous builds (whenever someone checks code in), we also perform full builds at regular intervals throughout the day. We also perform a more exhaustive nightly build, where in addition to compiling our suite, we also deploy the built suite to our lab server for more processing. This additional processing entails running a program to create database schemas on the 3 database platform we support (SQL Server, Oracle, & Access), perform conversions from one version of the suite to another, and validate the results.
Sounds straight-forward enough, right? Well, there’s one really big wrinkle here: our build servers are physical servers located in Texas, our lab servers (app server and two database servers) are virtual machines located in Florida, and between them is, shall I say, a really crappy network connection. How crappy? Well, if we run the database creation and validation app on our lab app server, the process takes about 45 minutes. If we run the creation and validation app on our build servers, the process takes significantly longer (I’ve been told 4-6 hours). That extended length is not acceptable, as it interferes with our ability to create the MSIs that we use the next day for integration testing and QA.
To get around this, our nightly build process stopped at copying a zip archive of the compiled suite to our lab server. We then relied on a couple of scheduled tasks on the lab server to run batch files to unzip the suite and run the database creation and validation app. This worked, but had a major draw back. If the creation and validation script failed, it sent an email. However, only a very small number of the team received this email. Since the CI server had no knowledge of this process, it could not fail the build, and the majority of the team would go on not knowing that the suite was effectively broken in a way that would only be apparent at run-time, and only if we happened to exercise a certain bit of functionality that relied on the database schema being valid.
In other words, the risk of massive wasted team productivity was very high.
