A build script and associated process of continuous integration (CI) combined are the heartbeat of an application. The build script provides a repeatable way of proving confidence in an application and the continuous integration maintains that confidence over time. The continuous and repeatable nature of these two essential practices enable a highly sought after skill for a software developer: laziness.
Over time the duration of a build script increases as more tests are added or validations are run over the source code. When this happens although a build can have run successfully confidence can be diminished through sidestepping the build. Here I will describe some practices I have found to work for projects I’ve been involved in.
In the following sections, each build is triggered by a successful build of the previous in order to create a pipeline.
Kicking things off
A CI server will usually poll a source control management (SCM) application to see if there are any changes on it and then retrieve the latest code and than run a build script. Some new CI servers can receive a web hook from the SCM to be notified when a repository has changed. The affect is the same in that both should trigger the first build.
Notifications and stopping the line
The strength of CI is enhanced if the team know when something has gone wrong. There are many forms of notifications including email, RSS, custom desktop applications and information radiators like Project Monitor. Whatever you choose, make sure it’s sufficiently annoying as your team should know when something has broken the build. If the team have a ‘stop the line’ mentality, this will also help keep the build going strong and giving confidence back to the team.
The Data Validation build
Applications generally allow validations to be placed within a database or within code, and when the occasion arises when data is changed without going through those validations, the data can be left in an invalid state, e.g. if a migration were run that didn’t go through the validation layer. The data validation build can take a copy of a production dataset (usually cleaned of sensitive data) and for each validation that it cares about, can run that against the dataset. This will show where a fracture has occurred between what the application thinks the data looks like and what the data actually is.
Tests and validations
Source code validations such as code quality metrics and static code analysis usually are quick to run and can be run alongside the tests which generally are hopefully also quick to run. Depending on the size of a test suite, a separate build may be considered for the journey tests. These are the ones that open a browser and generally test drive an application from the very outside of its edges. Everything that runs here should be green all the time, there should be no flakey/flickering tests here. These tests can pollute the confidence of a build by introducing too much noise into the signal that the CI servers are there to give to their teams.
Flakey tests
Flakey tests should be quarantined. These tests are noisy and if left in the main pipeline can diminish the confidence of the build. On a recent project the build took over an hour and would regularly fail due to one or two tests that had issues that we difficult to replicate on our developer machines. These tests would fail the whole build and it got to the point where someone would just trigger a build, or run multiple builds against the same revision to get at least one good build. This was not the intention of the of build. By separating those tests out the majority of the build would go through and then trigger the quarantined tests build. This build had an additional trigger where it would rebuild itself if it failed. This build took minutes instead of an hour to run, so would repeat a few times and then go green. This restored confidence in our build and enabled us to progress quicker.
Having a quarantined test build does not mean these tests are ignored. They are still part of the build pipeline and the final step in the pipeline will not be triggered until this build has gone green. Ultimately, this should not become a dumping ground and a team might even consider some trigger to fail the build automatically if there were over a given number of flakey tests within a run. This could enforce fixing, rewriting or finally deleting these tests if they are truly not giving the project any value.
Deployment
The final steps in the build pipeline is deployment to a staging server and the subject of automation is divided ground. With automation, deploy scripts become hardened and robust as they are tested again and again. Those deploy scripts may also include some sanity checks like loading the homepage and logging into the application to further enhance the feeling that when this build is green, things are good. However, with automation, there is no pair of eyes to make sure it looks good and to ensure that the small part of the application that has been changed is working as expected.
I believe the more automation the better. I want to know that when I make a checkin and that goes all the way through, then I want confidence that it’s done and ready to roll out.
Extra builds for extra confidence
[This section added November 1st, 2012]
The sections above describe a workflow triggered by a change in an applications source code but many applications depend on infrastructure or third party services. An application my be deployed on a given hardware infrastructure which itself is backed by a SCM for it’s configuration. The build pipeline could be triggered when the environment changes to make sure it’s still valid against the changes. It is also likely that at some point the code will change less frequently but still rely on integrations. This is why it is also important to consider having a ‘nightly build’ which tests these integrations. These will ensure the application behaves as expected against software as a service integrations. This build would be triggered at the same time every night, or once a week, just to ensure the stability of the maintained application.
Summary
Being able to trust the build is essential to a healthy project. That is the core reason for the build. The examples above may provide a framework to create or rebuild confidence in a build but these are not the only ways of doing so.
About the Author
