In an agile environment with frequent deployment, broken builds can be a big deal. There are many reasons a build can fail that are outside of the control of an individual developer, but a failure due to failing unit tests is just sloppy: it shows that you didn't even bother to test your code before you submitted it, and you're damn lucky that the unit tests actually caught it because otherwise this un-tested code could soon be in production. Even if it doesn't make it to production, your co-workers are likely to waste time cleaning up the mess.

The first "Fail Sign" in the Six Apart office. Still in prototype mode so not yet hung on a wall, but still showing that at the time one of the builds was was failing. ("ATP" is an internal name for the codebase containing the TypePad webapp and API.)

At Six Apart we used Jenkins to run our unit tests, produce deployment artifacts (in the case of TypePad, a set of RPMs), run unit tests, and finally push the new code to our release staging hosts for some final smoke-testing before the actual release.

TypePad is a codebase with a long history -- the current-generation app and API codebase was started in October 2005. This is an old codebase that has accumulated lots of features over the years, but I'm happy to say that its test coverage is pretty good, with good unit tests having always been part of the culture on the TypePad team.

Everyone fluffs a commit once in a while, and no-one likes to be nagged about it. The fail signs started as a way to draw attention to the frequency of build failures without directly pointing fingers at the culprits. My hope was that this "collective shaming" would be enough to inspire more care without creating a hostile environment.

When Six Apart merged with VideoEgg to form Say Media, we brought the fail signs (and Jenkins) with us. They now advertise failures on a variety of projects, although I think it's fair to say that the effectiveness of the signs at their cultural goal has been impacted by the fact that there are now so many different codebases reporting failure this way; we've become accustomed to something being in a fail state at any given time. Were I to attempt this again I'd try for something a little more local to a given team or group of teams.

The Technology

Given that the Fail Signs are a skunkworks project at best, the setup is pretty quick-and-dirty. Although the details have evolved over time, the basic mechanism is still the same as it was when I built the first prototype in 2009 or so.

Although it's possible to extend Jenkins with plugins, the most expedient way to get a push notification of a build failure out of a stock Jenkins is by email, so the first hop is via a notification set to a special email address. This address delivers to a script on the notification reciever host, which then parses (in the loosest possible sense) the standard Jenkins email notification to determine which project it relates to and whether it's a success or a failure.

This script then connects over HTTP to a simple Sign Controller program, which is responsible for the interface with the LED signs. The reason for this indirection is that the LED signs are controlled via RS232, so we need to have a computer close to each sign to act as the bridge between the network and the simple sign control protocol. In the more elaborate setup we built at Say Media we actually use the standard network/phone wiring as the physical connection for the RS232 link, just patching the other end into a serial port instead of into a network switch.

The code for this setup is on my GitHub account, but please be aware that it's not designed for easy re-use and of course you would need a compatible sign to make full use of it.

The signs themselves are a product of Adaptive Micro Systems. All (or at least most?) of their signs support their Alpha Sign Communications Protocol, which is a pretty high-level protocol for changing the messages and graphics on the signs. Compatible signs often show up on eBay for various prices. I got a pair of smaller signs for my own personal use for about $75 each. The larger ones are often a little more expensive, but if you're patient you might find a good deal.