Friday, February 17, 2012

Inversion of Control for Continuous Integration

Problem Description

Our build structure is pretty stable, but the exact content of the steps varies as we discover more smoke tests that we'd like to add to, or when we rearrange the location of these checks.

The CI servers I've used made this a rather cumbersome process:

  • First, I have to leave my development environment to go to the build servers configuration of choice - most of the time it is a web interface, and for some it is a config file

  • I have to point and click, and if it's a shell script, I have to make my modifications without syntax highlighting (for the config files usually take the shell command to execute as a string, so no syntax highlighting)

  • If it's a web interface, I have (or had) no versioning/backup/diff support for my changes (config files are better in this aspect).

  • If it's a config file, then I need to get it to the build server (we version control our config files), so that's at least one more command

  • I need to save my changes, and run a whole build to see whether my changes worked, which is a rather costly thing.

  • Most places have only one build server, so when I'm changing the step, I either edit the real job (bad idea) or make a copy of it, edit it, and then integrate it back to the real job. Of course, integrating back means: copy and paste.

  • If the build failed, I need to go back to the point and click and no syntax highlighting step to fix the failures

  • Last, but not least, with web interfaces, concurrent modifications of a build step lead to nasty surprises!

Normal development workflow

  • I have an idea what I want to do

  • I write the tests and code to make it happen

  • I run the relevant tests and repeat until it's working

  • I check for source control updates

  • I run the pre-commit test suite (for dvcs people: pre-push)

  • Once all tests pass I commit, and move on to the next problem

Quite a contrast, isn't it? And even the concurrent editing problem is solved!

Quick'n'Dirty Inversion of Control for builds

Disclaimer: the solution described below is a really basic, low tech, proof of concept implementation.

Since most build servers at the end of the day

  • invoke a shell command

  • and interpret exit codes, stdout, stderr, and/or log files

we defined the basic steps (update from version control, initialize database, run tests, run checks, generate documentation, notify) using the standard build server configuration, but the non-built in steps (all, except the version control update and the notification) are defined to invoke a shell script that resides in the project's own repository (e.g.: under bin/ci/oncommit/ These shell scripts' results can be interpreted by the standard ways CI servers are familiar with - exceptions and stack traces, (unit)test output, and exit codes.


  • adding an extra smoke test doesn't require me to break my flow, and I can more easily test my changes locally and integrating it back into the main build means just committing it to the repository, and the next build will already pick this up

  • I can run the same checks locally if I would like to

  • if I were to support a bigger team/organization with their builds, this would make it rather easy to maintain a standard build across teams, yet allow each of them to customize their builds as they see it fit

  • if I were to evaluate a new build server product, I could easily and automatically see how it would work under production load, just by:

    • creating a single parameterized build (checkout directory, source code repository)

    • defining the schedule for each build I have

    • and then replaying the past few weeks/months load - OK, I still would need to write the script that would queue the builds for the replay, but it still is more effective than to run the product only with a small pilot group and then see it crash under production load

Shortcomings, Possible Improvements

As said, the above is a basic implementation, but has served a successful proof of concept for us. However, our builds are simple:

  • no dependencies between the build steps, it is simply chronological

  • no inter-project dependencies, such as component build hierarchy (if the server component is built successfully, rerun the UI component's integration tests in case the server's API changed, etc.)

  • the tests are executed in a single thread and process, on a single machine - no parallelization or sharding

All of the above shortcomings could be addressed by writing a build server specific interpreter that would read our declarative build config file (map steps to scripts, define step/build dependencies/workflows), and would redefine the build's definition on the server. By creating a standard build definition format, we could just as easily move our builds between different servers as we can currently do with blogs - pity Google is not a player in the CI space, so the Data Liberation Front cannot help :).


Does this idea make sense for you? Does such a solution already exist? Or are the required building blocks available? Let me know in the comments!

Friday, February 3, 2012

There Is More To Clean Code Than Clean Code

A post written by Uncle Bob in January (I'm behind my reading list) offended me. I absolutely agree with Uncle Bob's analysis regarding the code itself, and I also prefer the refactored version, but I have a problem with insulting the programmer(s) reluctant to appreciate the change.

We write code in programming languages, and there are different levels of proficiency in a language.

As I'm currently learning a new spoken language, I'm painfully aware of this - initially I probably sounded like a caveman. The first impression you get about me is totally different depending on the language I speak - but I am the same person!

The learning curve of a language is not smooth - the steepness between consecutive levels of proficiency is different. Going from not speaking any German to speaking A1 (tourist) level was easy, getting the basic grammar required for the low intermediate (B1) level wasn't too bad, but to get my German to the level where my English is will take more effort than the sum of all my previous investments1.

Since it is my third foreign language I'm learning, I have no difficulty accepting that the level I think I speak is higher than the level I actually speak. Because of that, whenever someone rephrases my sentences in proper German 2, I start from the assumption that likely their version is better, even if I don't understand first why - and I take the effort to understand their reasoning 3. I do that despite that I was of course convinced that when I spoke, I expressed my thoughts in the best possible way.

However, I don't have much at stake - no ego to hurt, no reputation to loose, and the roles are clear: I'm the beginner, and the people around me are the more experienced ones. In a software team, the roles might not be so clear - I had told colleagues almost twice my age how they should write code after only a few weeks of working there. Bad idea. Since then, I have learned not to start improving the coding style of a team by rewriting the code they have written, and showing off how much better my version is. Rather, I wait until a situation arises when they don't mind having me demonstrate some code improvements. I demo it, and explain why I do it that way. In my experience, the second approach is more effective, though it doesn't have that instant satisfaction and relief the first provides.

As the joke goes, you need only one psychologist to change a light bulb, but the light bulb has to want the change real bad.

Driving Technical Change is hard, because it requires a mental/cultural change, and that change has to come from the inside - but can be catalyzed from the outside of course4. But just forcing practices or ways of working on unwilling recipients generates resistance (e.g.: the story of the EU technocrat appointed to recalculate the 2009 Greek budget).

I would like to see more public code reviews and public refactorings (e.g.: Andrew Parker, GeePawHill), but I would like to see less public judgement passing on people at the lower proficiency levels of programming.

1 there is a great Hanselminutes episode on learning a foreign language if interested. Beware, it may contain programming!

2 German readers might disagree, since most Germans I meet speak Frankish :)

3 Which of course, is sometimes harder for natives to properly explain than for novices to ask questions pointing out the seeming irregularities of the grammar

4 And we won't always be able to foster change in all environments (note: this does not mean the others are at fault for not changing!). The same programmer can be highly productive in one team, and be the one slowing down another team. There is nothing wrong with changing jobs after realizing we are a net loss to a given team.