Data Migrations As Acceptance Tests

While I have previously said that on migration projects both verification and regression tests are important, does it mean that the two should be separate? Like first, let's migrate the data, and then we'll rewrite the functionality, right? Or let's do it the other way around - we'll talk with the customer, incrementally figure out their requirements, deliver the software (with a proper regression test suite) that satisfies them, and then we migrate. Both approaches have problems:

• customers want to use the software with their current, real data - having only the data and no application to use it with is no value to them. Neither is having only an application with no data in it


• real data has lots of surprising scenarios that the domain expert might have forgotten to specify (see caveats though)


• requirements are not static, and new ones will be introduced during the development process, that inevitably will cause the new application's models to change, which means that the migration has a moving target it needs to migrate to.

Doing them in parallel

If the data source is organized chronologically (order by date in the migration script), and organized in a format that resembles what the system's end users will enter into the system, then we can use the new application's outmost automatable entry point (Selenium, HTTP POST, a module's public API) to enter this data during the migration from the old system to the new.

Why

While a clear disadvantage of this approach is speed of the migration - it will be likely slower than an INSERT INTO new_db.new_table select .... FROM old_db.old_table join ... where .... statement, but in the case of non-trivial migrations it will likely compensate for the slowness, because:

• changes to the new system's code/data structure become a problem localized to the new application code - no headaches to update the migration scripts in addition to the code


• when the client requests the demo deployment to be in sync with the old system, the code is ready (spare for the part to figure out which records have changed)


• the legacy data edge cases provides focus - no need to invent corner cases, for there will be enough in the data


• likely there will be releasable functionality sooner than with either of the above approaches

How

First, create the acceptance tests for the migration:

• Pick the data to be migrated


• find the view in the original system that displays this data to the users and find a way to extract the data from there


• define the equivalent view in the new system (it's about end to end, user visible features!)


• write the verification script that compares the above two (be sure to list the offending records in the failure message!)


• define the data entry point in the new system


• write the migration script - extract from the old system, transform if needed (only to match the entry points expectations of the format - no quality verification as in classic ETL!), then send it into the new system (using the above defined entry point)

At this point both the new view, and the data entry points are empty. From here on, the TDD cycle becomes a nested loop

• run the migration script. See which record it failed for


• analyze the failing acceptance test, and find the missing features for it


• (A)TDD the missing features


• run the migration script to restart the cycle

Caveats

While the existing data makes one focus on the real edge cases instead of the imagined one, beware - not everything has to (or can be) migrated - for instance, in a payment system, the system used to accept many currencies in the past, but now only €. IN this case, possibly the currency exchange handling logic could be dropped in the new system (and just to store the currency in a char field for the old ones); or in some other domains, maybe only the last ten years' data is needed. However, this should be a business decision, not a decision for a developer!

Source Data Quality is often a problem, one that will likely cause issues. If data needs to be fixed (as above, ask the stakeholders!), it should stay out from your application's code, and be in the Transform part of the migration script.

On Grassroots/Peer TDD Introduction

While I'm no trainer/coach, I've been involved in spreading TDD among my peers, with varying extent of success. Recently someone asked for advice on how they should go about it, and in the spirit of Seth Godin, instead of responding in a private email, I wrote this post, a mix of experience and hindsight ideas.

So, we are one (or two) developers in the team/company, already sold on TDD, with some unit testing experience. The peers are willing to listen to the concept, and management, while not willing to invest in formal training, is ok giving a chance, as long as it doesn't hurt the projects' performance. How do we go about it?

Don't introduce TDD on the project

It's common that teams learn/practice new skills on live projects after a basic overview/training. While this might work, I would not recommend this approach for any grassroots initiative, because the chances of failure are significant, and one bad experience with a practice/technology can doom any further attempts to introduce it later, even if not the practice was at fault, only was applied without enough practice. A project manager friend of mine recently complained to me that since the team that worked on a massive legacy project started doing TDD, tasks that used to take hours begun to take days, and clearly, he wasn't happy about that. It turned out that the problem wasn't due to TDD, but lack of discipline (developers went overboard with refactoring, touching parts of the codebase not even remotely connected to the task they've been working on). Despite this, he could have easily come to the conclusion that TDD equals to dropped productivity, without much benefit (bug reports kept coming in, even if for different features - we are talking about legacy code!). Any new skill has a learning curve, you will be slowed down, and you won't get it perfect the first time - don't give a chance for the others to associate failures with the new concept. If you are like me and would need a more concrete example about refactoring just enough, I suggest you read Ron Jeffries' Extreme Programming Adventures in C# book, and on advice for dealing with legacy code, the undisputed classic book is Michael Feathers' Working Effectively with Legacy Code.

Introducing/selling TDD to the other developers

A lot of technical presentations focus on the how instead of why it is beneficial to the audience. When it comes to TDD, I would also suggest presenting it simple - skip the emerging design concept (some of the early advocates noted that good design emerging from TDD might had to do with the fact that those early advocates actually had great design sense anyway), don't mention YAGNI - in short, don't overload them with new concepts. I rather liked the approach Paul Butcher took in his Debug It! book - we tend to program one baby step at a time, articulating in our mind what the next behavior we'll implement should be, then we code it, and move to the next behavior. TDD (or Test First Development for the nitpickers :)) is just a minor step from that, i.e.: we actually write down the articulated concept in code (which we tend to do anyways with the throwaway main methods used for debugging). And in addition, we get an automated regression test suite for free :). While a live demo could be cool, I would probably show one of the Kata casts instead of me typing on the projector. Also, be sure to point out that at first we won't be able to code at this speed (I still can't)!

Learning/practice

TDD is a rather simple concept, but so are all the great wisdoms - it takes a lot of time and experience to fully understand those concepts. After the initial TDD sales pitch, I would dedicate a session to going over the tools and the basic usage - how to install the test library, how to run the tests, how to debug a single test, and the basic assert statements - AreEqual vs. AreSame, Assert.Throws, and also an example to show that most likely there are more asserts in the library (e.g. StringAssert.EndsWith).

With those interested/committed, we can move on to the supervised practice step. We've worked on simple problems, in a structure inspired by this Peer learning presentation by Jason Gorman (his site even has a full case study of this applied at the BBC). The coding dojo is another possible format. Important thing is discuss the experiences, lessons learnt, and problems overcome with the whole group, because while practice makes one perfect, bad, practiced habits are hard to unlearn. It is OK not to have all the answers - just be ready to admit it, and ensure that you can find it out from somewhere. Meetups, user groups, and similar forums are invaluably helpful.

Be sure to eat elephant in a piecemeal fashion - while at first the simple problems seem contrived, don't attempt to practice testing existing, complex production code. Stick to the simple problems for practice, every now and then actually demonstrate that it is possibly to test legacy code (though it'll require quite some preparation from you), maybe have some of these examples as a group exercise, but be sure to keep the practice focused on the simple mechanics initially. Just like any sport practice - you repeat rather basic exercises over and over, because it does pay off in the long run, but to keep you motivated, you do have the free practice sessions too.

Don't force it

Not everyone has the same passion as you do (you must have if you read this far :)), and some take a longer time to learn, have different pre-existing concerns they should overcome. Lead by example, be available to talk about it, answer questions, clarify, but don't try to force it on them. Urban cycling didn't became popular because we, cyclists, were preaching to pedestrians and car drivers - we were just approachable to friends, coworkers, etc. and provided answers and listened to their concerns. Yes, it could take a long time, but patience pays off - self-motivated people are more likely to overcome the hurdles they inevitably will face in the learning process!

Bringing it to production

I wouldn't set strong rules or hard deadlines like "two months from now, you all must write all production code in a TDD fashion", but rather just tell people to feel free to experiment during their daily work, but ensure they don't confuse work and practice. If they get stuck on testing some problem, they should just make a note, and deliver the feature untested, as they used to do before. Later, together with the group they should discuss it and potentially find a way to test it.

Further resources

This post is just a high level overview, and I have simply glossed over many of the actual difficulties about writing good unit tests, keeping them maintainable, etc.. There are books written about the topic, in almost all programming languages. Coming from a .NET background, I can recommend The Art of Unit Testing with examples in .NET by Roy Osherov, and I've found James Shore's Let's Play TDD Java episodes great - the best thing about the latter is the thought process narrative that goes with it. Driving Technical Change might be another good book, though I haven't yet read that.

Good luck!