Unit testing is… well, flawed.
- by Dewald Galjaard
Hey someone had to say it.
I clearly recall my first IT job. I was appointed Systems
Co-coordinator for a leading South African retailer at store level. Don’t get
me wrong, there is absolutely nothing wrong with an honest day’s labor and in
fact I highly recommend it, however I’m obliged to refer to the designation cautiously;
in reality all I had to do was monitor in-store prices and two UNIX front
line controllers. If anything went wrong – I only had to phone it in…
Luckily that wasn’t all I did. My duties extended to some other
interesting annual occurrence – stock take. Despite a bit more curious affair,
it was still a tedious process that took weeks of preparation and several nights
to complete. Then also I remember that no
matter how elaborate our planning was, the entire exercise would be rendered
useless if we couldn’t get the basics right – that being the act of counting.
Sounds simple right? We’ll with a store which could potentially
carry over tens of thousands of different items… we’ll let’s just say I believe
that’s when I first became a coffee addict.
In those days the act of counting stock was a very humble process.
Nothing like we have today. A staff member would be assigned a bin or shelve
filled with items he or she had to sort then count. Thereafter they had to
record their findings on a complementary piece of paper.
Every night I would manage several teams. Each team was divided
into two groups - counters and auditors. Both groups had the same task,
only auditors followed shortly on the heels of the counters, recounting stock levels,
making sure the original count correspond to their findings.
It was a simple yet hugely responsible orchestration of
people and thankfully there was one fundamental and golden rule I could always abide
by to ensure things run smoothly – No-one
was allowed to audit their own work. Nope, not even on nights when I didn’t
have enough staff available. This meant I too at times had to get up there and get
counting, or have the audit stand over until the next evening. The reason for
this was obvious - late at night and with so much to do we were prone to make some
mistakes, then on the recount, without a fresh set of eyes, you were likely to
repeat the offence.
Now years later this rule or guideline still holds true as
we develop software (as far removed as software development from counting
stock may be). For some reason it is a fundamental guideline we’re simply ignorant
of. We write our code, we write our tests and thus commit the same horrendous
offence. Yes, the procedure of writing unit tests as practiced in most
development houses today – is flawed.
Most if not all of the tests we write today exercise
application logic – our logic. They are based on the way we believe an
application or method should/may/will behave or function. As we write our
tests, our unit tests mirror our best
understanding of the inner workings of our application code. Unfortunately
these tests will therefore also include (or be unaware of) any imperfections
and errors on our part. If your logic is flawed as you write your initial code,
chances are, without a fresh set of eyes, you will commit the same error second
time around too.
Not even experience seems to be a suitable solution. It
certainly helps to have deeper insight, but is that really the answer we should
be looking for? Is that really failsafe?
What about code review? Code review is certainly an answer.
You could have one developer coding away and another (or team) making sure the
logic is sound. The practice however has its obvious drawbacks. Firstly and
mainly it is resource intensive and from what I’ve seen in most development
houses, given heavy deadlines, this guideline is seldom adhered to. Hardly ever
do we have the resources, money or time readily available.
So what other options are out there?
A quest to find some solution revealed a project by
Microsoft Research called PEX. PEX is a framework which creates several test
scenarios for each method or class you write, automatically. Think of it as your own personal
auditor. Within a few clicks the framework will auto generate several unit
tests for a given class or method and save them to a single project.
PEX help to audit your work. It lends a fresh set of eyes to
any project you’re working on and best of all; it is cost effective and fast.
Check them out at
http://research.microsoft.com/en-us/projects/pex/
In upcoming posts we’ll dive deeper into how it works and how it can help you.
Certainly there are more similar frameworks out there and I
would love to hear from you. Please share your experiences and insights.
