How to test the tests?

Posted by Ryszard Szopa on Programmers See other posts from Programmers or by Ryszard Szopa
Published on 2010-10-12T23:04:46Z Indexed on 2011/02/20 23:32 UTC
Read the original article Hit count: 254

We test our code to make it more correct (actually, less likely to be incorrect). However, the tests are also code -- they can also contain errors. And if your tests are buggy, they hardly make your code better.

I can think of three possible types of errors in tests:

  1. Logical errors, when the programmer misunderstood the task at hand, and the tests do what he thought they should do, which is wrong;

  2. Errors in the underlying testing framework (eg. a leaky mocking abstraction);

  3. Bugs in the tests: the test is doing slightly different than what the programmer thinks it is.

Type (1) errors seem to be impossible to prevent (unless the programmer just... gets smarter). However, (2) and (3) may be tractable. How do you deal with these types of errors? Do you have any special strategies to avoid them? For example, do you write some special "empty" tests, that only check the test author's presuppositions? Also, how do you approach debugging a broken test case?

© Programmers or respective owner

Related posts about unit-testing

Related posts about test-driven-development