How to test the tests?
- by Ryszard Szopa
We test our code to make it more correct (actually, less likely to be incorrect). However, the tests are also code -- they can also contain errors. And if your tests are buggy, they hardly make your code better.
I can think of three possible types of errors in tests:
Logical errors, when the programmer misunderstood the task at hand, and the tests do what he thought they should do, which is wrong;
Errors in the underlying testing framework (eg. a leaky mocking abstraction);
Bugs in the tests: the test is doing slightly different than what the programmer thinks it is.
Type (1) errors seem to be impossible to prevent (unless the programmer just... gets smarter). However, (2) and (3) may be tractable. How do you deal with these types of errors? Do you have any special strategies to avoid them? For example, do you write some special "empty" tests, that only check the test author's presuppositions? Also, how do you approach debugging a broken test case?
