Is it feasible and useful to auto-generate some code of unit tests?
- by skiwi
Earlier today I have come up with an idea, based upon a particular real use case, which I would want to have checked for feasability and usefulness. This question will feature a fair chunk of Java code, but can be applied to all languages running inside a VM, and maybe even outside. While there is real code, it uses nothing language-specific, so please read it mostly as pseudo code.
The idea
Make unit testing less cumbersome by adding in some ways to autogenerate code based on human interaction with the codebase. I understand this goes against the principle of TDD, but I don't think anyone ever proved that doing TDD is better over first creating code and then immediatly therafter the tests. This may even be adapted to be fit into TDD, but that is not my current goal.
To show how it is intended to be used, I'll copy one of my classes here, for which I need to make unit tests.
public class PutMonsterOnFieldAction implements PlayerAction {
private final int handCardIndex;
private final int fieldMonsterIndex;
public PutMonsterOnFieldAction(final int handCardIndex, final int fieldMonsterIndex) {
this.handCardIndex = Arguments.requirePositiveOrZero(handCardIndex, "handCardIndex");
this.fieldMonsterIndex = Arguments.requirePositiveOrZero(fieldMonsterIndex, "fieldCardIndex");
}
@Override
public boolean isActionAllowed(final Player player) {
Objects.requireNonNull(player, "player");
Hand hand = player.getHand();
Field field = player.getField();
if (handCardIndex >= hand.getCapacity()) {
return false;
}
if (fieldMonsterIndex >= field.getMonsterCapacity()) {
return false;
}
if (field.hasMonster(fieldMonsterIndex)) {
return false;
}
if (!(hand.get(handCardIndex) instanceof MonsterCard)) {
return false;
}
return true;
}
@Override
public void performAction(final Player player) {
Objects.requireNonNull(player);
if (!isActionAllowed(player)) {
throw new PlayerActionNotAllowedException();
}
Hand hand = player.getHand();
Field field = player.getField();
field.setMonster(fieldMonsterIndex, (MonsterCard)hand.play(handCardIndex));
}
}
We can observe the need for the following tests:
Constructor test with valid input
Constructor test with invalid inputs
isActionAllowed test with valid input
isActionAllowed test with invalid inputs
performAction test with valid input
performAction test with invalid inputs
My idea mainly focuses on the isActionAllowed test with invalid inputs. Writing these tests is not fun, you need to ensure a number of conditions and you check whether it really returns false, this can be extended to performAction, where an exception needs to be thrown in that case.
The goal of my idea is to generate those tests, by indicating (through GUI of IDE hopefully) that you want to generate tests based on a specific branch.
The implementation by example
User clicks on "Generate code for branch if (handCardIndex >= hand.getCapacity())".
Now the tool needs to find a case where that holds.
(I haven't added the relevant code as that may clutter the post ultimately)
To invalidate the branch, the tool needs to find a handCardIndex and hand.getCapacity() such that the condition >= holds.
It needs to construct a Player with a Hand that has a capacity of at least 1.
It notices that the capacity private int of Hand needs to be at least 1.
It searches for ways to set it to 1. Fortunately it finds a constructor that takes the capacity as an argument. It uses 1 for this.
Some more work needs to be done to succesfully construct a Player instance, involving the creation of objects that have constraints that can be seen by inspecting the source code.
It has found the hand with the least capacity possible and is able to construct it.
Now to invalidate the test it will need to set handCardIndex = 1.
It constructs the test and asserts it to be false (the returned value of the branch)
What does the tool need to work?
In order to function properly, it will need the ability to scan through all source code (including JDK code) to figure out all constraints. Optionally this could be done through the javadoc, but that is not always used to indicate all constraints. It could also do some trial and error, but it pretty much stops if you cannot attach source code to compiled classes.
Then it needs some basic knowledge of what the primitive types are, including arrays. And it needs to be able to construct some form of "modification trees". The tool knows that it needs to change a certain variable to a different value in order to get the correct testcase. Hence it will need to list all possible ways to change it, without using reflection obviously.
What this tool will not replace is the need to create tailored unit tests that tests all kinds of conditions when a certain method actually works. It is purely to be used to test methods when they invalidate constraints.
My questions:
Is creating such a tool feasible? Would it ever work, or are there some obvious problems?
Would such a tool be useful? Is it even useful to automatically generate these testcases at all? Could it be extended to do even more useful things?
Does, by chance, such a project already exist and would I be reinventing the wheel?
If not proven useful, but still possible to make such thing, I will still consider it for fun. If it's considered useful, then I might make an open source project for it depending on the time.
For people searching more background information about the used Player and Hand classes in my example, please refer to this repository. At the time of writing the PutMonsterOnFieldAction has not been uploaded to the repo yet, but this will be done once I'm done with the unit tests.
