How to keep the trunk stable when tests take a long time?
- by Oak
We have three sets of test suites:
A "small" suite, taking only a couple of hours to run
A "medium" suite that takes multiple hours, usually ran every night (nightly)
A "large" suite that takes a week+ to run
We also have a bunch of shorter test suites, but I'm not focusing on them here.
The current methodology is to run the small suite before each commit to the trunk. Then, the medium suite runs every night, and if in the morning it turned out it failed, we try to isolate which of yesterday's commits was to blame, rollback that commit and retry the tests. A similar process, only at a weekly instead of nightly frequency, is done for the large suite.
Unfortunately, the medium suite does fail pretty frequently. That means that the trunk is often unstable, which is extremely annoying when you want to make modifications and test them. It's annoying because when I check out from the trunk, I cannot know for certain it's stable, and if a test fails I cannot know for certain if it's my fault or not.
My question is, is there some known methodology for handling these kinds of situations in a way which will leave the trunk always in top shape? e.g. "commit into a special precommit branch which will then periodically update the trunk every time the nightly passes".
And does it matter if it's a centralized source control system like SVN or a distributed one like git?
By the way I am a junior developer with a limited ability to change things, I'm just trying to understand if there's a way to handle this pain I am experiencing.
