Tips for achieving "continual" delivery
- by Ben
A team is experiencing difficulty releasing software on a frequent basis (once every week). What follows is a typical release timeline:
During the iteration:
Developers work on stories on the backlog on short-lived (this is enthusiastically enforced) feature branches based on the master branch.
Developers frequently pull their feature branches into the integration branch, which is continually built and tested (as far as the test coverage goes) automatically.
The testers have the ability to auto-deploy integration to a staging environment and this occurs multiple times per week, enabling continual running of their test suites.
Every Monday:
there is a release planning meeting to determine which stories are "known good" (based on the testers' work), and hence will be in the release. If there is a known issue with a story, the source branch is pulled out of integration.
no new code (only bug fixes requested by the testers) may be pulled into integration on this Monday to ensure the testers have a stable codebase to cut a release from.
Every Tuesday:
The testers have tested the integration branch as much as they possibly can have given the time available and there are no known bugs so a release is cut and pushed out to the production nodes slowly.
This sounds OK in practise, but we have found that it is incredibly difficult to achieve. The team sees the following symptoms
"subtle" bugs are found on production that were not identified on the staging environment.
last minute hot-fixes continue into the Tuesday.
problems on the production environment require roll-backs which blocks continued development until a successful live deployment is achieved and the master branch can be updated (and hence branched from).
I think test coverage, code quality, ability to regression test quickly, last minute changes and environmental differences are at play here. Can anyone offer any advice regarding how best to achieve "continual" delivery?
