Designing a CI/CD pipeline developers will not route around
A pipeline succeeds when it is fast enough to trust and strict enough to matter. Getting that balance wrong is why teams disable checks.
ZooBieDoo Infotech Engineering TeamRead
Key takeaways
- Blue-green gives instant rollback at the cost of double capacity.
- Canary limits exposure but needs trustworthy metrics.
- Feature flags separate deployment risk from release risk.
- Backward-compatible database changes are what make rollback possible.
The patterns
Blue-green runs two production environments and switches traffic between them. Rollback is a traffic switch, which makes it fast and predictable, at the cost of running double capacity during a release.
Canary routes a small proportion of traffic to the new version, watches health signals, then increases gradually. It limits exposure and needs good metrics to be meaningful.
Feature flags decouple deploy from release
Flags let you ship code to production disabled, then enable it for a subset of users. Deployment stops being the moment of risk, which reduces the pressure on release windows considerably.
The discipline is removing flags once a feature is fully released. A codebase carrying dozens of stale flags becomes very difficult to reason about.
What makes rollback actually work
- Database changes that are backward compatible, so the previous version still runs.
- Artefacts retained so the previous build can be redeployed immediately.
- A single documented command or action, rehearsed regularly.
- Clear criteria for when to roll back rather than debate it during an incident.
Choosing
For most business applications, rolling deployment with health checks plus feature flags provides the majority of the benefit at modest cost. Blue-green suits systems where an instant, total switch is valuable. Canary suits high-traffic services where a small percentage still provides a statistically useful signal.
The wrong answer is a big-bang release with no rehearsed way back, which is still surprisingly common.
Automating the decision to roll back
Deciding whether to roll back during an incident is stressful and slow, and the pressure pushes people towards waiting. Agreeing the criteria in advance removes the debate.
Define thresholds — error rate above a level, latency beyond a bound, a critical funnel step failing — and where possible automate the rollback against them. Automated rollback on a clear signal is faster and more consistent than human judgement at three in the morning.
Keep a manual override. Automation handles the obvious cases; people handle the ambiguous ones, and they should not have to fight the automation to do it.
Frequently asked questions
No. Load balancers, managed container services and traditional web servers all support rolling and blue-green approaches. Kubernetes makes some of it more convenient, not possible.
Long enough to gather a meaningful signal for your traffic volume often under an hour for high-traffic services, longer for low-traffic ones where a small percentage takes time to accumulate.
Split it: deploy the additive change, migrate data, deploy code using the new shape, then remove the old column in a later release. It is more steps and it keeps rollback available throughout.
