Designing a CI/CD pipeline developers will not route around
A pipeline succeeds when it is fast enough to trust and strict enough to matter. Getting that balance wrong is why teams disable checks.
ZooBieDoo Infotech Engineering TeamRead
Key takeaways
- Assign coordination, communication and investigation roles explicitly.
- Keep a timestamped log during the incident, not after.
- Blameless means examining the system, not excusing everything.
- Actions need owners, dates and a place in the real backlog.
Structure during the incident
Assign roles explicitly: someone coordinates, someone communicates, someone investigates. Without that, everyone investigates and nobody tells stakeholders anything, which generates a second problem alongside the first.
Keep a timestamped log as you go. Reconstructing the sequence afterwards from memory is unreliable, and the log is the most valuable input to the review.
Blameless means focused on the system
People make reasonable decisions given the information and pressure they have. If a person could cause an outage by making one mistake, the system permitted it, and the system is what to change.
Blameless does not mean uncritical. It means asking why the system allowed the failure rather than who should have prevented it.
Actions with owners and dates
- Every action has a named owner and a due date.
- Actions go into the normal work backlog, not a separate document.
- Prevention and detection actions both count — finding it faster is a genuine improvement.
- Review outstanding actions periodically; unfinished ones indicate the process is not real.
Tracking whether anything changed
The measure of a post-mortem process is not how well the documents read but whether incidents stop recurring. Tag incidents by cause and review the pattern each quarter.
Repeated causes indicate that actions are not being completed, or that they addressed a symptom. Both are worth knowing, and neither is visible from individual reviews.
Report the trend to leadership alongside delivery metrics. Reliability improvement competes for capacity like anything else, and it needs evidence to compete successfully.
Frequently asked questions
Anything with customer impact, anything that took unexpectedly long to resolve, and any near miss that could have been serious. Near misses are the cheapest learning available.
Within a few days, while memory is fresh but after people have rested. Immediately afterwards, the team is tired and the analysis suffers.
Everyone involved, plus someone independent to ask the obvious questions. Keep it small enough for genuine discussion.
