Test automation strategy: automating the right things
Suites get abandoned when they are slow, flaky or testing the wrong level. A deliberate strategy prevents all three.
ZooBieDoo Infotech Engineering TeamRead
Key takeaways
- Escaped defect rate measures the process, not the activity.
- Analyse each escape to find the missing test level.
- Test case counts and raw bug counts mislead.
- Report customer impact and trend, not dashboards.
Escaped defects are the primary measure
Defects found in production are what quality assurance exists to prevent. Tracking the escape rate — defects found after release as a proportion of total — measures the process rather than the activity.
Analyse each escape: which test level should have caught it, and why did it not? That analysis improves the process; counting alone does not.
Measures that inform decisions
- Escape rate — is quality improving?
- Time to detect and to fix — how quickly do problems surface and resolve?
- Suite reliability — what proportion of failures are genuine?
- Coverage of critical paths — more meaningful than overall coverage percentage.
Measures that mislead
Test case count measures activity. Bug count without severity rewards raising trivial issues. Individual defect counts encourage blame and discourage honest reporting.
Any measure that becomes a target stops measuring what it did. Report them as instruments for improvement, not as a scoreboard.
Report in business terms
Executives care about release confidence, customer impact and whether quality is improving. Translate accordingly: how many customer-affecting defects reached production this quarter, and what the trend is.
A short narrative with two or three numbers gets read. A dashboard with twenty does not.
Using metrics to change something
Metrics only earn their collection cost if they lead to action. Pair each measure with the decision it informs, and drop any that inform nothing.
A rising escape rate should trigger analysis of where those defects should have been caught. A falling suite reliability figure should trigger flake remediation. If a number moves and nothing happens, it is decoration.
Review the metric set itself annually. Measures that were useful during one improvement effort often stop being informative once that problem is solved.
Frequently asked questions
Improvement in your own trend matters more than any benchmark, because counting conventions vary. What matters is that the number is falling.
No. It discourages honest reporting and ignores that defects usually reflect system and process conditions rather than individual capability.
Look at whether tests actually catch defects before release, and analyse escapes for the coverage gap they reveal.
