Test automation strategy: automating the right things
Suites get abandoned when they are slow, flaky or testing the wrong level. A deliberate strategy prevents all three.
ZooBieDoo Infotech Engineering TeamRead
Key takeaways
- Suites decay unless actively maintained.
- Parallelisation and level-shifting cut runtime without cutting coverage.
- Tiered execution puts the right tests at the right moment.
- Deleting tests that never catch anything is legitimate maintenance.
Suites decay predictably
Tests accumulate as features are added and are rarely removed. Runtime grows, redundancy creeps in, and eventually the suite runs less often — which defeats its purpose.
Treat the suite as a maintained asset with a budget, not as an archive that only grows.
Reduce runtime before reducing coverage
- Parallelise across workers — usually the largest single win.
- Move verification down a level where an API test would suffice.
- Remove duplicate tests covering the same path.
- Share expensive setup rather than repeating it per test.
Run the right subset at the right time
Not every test needs to run on every change. A fast smoke suite on each push, a broader suite on merge, and the full suite nightly is a reasonable default.
Test impact analysis — selecting tests based on what changed — helps in large codebases and needs reliable dependency mapping to be trustworthy.
Review the suite periodically
Look at which tests have never failed, which fail frequently for non-defect reasons, and which cover code paths that no longer exist.
Deleting a test feels like reducing safety. Keeping a test that has never caught anything and costs runtime on every build is a real cost with no return.
Choosing what to run before release
Even with a fast suite, some releases warrant more than the standard run. Deciding what extra to run should follow the change rather than habit.
A change to authentication, payments or data handling warrants the full suite plus targeted manual testing. A copy change on a marketing page does not. Making this explicit prevents both over-testing routine changes and under-testing risky ones.
Record what was run for each release. When a defect escapes, knowing which suites ran tells you where the gap is.
Frequently asked questions
Short enough to run on merge without blocking people typically well under an hour. Longer suites should be tiered.
Yes, when they duplicate other coverage or test removed behaviour. Record why, in case the decision needs revisiting.
Reserve it for exploratory testing and experiential judgement, which automation is genuinely bad at. Repetitive manual regression is expensive and unreliable.
