Test automation strategy: automating the right things
Suites get abandoned when they are slow, flaky or testing the wrong level. A deliberate strategy prevents all three.
ZooBieDoo Infotech Engineering TeamRead
Key takeaways
- Define a specific, numeric question before testing.
- Soak testing finds what load and stress testing cannot.
- Realistic journeys and data volumes are essential.
- Attribute the bottleneck, fix, then re-measure.
Ask a specific question
“Is it fast enough” is not testable. “Can we serve 500 concurrent users with 95th percentile response under one second” is.
Derive targets from real expectations — peak historical traffic with headroom, or a documented business projection — and agree them before testing.
The test types and what they find
- Load — behaviour at expected volume.
- Stress — where it breaks and how it fails.
- Soak — degradation over time, which finds leaks and connection exhaustion.
- Spike — response to sudden surges, relevant for campaigns and launches.
Soak testing is skipped most often and finds problems the others cannot, because some failures only appear after hours of sustained load.
Realistic scenarios and data
Traffic that hits one endpoint repeatedly against one record tells you about caching, not about your application. Model realistic journeys with realistic think time and a data set large enough to defeat trivial caching.
Test with production-like data volumes. Queries that are instant against ten thousand rows behave differently against ten million.
Analyse the bottleneck, not just the number
A failed test is only useful if you can attribute it. Correlate response times with application traces, database metrics and resource utilisation to identify the actual constraint.
Then re-test after the fix. Performance work without before-and-after measurement is guesswork.
Testing the things around the application
Application code is rarely the whole bottleneck. Databases, caches, message queues, third-party services and network paths all contribute, and testing only the application misses them.
- Confirm connection pools are sized for the concurrency being tested.
- Check third-party rate limits — a load test can trigger them and produce misleading results.
- Watch database connection counts and lock contention, not just query time.
- Test with caches cold as well as warm.
Cold-cache behaviour is what users experience after every deployment, and it is routinely untested.
Frequently asked questions
Early enough to influence architecture, then before each significant release. Leaving it until immediately before launch means findings arrive when they cannot be acted on.
Carefully, with realistic traffic shadowing or a small controlled load during quiet periods. It gives the most accurate answer and needs safeguards.
k6 suits developer-owned testing with code-based scenarios; JMeter has broader protocol support. Either is fine; the scenario design matters far more.
