Test automation engagements are usually scoped in coverage percentage, which is the wrong unit. A suite at eighty per cent coverage that takes forty minutes and fails intermittently gets ignored, and an ignored suite has negative value - it costs money to maintain and provides no signal. The metric that predicts whether automation pays back is how often the team trusts a red build.
01Coverage is an input, trust is the output
Coverage measures which lines executed during a test run. It does not measure whether the assertions were meaningful, whether the failure modes users actually hit are exercised, or whether anyone believes the result. A suite can reach high coverage while asserting almost nothing.
The number worth tracking is the proportion of red builds that turn out to be real. When that drops below roughly nine in ten, engineers start re-running failed builds by reflex, and at that point the suite has stopped being a quality gate and become a tax on merging.
02Where the money goes
Writing tests is the cheap part and it is what proposals are priced on. The expensive parts are test data management, environment provisioning, and the ongoing maintenance of tests that break when the UI changes for reasons unrelated to behaviour.
Test data is usually the largest hidden cost. Tests that depend on a shared environment with mutable state are flaky by construction, and fixing that means either generating data per test run or building isolation into the application - both real engineering projects that rarely appear in a QA proposal.
| Activity | Share of effort | Usually in the proposal? |
|---|---|---|
| Writing test cases | 20-30% | Yes - this is what gets priced |
| Test data setup and isolation | 20-30% | Rarely |
| Environment and CI infrastructure | 15-20% | Sometimes |
| Maintenance of existing tests | 25-35% ongoing | Almost never |
| Triage of flaky failures | Varies - unbounded if unmanaged | No |
03The shape that holds up
Fast unit tests carry the bulk of the assertions because they are cheap to run and stable by construction. Integration tests cover the boundaries where components actually break - the database access layer, the external API client, the message consumer. End-to-end tests cover a small number of journeys that must never break, and no more, because they are the slowest and flakiest layer.
The common failure is inverting this: heavy investment in end-to-end coverage because it demonstrates progress visibly, producing a suite that takes forty minutes, fails for environmental reasons twice a day, and gets bypassed under deadline pressure.
- Target a full pre-merge suite under ten minutes; beyond that, people work around it
- Quarantine flaky tests immediately and fix or delete them within a fixed window
- End-to-end tests only for journeys where failure is unacceptable - typically under twenty
- Track mean time to diagnose a failure, not just pass rate
04When outsourcing QA works and when it does not
An external team building and maintaining the automation infrastructure works well when the contract is about the suite's health rather than its size. Measured on test count, an outsourced QA team will produce test count - including brittle end-to-end tests that inflate the number and degrade the signal.
Measured on suite runtime, flake rate and escaped defects, the same team optimises for a suite the engineers trust. The contract shapes the outcome far more than the team's skill does, and this is the most common reason QA outsourcing disappoints.
Topics
Priya Iyer
Staff Engineer · SyncTrix
Writes about the engineering decisions behind production systems - architecture, delivery and the trade-offs that only show up at scale.
Building something like this?
SyncTrix engineers AI, SaaS, platform and cloud systems for enterprises and high-growth teams. Tell us what you're shipping and we'll scope it with you.