Making CI fast: why your pipeline takes 40 minutes and how to fix it

01Measure the stages before optimising

Total duration is not actionable. Break the pipeline into stages and measure each across many runs, looking at both median and the tail. The p95 matters because that is the experience people remember and plan around.

The distribution is usually lopsided in a way nobody expects. Dependency installation, container image builds and environment setup routinely consume more time than the tests everyone assumes are the bottleneck. Teams spend weeks parallelising a test suite that accounts for a quarter of the wall clock.

• Measure per stage, tracking both median and p95.
• Include queue time - waiting for a runner is part of the experience.
• Separate cold-cache from warm-cache runs; they are different problems.
• Track the trend so regressions are visible before they become normal.

02Caching, correctly keyed

Caching dependencies is the highest-return change available, and the most commonly misconfigured. The cache key must be derived from the lockfile hash so it invalidates precisely when dependencies actually change. Keys based on a branch name or a date either serve stale content or miss constantly.

Layer beyond dependencies once that is working: compiler output, build artifacts between stages, and container layers ordered so the frequently-changing application code sits last. A Dockerfile that copies the entire source before installing dependencies rebuilds everything on every commit, and reordering two lines can remove several minutes.

03Run less, not just faster

The fastest test is one that does not need to run. In a monorepo or a modular codebase, changes to one package rarely require validating everything, and affected-target detection can eliminate the majority of work on a typical commit.

This requires an accurate dependency graph, which is the real cost. Get it wrong and you skip a test that would have caught the bug - far worse than a slow pipeline. Build tools that compute the graph from actual imports are trustworthy; hand-maintained mappings drift and should be avoided.

04Parallelise with the right split

Splitting tests across runners is straightforward until the split is uneven, at which point total time equals the slowest shard and adding runners stops helping. Splitting by file count is the naive approach and performs badly, because test files vary enormously in duration.

Split by recorded timing instead, so shards balance by actual execution time. Most modern runners support this with a stored timing manifest. Watch for the fixed overhead per runner too - container startup, dependency restore, database setup - because past a certain point that overhead exceeds the parallelism benefit and more shards make the pipeline slower.

05Flaky tests cost more than slow ones

A test that fails randomly one time in twenty destroys trust in the entire suite. Engineers learn to re-run on red without reading the output, which means genuine failures get re-run too and broken code reaches production with a green check.

Treat flakes as production incidents. Track them automatically, quarantine them out of the blocking path immediately, and assign an owner with a deadline. A quarantined test that nobody fixes should be deleted - an unmaintained test provides no value and its presence implies coverage that does not exist.