When a pipeline becomes slow, teams often start by increasing runner size or considering another CI product. Those changes can help, but they are expensive ways to avoid finding where the time actually goes. We first split the path into queue time, setup time, build time, tests, security checks, artifact handling and deployment. The bottleneck is usually obvious once those are measured separately.
Start with queue time, not execution time
If a job waits five minutes for a runner and executes in three, optimizing the build script is not the first move. We check runner saturation by time of day, concurrency limits, long-running jobs occupying scarce executors and whether unrelated workloads share the same pool. Queueing problems need capacity or scheduling changes, not build flags.
Find repeated setup work
- Dependencies downloaded from the public internet on every job instead of using a reliable cache or proxy.
- Container images rebuilt from an unchanged base layer because cache keys are too broad.
- Toolchains installed from scratch instead of using a versioned runner image.
- Repositories or monorepos checked out more deeply than the job needs.
Caching is not free. A cache that is rarely hit or frequently corrupts is operational debt. We measure hit rate and restore time before adding another cache layer.
Separate fast feedback from full confidence
The first pipeline stage should answer a narrow question quickly: is this change obviously broken? Unit tests, static checks and targeted builds belong early. Expensive integration suites, browser tests and environment-dependent checks can run after the change has passed the cheap filters. Running every test for every change makes the pipeline thorough on paper and slow enough that developers route around it in practice.
Look for accidental serialization
Many pipelines are slow because jobs that could run independently are chained together. We draw the dependency graph and ask whether each edge is real. Build, unit tests, linting and several security checks often run in parallel. Deployment promotion, database migrations and shared-environment tests usually need stricter ordering.
Treat flaky tests as latency
A test that fails intermittently and passes on retry adds more than its runtime. It creates queueing, human investigation and uncertainty about whether red means broken. We track retry rate and quarantine only with an owner and expiry date. Permanent retry loops are a way of hiding a broken test suite.
Check artifact and deployment boundaries
A pipeline should normally build an immutable artifact once and promote that artifact through environments. Rebuilding per environment makes the path slower and means production is not necessarily running what staging tested. We also look for deployment steps that wait on broad environment locks when only one service or namespace actually needs serialization.
The target is not the shortest possible pipeline. It is the shortest feedback loop that still protects the risks you actually care about.
Measure the change by developer wait time
Pipeline duration is useful, but lead time from commit to usable feedback is the better engineering signal. A ten-minute pipeline that starts instantly can be less disruptive than a four-minute pipeline that waits twelve minutes for capacity. We measure the path the developer experiences, then optimize the stage that dominates it.