How to make slow CI builds fast again

Why this matters

Slow CI is usually two problems wearing one coat. Queue time is how long a job waits for a runner. Run time is what happens after it starts. Most people only look at the second. If your tests take 12 minutes but sit in a queue for 20, a bigger machine changes nothing.

The bigger cost is what slow builds do to how people work. When feedback takes 45 minutes, engineers batch three changes into one push. A broken main branch stays broken for hours because nobody saw the alert. Diffs get bigger and reviews get worse.

Speed also decays quietly. Someone adds an end-to-end test, then another. Someone adds a scan that runs on every commit. No one tracks the total, so no one sees the drift until a pull request takes an hour and the team starts complaining in chat.

Before you start

Start with numbers, not impressions. Pick your slowest pipeline and record the wall clock from push to green across ten runs. Split it into queue wait, checkout, dependency install, build, tests and artifact upload. Your CI provider shows job duration already. For the inner stages you may need to add timestamps or open the step timing view.

Use p50 and p95, not the average. The median says what a normal push feels like. The 95th percentile says what happens when the cache misses or the queue is busy, and that is the experience that makes people stop trusting the pipeline.

Set a target before you touch anything. Ten minutes for pull request feedback changes how a team works. Main branch and nightly pipelines can take longer. Write the target down, because the work usually means moving or deleting tests, and that argument is easier with a number on the wall. The rest of How to make slow CI builds fast again assumes you have that baseline.

How to do it

Work in order of the biggest win. Queue time first when it dominates, because it is usually the cheapest fix. Repeated work next: dependency installs, compilation, image builds. Parallelism after that. Deleting work outright comes last, because it needs the most agreement from the team.

Change one thing per week and re-measure. If you add caching, split jobs and resize runners in the same afternoon, you learn nothing when the number moves. Small steps also make it easy to revert the one that made things worse.

Part of this is a people problem. Path filters and test quarantine mean someone has to decide what does not run on every pull request. Agree who owns the pipeline before you remove jobs, or the first red main branch sends everyone back to the old setup.

1. Measure every stage of one pipeline run. Add timestamps around checkout, dependency install, build, tests and artifact upload, then run the pipeline ten times. Record p50 and p95 for each stage, plus the time spent waiting for a runner to pick the job up.
2. Fix queue time before anything else. Look at how long jobs wait before they start. If waiting is a large share of the total, add concurrency, use autoscaling runners or move heavy jobs to a larger machine type. Speeding up the job itself does nothing when the queue is the bottleneck.
3. Key your caches on the lockfile. Cache package manager directories and build outputs with a key built from the lockfile hash plus the tool version. Check the hit rate after a week. A cache that never hits costs upload time and hides nothing.
4. Split the pipeline at real seams. Separate lint, unit tests, integration tests and build into jobs that run in parallel. Keep tests that share a database in the same job, or give each job its own namespace. Parallelism only helps when the jobs do not block each other.
5. Run only what changed. Use path filters to skip pipelines for docs only commits, and affected project detection so a change in one package tests that package and its dependents. Keep a full run on the main branch so nothing escapes.
6. Move slow tests out of the pull request lane. Put browser end to end tests, migrations and long integration suites on a nightly or pre-merge lane. Quarantine flaky tests into a separate job with a ticket attached, and delete tests that assert nothing useful.
7. Trim the checkout and the image size. Use a shallow clone or sparse checkout when the repository is large. Pull prebuilt images instead of building tools on every run, and order Dockerfile layers so dependency install sits above the source copy.
8. Remove duplicate work. Look for steps that install the same dependencies twice, run the same linter in two jobs, or rebuild an artifact for each environment. Build it once, upload it, and download it downstream.
9. Set a build time budget with an alert. Pick a p95 target for the pull request pipeline and track it on a dashboard. Warn when a run crosses it and note the cause in the pull request. Without a budget, the time creeps back within a few months.

How to check it worked

Compare p95 before and after, on the same pipeline with a similar commit volume. One fast run proves nothing. Look for the tail shrinking, because the tail is what people remember when they talk about the build being slow.

Watch the quality signals at the same time: flaky test rate, how often main goes red, how many bugs reach staging. A pipeline that got faster because it stopped running integration tests is not faster, it is quieter. If you moved tests to a nightly lane, confirm the lane runs and that someone reads it.

Check cost and cache behavior too. Runner minutes per week, cache storage, cache hit rate, and how much time is now spent waiting for capacity. Run time often drops while queue time grows, because the same runners now serve more jobs. Measure again a month later.

Mistakes to avoid

Caching the wrong thing is the classic mistake. A key without the lockfile gives you stale dependencies and strange failures. A key built from the whole repository hash gives you a near zero hit rate and an upload cost on every run. Key on the lockfile plus the tool version.

Blanket retries make a pipeline look stable while doubling the worst case. A test that fails once in twenty runs should be quarantined with an owner, not retried. And do not make everything parallel at once: tests that share a database or a fixed port will collide.

Optimizing the wrong stage is the other common one. Teams spend a week shaving a four minute test job while twenty minutes of queue time sits untouched. Look at the totals before you pick a target.

Pipelines grow back when nobody owns them. A p95 budget, an alert and a named owner keep it honest. If you would rather have someone build the thing, Cloudverg's Delivery platform setup package covers pipelines, environments, secrets, monitoring and rollback for $14,500 at a fixed price, over four weeks. It is one option among several.

Frequently asked questions