Your CI isn't flaky — it's just doing the slow path in public

Here's the key insight: when a rerun magically fixes a red CI job, that's not flakiness — that's evidence. A test that fails once and passes on retry with zero code changes is telling you something deterministic happened the first time. The most common culprit nobody checks: the cache expired, the job took the slow path, and it ran straight into a timeout.

The story behind this piece is almost comically mundane. A one-line change to a blog page turned a lint job red — a lint job that found zero issues. Someone reran it, it passed, everyone shrugged and moved on. That shrug is the expensive part. It's the moment a real, fixable performance regression gets filed under "CI is just flaky sometimes" and never investigated again.

What's actually happening under the hood

Most CI platforms (GitLab, GitHub Actions, CircleCI) have layered timeout limits — project-level, job-level, sometimes runner-level — and jobs get killed the instant they cross that line, no matter why they were slow. The job doesn't distinguish between "the test hung" and "npm ci took four extra minutes because the cache was cold." It just times out, and the failure message looks identical either way.

Caches almost always have a retention window — seven days is a common default across CI providers. When a cache is warm, dependency installs, Docker layer pulls, and asset builds are fast, and your pipeline comfortably clears its timeout with room to spare. When the cache expires, the job silently falls back to the cold path: full dependency resolution, full image rebuild, full asset compile. If your timeout was set based on the warm-cache runtime — which it usually was, because that's what everyone sees day to day — the cold path blows straight through it.

This creates a failure pattern that looks random but is actually periodic:

• Pipeline runs fine for days
• Cache quietly expires on schedule
• Next run goes cold, times out
• Someone reruns it, cache gets repopulated, it passes
• Cycle repeats roughly every cache-retention window

If you've ever noticed CI failures that seem to cluster on a schedule — Mondays, or "every week or so" — this is very likely why.

How to actually check this instead of guessing

Don't just rerun and move on. Rerun and log what changed. The fastest way to confirm the theory is to deliberately force a cold-cache run and compare timing against a warm one.

1# GitHub Actions example: force a cache miss to measure cold-start cost
2- name: Install dependencies (cold)
3  run: npm ci
4  env:
5    CACHE_KEY_OVERRIDE: force-miss-${{ github.run_id }}

Better yet, instrument each phase of the pipeline separately instead of measuring total job time as one blob:

1echo "checkout_start=$(date +%s)" >> timing.log
2# ... checkout step ...
3echo "checkout_end=$(date +%s)" >> timing.log
4
5echo "install_start=$(date +%s)" >> timing.log
6npm ci
7echo "install_end=$(date +%s)" >> timing.log
8
9echo "test_start=$(date +%s)" >> timing.log
10npm test
11echo "test_end=$(date +%s)" >> timing.log

Parse timing.log across a week of runs and you'll usually see one phase — almost always dependency install or image build — spike dramatically on a regular cadence. That spike is your cache expiring, not your tests destabilizing.

The fix is boring, which is why nobody does it

Once you've confirmed cache expiry is the trigger, the fix isn't "increase the timeout." That just delays the same problem and burns more compute every cold run. The actual fixes:

• Give timeouts real headroom. If your job normally finishes in 8 minutes with a warm cache, don't set the timeout at 10. Measure the cold-cache worst case and set the limit above that, not above the happy path.
• Decouple cache lifetime from your deploy cadence. If your team ships less frequently than your cache expires, you're guaranteed to hit cold paths in production-critical pipelines.
• Warm caches proactively. A scheduled job that refreshes the cache before it expires (say, every 5 days instead of letting it die at 7) avoids the cold path entirely for teams that ship daily.
• Optimize the actual slow step, not the timeout around it. If npm ci or a Docker layer rebuild dominates cold-start time, that's a lockfile/dependency-mirroring problem worth solving on its own merits — it'll pay off even outside CI.

Why this matters

The expensive mistake here isn't the timeout — it's the retry culture that hides it. Every time a team reruns a red build without asking why it went red, they're spending engineer time to mask a measurable, fixable performance regression instead of fixing it once. Treat retries as a signal, not a workaround: log cache state, queue time, and phase timings on every failure, and you'll usually find the real bug is in your pipeline's assumptions about warm caches, not in your code.