Skip to content
NewSee how this site publishes in seconds
Halftone

EngineeringInfrastructure

A field guide to incremental builds

Why the boring fix beat the clever one, and how we proved it. A short tour of the decisions behind a faster, calmer system.

A field guidetoincrementalbuildsHALFTONE / JOURNAL
Ivo Santos
Published 3 min read

Caching is easy to add and hard to trust. The rule we settled on is simple: a cache key must include everything that can change the output, and nothing that cannot. This post covers what we changed, why, and what it means for you.

#The details

For most of last year, a full build took eleven minutes. Nobody decided that was acceptable; it simply grew there, one dependency and one test suite at a time, until waiting became part of the job.

Latency at the median improved, but the tail is what people feel. Our p95 dropped from 4.1 seconds to 900 milliseconds, and support tickets about slowness fell by more than half.

We started by measuring instead of guessing. Every step got a timer, every run wrote its numbers to a table, and within a week the picture was clear: most of the time went to setup, not to the work itself.

#Results

The part we underestimated was rollback. A faster pipeline is only better if you can undo it just as fast, so every deploy now keeps the previous version warm and one click away.

We kept the old path running next to the new one for two weeks and compared outputs byte for byte. Most differences were harmless. Three were real bugs that the old path had been hiding for years.

  • Skip what has not changed
  • Cache what never changes
  • Run independent steps in parallel
  • Keep the previous version one click away

#What we measured

The fix was less clever than we expected. We cached what never changes, skipped what had not changed, and stopped doing work in series that could happen side by side.

None of this needed a new framework. It needed a clear owner, a dashboard people actually looked at, and permission to delete code that no longer earned its keep.

text
# p95 before / after
build      41.2s -> 6.8s
deploy     18.0s -> 4.4s
end-to-end 11m   -> 1m 52s

#Trade-offs

Caching is easy to add and hard to trust. The rule we settled on is simple: a cache key must include everything that can change the output, and nothing that cannot.

For most of last year, a full build took eleven minutes. Nobody decided that was acceptable; it simply grew there, one dependency and one test suite at a time, until waiting became part of the job.

Latency at the median improved, but the tail is what people feel. Our p95 dropped from 4.1 seconds to 900 milliseconds, and support tickets about slowness fell by more than half.

#Where we started

We started by measuring instead of guessing. Every step got a timer, every run wrote its numbers to a table, and within a week the picture was clear: most of the time went to setup, not to the work itself.

The part we underestimated was rollback. A faster pipeline is only better if you can undo it just as fast, so every deploy now keeps the previous version warm and one click away.

We kept the old path running next to the new one for two weeks and compared outputs byte for byte. Most differences were harmless. Three were real bugs that the old path had been hiding for years.

ts
export async function publish(post: Post) {
  const problems = validate(post);
  if (problems.length) return { ok: false, problems };
  await commit(post);
  return { ok: true, url: `/blog/${post.slug}/` };
}

Engineering8 min

What we learned rewriting log retention

What changed when we stopped guessing and started measuring every step. Why the boring fix beat the clever one, and how we proved it.

Kit Moreau / Dee Marsh

Engineering4 min

A field guide to our build pipeline

The constraint, the fix, and the numbers before and after. What changed when we stopped guessing and started measuring every step.

Fen Alvarez

Engineering8 min

A field guide to request tracing

Why the boring fix beat the clever one, and how we proved it. The constraint, the fix, and the numbers before and after.

Bo Lindqvist