Back to all notes

In Praise of Boring Infrastructure

The most interesting systems I have built are the ones nobody has to think about.

The best infrastructure I have ever shipped is the kind nobody talks about. It does not get a launch post. It does not trend on Hacker News. It just sits there, quietly returning 200s, and the only time anyone remembers it exists is when it does not.

That is the goal, and it is much harder to build than the alternative.

Interesting is a cost

When a system is interesting, it means someone has to hold a model of it in their head. They have to know which queue drains first, which cache is authoritative, and why the retry budget is 3 and not 5. Every one of those facts is a small tax on the next person.

The tax compounds. A clever architecture that saves a team a week up front can cost them a year of onboarding, incidents, and quiet fear of touching the wrong file. Simple is not the same as easy - a simple system is one whose complexity has been paid down deliberately.

A complex system that works is invariably found to have evolved from a simple system that worked.
John Gall, Systemantics

What boring looks like

  • One way to do a thing. Two paths to the same outcome are two places to fix a bug.
  • Explicit over clever. A plain loop beats a regex nobody wants to debug at 3am.
  • Bounded failure. Every dependency has a timeout, and every timeout has a fallback.
  • Observable by default. If you cannot answer 'is it healthy?' with one query, it is not done.

The boring test

I have started asking one question in design review: what happens when this is at its worst? Not on the happy path, not in a demo, but on the day the queue backs up and the on-call engineer has had four hours of sleep. If the answer is 'it depends on the internals', the design is asking for heroics. Boring systems do not need heroes.

timeout.ts
export async function withTimeout<T>(
  work: Promise<T>,
  ms: number,
): Promise<T> {
  const timeout = new Promise<never>((_, reject) => {
    setTimeout(
      () => reject(new Error("timed out after " + ms + "ms")),
      ms,
    );
  });
  return Promise.race([work, timeout]);
}

Fourteen lines. No abstraction to learn, no config to load. It will still make sense in two years, and that is the entire point.