Storage · 2026-07-14

Storage is a distributed systems problem wearing a boring hat

Availability failure someone is paged Correctness failure write replica replica backup nothing fires the bad value is now everywhere, including the copy you would restore from
An availability failure tells you it happened. A correctness failure propagates quietly to every replica and every backup first.

Storage has an image problem. It reads as plumbing — necessary, unglamorous, basically solved — while the work that sounds interesting happens somewhere with a nicer architecture diagram. I have had engineers ask me, politely, whether I was worried about being stuck in it.

Sixteen years in and I would argue close to the opposite. Enterprise storage is one of the least forgiving distributed systems domains there is, and it is unforgiving in a way that teaches you things the rest of the industry tends to learn later and more expensively.

The reason is the failure semantics.

In most distributed systems the worst realistic outcome of a bug is that something is unavailable for a while. Someone gets paged, a dashboard goes red, a status page gets updated, and the system comes back. In storage the worst outcome is that something is wrong — silently, durably, and in a way that will propagate to every replica and every backup before a single human being notices. Availability problems announce themselves. Correctness problems do not, and they are patient.

That one asymmetry shapes everything downstream of it.

It is why replication semantics get argued about at a level of pedantry that looks excessive from outside the room. It looks excessive right up until you have watched what happens when two sides of a replicated pair hold different opinions about which of them is authoritative. I spent a long stretch on a disaster-recovery feature that synchronised NAS data across an enterprise replication link, and the engineering was never the hard part. The hard part was being precise about what each side was permitted to believe, and when.

It is why the interesting question about a failover is never whether it failed over. It is what the two sides believed about each other while it was happening, and whether there was any interval in which both would have accepted a write.

It is why migration and cutover work is genuinely difficult rather than merely tedious, which I think is the most underrated distinction in this field.

There is a career argument buried in here, and it is the one I actually give when asked. The question is usually some version of whether storage is a dead end now that everything is moving to cloud and AI. I think that has it backwards. The AI systems being built right now are enormous data-movement problems with hard correctness requirements bolted on. Somebody in those rooms has to understand what happens when the bytes do not arrive, or arrive twice, or arrive stale, or arrive correct but from a snapshot taken before the thing you care about.

That instinct does not come from reading about it. It comes from having been wrong about it once, in production, with a customer on the call.

The hat is boring. The problem is not.

← All posts

Your big opportunity may be right where you are now.

Napoleon Hill