What Actually Breaks When You Put an LLM in Production

Every LLM postmortem I've been part of has the same shape: the model behaved roughly the way it behaved in the demo. What broke was everything built around it: the assumption that a call would return in two seconds, the assumption that output would parse as JSON, the assumption that cost would scale linearly with usage instead of with whichever user found the one prompt that generates four thousand tokens of output.

The fix is almost never a better model. It's the boring engineering that teams skip because the demo worked without it: timeouts and fallbacks for slow calls, schema validation for structured output, a hard budget per request instead of a hope, and logging detailed enough to answer 'what did the model actually see' six weeks after an incident.

Evaluation is the piece that gets skipped longest, because it doesn't block the demo. It only starts to matter once the feature is live and someone asks whether last week's prompt change made things better or worse: a question that's unanswerable without a fixed set of test cases to run both versions against.

None of this is exotic. It's the same discipline every production system needs (bounded failure modes, budgets, regression tests) applied to a component that happens to be probabilistic. Treating the model as the interesting part and the plumbing as an afterthought is exactly backwards from where the risk actually lives.