RAG Is Simple in Theory and a Systems Problem in Practice

Retrieval-augmented generation has a deceptively short description: embed your documents, retrieve the closest ones to a query, and hand them to a model as context. Every tutorial makes it look like three function calls. In practice, it's a systems problem wearing a three-function-call costume.

Chunking is the first place it gets hard. Split documents too small and you lose the surrounding context that made a passage meaningful; split them too large and irrelevant text dilutes the embedding, dragging down retrieval quality for the exact query it should have nailed. The right chunk size is a property of your documents, not a constant you can copy from a blog post.

Retrieval quality is the second place, and it's the one teams measure the least. It's entirely possible to ship a RAG system where the generation step works perfectly and the answers are still wrong, because the retrieval step never found the right passage in the first place. Without a labeled evaluation set, you can't tell which failure mode you're looking at.

The honest version of a RAG project spends more time on the retrieval evaluation harness than on the prompt. That's not the part anyone wants to hear before they start, but it's the part that determines whether the system is trustworthy enough for anyone to actually rely on it.