Retrieval-augmented generation gets discussed as a model technique. Almost every time I have seen one of these systems disappoint, the model was not the binding constraint. The retrieval was.
The logic is not subtle. If the right passage is not in the context, no model recovers it — it will produce something fluent instead, which is worse than nothing because it reads like an answer. And if five nearly identical passages are in the context, the answer gets muddier rather than sharper.
I keep coming back to a debugging experience that had nothing to do with RAG on paper. An agent cracked a deployment failure our team had been carrying for two weeks, and afterwards it was tempting to credit the model. But the model succeeded because the logs were detailed, the technical documentation was accurate and the knowledge base articles were written by people who cared. Change any one of those and the same model produces a confident guess. That is a retrieval story wearing different clothes.
The parts that decide quality
Chunking. Splitting on a fixed token count cuts sentences away from the thing that makes them mean anything. Split on structure instead — sections, functions, headings, rule boundaries — and what you retrieve is a unit of meaning rather than a unit of length.
What you embed versus what you return. Embedding a summary and returning the full passage often beats doing either on both. They are two different jobs: one is for finding, the other is for answering.
Recency and authority. Two documents can both be relevant and one of them can be three years obsolete. Pure similarity has no opinion about that whatsoever. Your ranking must have one, and somebody has to decide what authoritative means in your organisation, which is a political question as much as a technical one.
Evaluate retrieval on its own, first. Before judging any answer, measure whether the correct source made it into the context at all. Most disappointing systems fail right there, and no amount of prompt engineering repairs it. This is the cheapest diagnostic available and it is routinely skipped, because it is less interesting than tuning the prompt.
The unglamorous conclusion
The work is librarianship. Deciding what is worth keeping, how it is organised, what supersedes what, and which source wins when two disagree.
That was true of search long before any of this, and it stayed true. The uncomfortable implication for a team standing up a RAG system is that the hard part is not the part you can buy.