Nobody’s RAG system fails at the model. It fails at retrieval.
Wiring up retrieval-augmented generation takes an afternoon. Making it return good answers is the actual project — and almost all of that work happens before the model is involved.
Retrieval-augmented generation has a reputation for being straightforward, and the first version genuinely is. Split some documents, embed them, put them in a vector index, retrieve on a question, hand the results to a model. You can have something answering questions the same day.
Then people start using it, and about a fifth of the answers are subtly wrong or unhelpfully vague, and the instinct is to go and improve the prompt. That is almost always the wrong place to look.
The model can only work with what it is given
If the passage containing the answer was not retrieved, no amount of prompt engineering will produce a correct answer. The model will do exactly what you asked it to do — answer from the provided context — with context that does not contain the answer.
This is why the first thing worth measuring is not answer quality at all. It is whether the correct passage appeared in the retrieved set. Those are two different failures with two different fixes, and conflating them is how teams spend weeks tuning prompts for a retrieval problem.
Chunking is a modelling decision, not a parameter
The single largest quality lever I have found is how documents get split. Fixed-size chunking is the default in every tutorial and it is fine for uniform prose, but most real corpora are not uniform prose. They are structured documents where a section is a unit of meaning and a table is meaningless once split down the middle.
Chunking along the document’s own structure — sections, clauses, question-and-answer pairs, whatever the material actually uses — consistently outperforms chunking by character count, because it preserves the thing that makes a passage answerable on its own.
The related question is how much surrounding context each chunk carries. A clause that says “this does not apply in the circumstances described above” is worse than useless in isolation.
Metadata does more work than people expect
Attaching source, section, document type and date to every chunk costs almost nothing at ingestion and pays back repeatedly. It lets you filter before you search, which is often far more effective than trying to make semantic similarity do that job. It lets you prefer recent material over superseded material. And it is what makes citation possible, which is what makes people trust the system.
Semantic similarity is not relevance
Embedding search finds passages that are about the same subject. That is related to, but not the same as, passages that answer the question — which is why retrieving more candidates and then re-ranking them tends to beat retrieving fewer and trusting the order.
It is also why questions containing an exact term — a policy number, a product code, a proper noun — often do worse than vague ones. Semantic search will happily return ten passages about the general topic and miss the one containing the literal string. Combining keyword and vector retrieval fixes a whole class of complaints that otherwise look inexplicable.
“I do not know” has to be a designed behaviour
A system that answers everything is a system that invents things when retrieval fails. Refusing to answer when the retrieved context does not support one has to be specified, prompted for, and then actually tested — including with questions you know the corpus cannot answer.
This is the behaviour that decides whether people keep using the thing. One confidently invented policy will do more damage to adoption than fifty slightly clumsy correct answers.
Build the evaluation before you need it
You need a fixed set of questions with known good answers, and you need it early. Without it every change is a matter of opinion, and you end up in a loop where each fix appears to help and quality never actually improves.
It does not have to be elaborate. Thirty to fifty real questions from the people who will use the system, with the passage that should answer each one, is enough to tell you whether last week’s change was an improvement or just a difference.
The short version
Choosing a different model is the easiest thing to try and rarely the thing that helps. Chunking, metadata, hybrid retrieval, re-ranking and honest refusal are where the quality actually comes from — and all but the last of those happen before the model sees anything at all.