Debugging RAG: why your retrieval is wrong before your model is

01Separate the two failure modes first

A wrong answer given correct context is a generation problem: the model had what it needed and produced something else. A wrong answer given missing or irrelevant context is a retrieval problem, and the model behaved reasonably given what it was handed.

Score them separately or you will spend weeks tuning the wrong layer. The check is mechanical - for each failing case, look at what was actually retrieved and ask whether a careful human could have answered correctly from it. Teams that add this single step stop guessing almost immediately.

02Chunking causes more failures than embeddings

Fixed-size chunking splits documents mid-sentence, mid-table and mid-clause. A chunk that starts halfway through a definition and ends halfway through the next is semantically incoherent, and no embedding model can rescue it.

Structure-aware chunking - respecting headings, keeping tables intact, preserving list items with their introduction - improves retrieval more than switching embedding models in most systems we have worked on. Overlap between chunks helps too, at the cost of index size.

• Never split tables, code blocks or clauses mid-structure.
• Keep headings with the content they introduce.
• Use overlap so context at chunk boundaries is not lost.
• Store the source document and position for every chunk.

03Vector search alone misses exact matches

Embeddings capture semantic similarity, which is exactly wrong for identifiers. A query for a specific error code, SKU, version number or surname returns semantically similar things rather than the exact one - and those are precisely the queries users type when they want a specific answer.

Hybrid search fixes this: run keyword search alongside vector search and fuse the rankings. The improvement is usually larger than any gain from a better embedding model, and it is the first thing to add when exact-match queries are failing.

04Reranking is the cheapest large win

Vector search returns plausible candidates; a reranker orders them by actual relevance to the query. Retrieving a wider candidate set and reranking down to the top few consistently improves answer quality without touching the model or the prompt.

It costs an extra call and some latency. In most systems that is a good trade, because the alternative is passing marginally relevant context to the model and hoping it picks correctly - which is precisely where confident wrong answers come from.

05Permissions must be part of retrieval

Retrieval that ignores permissions will eventually surface something a user should not see, and it will do so in a generated answer where the provenance is unclear. Filtering after retrieval is not sufficient - it can return far fewer results than requested, or none, in ways that look like a quality problem.

Filter within the search itself, and test with real tenancy and permission shapes rather than generic filters. This is one of the few RAG issues that is a security incident rather than a quality complaint.

• Apply permission filters inside the query, not after it.
• Test with real tenant and role shapes, not synthetic ones.
• Handle the empty-result case explicitly and honestly.
• Never embed restricted content into a shared index.

06Build the evaluation set from real questions

Sample actual user queries, have someone knowledgeable write the correct answer and identify which documents contain it, and keep that set in version control. A hundred real examples beat a thousand synthetic ones, because synthetic questions share the blind spots of whoever generated them.

Then measure retrieval precision and answer accuracy separately, and re-run both on every change to chunking, embeddings, retrieval or prompts. Without that, every change is a guess and regressions are invisible until users complain.