The feature performed well through development on the questions the team tried. In production it confidently returns incorrect answers, and users notice. The model did not change - the input distribution did. Test questions were written by people who knew the answer and the data; real questions come from people who do not.
01Retrieval is usually the failure, not generation
In a retrieval-augmented system, a wrong answer most often means the right document was never supplied. The model produced a reasonable response from irrelevant context, which reads as a hallucination while actually being a search problem.
Evaluate retrieval independently. For a set of real questions, check whether the passage containing the answer appears in what was retrieved. If it does not, no amount of prompt refinement will help, and effort spent there is wasted.
| Check | If it fails | Fix |
|---|---|---|
| Was the right passage retrieved? | Retrieval problem | Chunking, embeddings, hybrid search |
| Was it in the context window? | Truncation problem | Ranking, context budget |
| Did the model use it? | Prompt problem | Instructions, grounding constraints |
| Does the source document exist? | Content gap | Add the content or say you cannot answer |
| Is the source outdated? | Freshness problem | Reindex, add recency handling |
02Chunking decides what can be found
Documents split at arbitrary character counts cut sentences and separate context from statements. A chunk containing a conclusion without the condition it depends on retrieves well and produces confidently wrong answers, because the qualification was in the neighbouring chunk.
Split on structure - sections, headings, paragraphs - and include a small overlap so context spanning a boundary is not lost. Attaching document title and section heading to each chunk gives both the retriever and the model information they otherwise lack.
03Build an evaluation set from real questions
Demo questions are unrepresentative because their authors know what the system can answer. Collect actual user queries, including the ones that failed, and construct a set of a few hundred with verified expected answers. That set is the only reliable measure of whether a change improves things.
Without it, prompt changes are evaluated by trying a handful of examples, which reliably produces improvement on those examples and regression elsewhere. The evaluation set converts subjective assessment into a number you can compare across versions.
| Category | Why |
|---|---|
| Questions with no answer in the corpus | Should decline, not invent |
| Ambiguous questions | Should ask rather than guess |
| Questions needing multiple documents | Tests synthesis |
| Near-duplicate documents | Tests ranking |
| Time-sensitive questions | Tests freshness handling |
| Adversarial or injected instructions | Tests safety boundaries |
04Make declining an acceptable answer
Models default to producing a plausible response. If the retrieved context does not contain the answer, the correct behaviour is to say so, and that requires explicit instruction plus evaluation cases that reward it.
Requiring citations makes this verifiable. When every claim must reference a retrieved passage, unsupported statements become visible to the user and to your own automated checks, which is more reliable than instructing the model not to speculate.
05Log everything and review it weekly
Store the question, the retrieved passages with their scores, the prompt, the response and any user feedback. Without that record, a report of a wrong answer cannot be investigated - you cannot reconstruct what the model was given.
Review a sample every week. Patterns emerge quickly: a category of question that consistently retrieves poorly, a document that should exist and does not, a phrasing that confuses the retriever. Those observations are what actually improve the system over time.
Topics
Priya Iyer
Staff Engineer · SyncTrix
Writes about the engineering decisions behind production systems - architecture, delivery and the trade-offs that only show up at scale.
Building something like this?
SyncTrix engineers AI, SaaS, platform and cloud systems for enterprises and high-growth teams. Tell us what you're shipping and we'll scope it with you.
Talk to an engineer