A retrieval prototype over a curated document set is genuinely easy now, and that ease is misleading. The failures that matter appear at scale, with real permissions, real document churn and real users asking questions the corpus does not answer. None of them are model problems, which is why prompt engineering does not fix them.
01Permissions are a retrieval problem, not a filter
The most consequential failure is a user receiving an answer synthesised from a document they cannot access. Filtering results after retrieval is not sufficient: the model has already seen the content if it was in context, and top-k retrieval that returns documents the user cannot see means the accessible answer never makes it into the window at all.
Permissions must be applied inside the vector search, as a pre-filter, using the same identity model as the source system. That also means permission changes need to propagate to the index promptly, because a stale ACL in a vector store is a data leak with a delay.
| Failure | Looks like | Real cause |
|---|---|---|
| Cross-permission leak | User sees restricted content | Post-filtering instead of pre-filtering |
| Confidently wrong answer | Model hallucination | Retrieval returned nothing relevant |
| Stale answers | Model is out of date | Index not updated on source change |
| Good demo, bad production | Quality regression | Corpus grew, chunking assumptions broke |
| Cannot answer aggregate questions | Model limitation | Retrieval cannot do 'how many' |
02Retrieval failure looks exactly like hallucination
When a user reports a wrong answer, the instinct is to blame the model. Far more often the retrieval step returned nothing useful and the model did what it does with an empty context: produced something plausible.
Log the retrieved chunks alongside every answer. That single change makes the distinction obvious and redirects effort correctly - to chunking, embeddings or query rewriting rather than to prompt tuning that cannot fix a context that never contained the answer.
03Aggregate questions are the wrong shape entirely
'How many customers in Bengaluru upgraded last quarter' cannot be answered by retrieving five relevant chunks. Semantic search finds documents that resemble a query; it does not count, sum or group. Users will nonetheless ask these questions constantly, because the interface invites them.
The answer is routing, not better retrieval. Classify the query, send aggregate questions to a SQL layer over your actual data, and send document questions to retrieval. Systems that try to do everything through the vector store answer numerical questions confidently and wrongly.
| Question type | Route to | Why |
|---|---|---|
| What does the policy say about X | Vector retrieval | Semantic match on documents |
| How many / total / average | SQL over the warehouse | Retrieval cannot aggregate |
| What changed between versions | Structured diff | Needs ordered comparison |
| Show me the document about X | Keyword plus semantic | Exact terms matter |
04Keeping the index honest
Documents change, get deleted and get permissions revoked. An index refreshed nightly is answering with yesterday's truth, which is fine for a policy handbook and unacceptable for anything operational.
Update on change rather than on a schedule where it matters, and treat deletion as urgent - a deleted document that stays in the index is both wrong and potentially a compliance issue. Track index freshness as an operational metric with alerting, in the same way you would track replication lag.
05Evaluate before and after every change
Without an evaluation set, every change to chunking, embeddings or prompts is a guess, and quality drifts in both directions without anyone noticing. A set of a few hundred question-and-expected-source pairs, run in CI, catches regressions before users do.
Measure retrieval and generation separately. Retrieval quality is whether the right source appeared in the top results, which is objectively checkable. Generation quality is whether the answer was faithful to the retrieved context. Conflating them means you cannot tell which half broke.
Topics
Aarav Patel
Principal Engineer · SyncTrix
Writes about the engineering decisions behind production systems - architecture, delivery and the trade-offs that only show up at scale.
Building something like this?
SyncTrix engineers AI, SaaS, platform and cloud systems for enterprises and high-growth teams. Tell us what you're shipping and we'll scope it with you.
Talk to an engineer