A conventional service fails loudly - errors, timeouts, saturation. An LLM feature fails quietly. It returns a confident, well-formatted, plausible answer that happens to be wrong, and every dashboard stays green. Latency is fine, error rate is zero, and your users are being misinformed. Observability for these systems has to measure something dashboards were never designed to capture.
01Trace the whole chain, not the model call
The model call is one span among many, and rarely the interesting one. A single user request might rewrite the query, retrieve twenty documents, rerank them, call two tools, and generate a response. When the answer is wrong, you need to know which link broke - and an aggregate latency number tells you nothing about that.
Capture the full chain as a trace with the inputs and outputs of every step. Record retrieved document identifiers and scores, tool arguments and results, the prompt template version, model and parameters, and token counts. The retrieval step is the most common culprit and the one teams instrument last.
- One trace per user request, spanning every step including retrieval and tools.
- Store retrieved document IDs and relevance scores, not just the final answer.
- Record prompt template version and model version on every span.
- Link traces to user feedback so complaints resolve to a specific execution.
02Evals are your regression tests
You would not ship a refactor without running the test suite. Prompt changes, model upgrades, retrieval tuning and chunking changes are all refactors of a system whose behaviour you cannot verify by reading the diff. An evaluation set is the only mechanism that makes those changes safe.
Build it from real traffic. Sample actual user queries, have someone knowledgeable write the correct answer and identify the correct sources, and keep it in version control alongside the code. A hundred well-chosen real examples beat a thousand synthetic ones, because synthetic queries share the blind spots of whoever generated them.
03What to measure in production
Offline evals catch regressions before deployment. Production monitoring catches the drift they miss - changing user behaviour, changing source data, silent provider-side model updates. The two are complementary and you need both.
| Signal | What it catches | How to collect |
|---|---|---|
| Retrieval empty rate | Corpus gaps, broken indexing, over-tight filters | Log when retrieval returns nothing or scores below threshold |
| Refusal / fallback rate | Prompt regressions, model changes, bad input | Classify responses; alert on rate change |
| Explicit user feedback | Real quality as perceived by users | Thumbs up/down wired to the trace ID |
| Implicit signals | Dissatisfaction users never report | Rephrase rate, abandonment, copy events |
| Output validation failures | Schema drift, truncation, format breakage | Validate structured output; log failures |
04Version everything that affects output
When quality drops, the first question is what changed. That question is unanswerable unless prompts, retrieval configuration, model identifiers and tool definitions are all versioned and recorded per request. Prompts living as string literals edited directly in production are the single most common reason teams cannot debug their own regressions.
Treat prompts as code: version controlled, reviewed, deployed deliberately, and pinned per request in your traces. Pin model versions explicitly rather than following a floating alias, so a provider-side update is a change you schedule rather than one you discover through a support ticket.
05LLM-as-judge, with appropriate scepticism
Using a model to grade another model's output scales evaluation past what humans can review, and it is genuinely useful for relative comparisons - did this prompt change make things better or worse. It is much weaker as an absolute quality measure, because judges are biased toward verbose, confident, well-formatted answers regardless of correctness.
Calibrate the judge against human labels on a sample before trusting it, and re-calibrate whenever you change the judge model. Use it to triage and prioritise what humans review, not to replace that review entirely.
Topics
Priya Iyer
Staff Engineer · SyncTrix
Writes about the engineering decisions behind production systems - architecture, delivery and the trade-offs that only show up at scale.
Building something like this?
SyncTrix engineers AI, SaaS, platform and cloud systems for enterprises and high-growth teams. Tell us what you're shipping and we'll scope it with you.