Evaluating LLM output quality: building evals that catch regressions before users do

01Why manual checking fails specifically here

Language models are non-deterministic and their failures are non-local. Changing one sentence in a system prompt can improve the ten cases you are looking at and degrade thirty you are not, because the change shifted behaviour across the whole input distribution rather than patching a branch. There is no compiler and no type error to catch it.

The second problem is that quality is not binary. A summary can be accurate but omit the point, or fluent but subtly wrong about a number. Teams that only test for crashes and schema validity are measuring the two things that were never at risk, and are blind to the failure mode that actually loses customers.

02Build the dataset from production, not imagination

The most common mistake is writing eval cases by hand. Hand-written cases reflect what the author thinks users do, which is a much narrower and much politer distribution than reality. Real inputs are truncated, misspelled, in the wrong language, pasted from a spreadsheet with the formatting intact, and occasionally adversarial.

Sample from logged production traffic, stratify so that rare-but-important categories are represented rather than swamped by the common case, and add every case that ever caused a complaint. That last rule is what turns the suite into a ratchet: a bug fixed once cannot silently return. A hundred well-chosen cases beat a thousand generated ones, and the cost of maintaining them is what keeps you honest about the size.

• Stratify by input type, length, language and difficulty - not uniform random.
• Every production incident contributes a permanent case.
• Keep a held-out slice you do not iterate against, or you will overfit the prompt to the eval.
• Version the dataset alongside the prompt; a score is meaningless without both.

03Pick the cheapest grader that can detect the failure

Not everything needs a model to grade it. A large share of real regressions are catchable deterministically - the output stopped being valid JSON, it lost a required field, it exceeded the length limit, it leaked the system prompt, it dropped the citation. Those checks are free, instant and never flaky, so run them first and fail fast.

Reserve model-based grading for the genuinely subjective properties: is this answer faithful to the retrieved context, does it address what was asked, is the tone right. That layering keeps cost and latency manageable, which matters because an eval suite that takes forty minutes and costs real money per run will be skipped exactly when it is most needed.

04Using a model as a judge without fooling yourself

LLM-as-judge works, but only with discipline. A judge asked to 'rate this 1-10' produces numbers that cluster around 7 and move for reasons unrelated to quality. A judge given a specific rubric with concrete criteria, asked for a binary or three-point decision per criterion, produces something you can actually track over time.

Judges also have known biases: they prefer longer answers, they prefer the first option in a pairwise comparison, and they rate their own family's output more highly. Mitigate by randomising position, keeping the rubric criteria independent, and - the step almost everyone skips - calibrating the judge against a few hundred human-labelled examples. If you have never measured your judge's agreement with a human, you do not know whether your metric measures quality or verbosity.

05Wire it into the pipeline or it will not happen

Evals that live in a notebook get run when someone remembers. The suite has to run automatically on every change to a prompt, a model version, a retrieval configuration or a tool definition, and it has to be able to fail the build. Treat the fast deterministic layer as a required check and the expensive model-graded layer as a scheduled or pre-release gate if cost demands it.

Report per-category scores rather than one aggregate. A single headline number hides the case where overall quality rose two points while the legal-document category collapsed, and that is precisely the regression you needed to see. Set the threshold on the categories that matter to the business, not on the mean.

06Close the loop with production signals

Offline evals measure what you thought to test. Production tells you what you did not, so instrument the live system for the signals that correlate with dissatisfaction: explicit thumbs-down, user rephrasing the same question, abandoning the session, escalating to a human. Those are cheap to collect and they point at gaps in the dataset.

The working loop is short. A production signal identifies a failure, the failure becomes a permanent eval case, the fix is validated against the whole suite, and the suite gets slightly better at catching the next one. Teams that run this loop for a quarter end up with something that genuinely prevents regressions; teams that build the harness and never feed it end up with a passing test suite and unhappy users.