Evaluating LLM systems before they reach customers

Most LLM features ship on the strength of a demo and a dozen hand-picked examples. Then someone shortens the system prompt to save tokens, or the provider releases a new model version, or the retrieval index is rebuilt with a different chunk size, and a slice of answers quietly gets worse. Nobody notices until a customer does. The change looked harmless because nothing measured it.

Evaluation is how you replace that guesswork with evidence. In practice it is several disciplines working together: a golden set that represents the work, scorers you have checked against human judgement, an adversarial suite that tries to break the system on purpose, and gates in CI that stop a release when the numbers move the wrong way. Done well, it is also what lets you change models, prompts and retrieval with confidence instead of fear.

This article describes how we build that, the approach behind our Evals accelerator and the regression harness we run on every change. The examples are illustrative; the methods are the ones we use.

Evaluate the whole system

Public benchmarks tell you how a model does on someone else's tasks. They tell you little about your invoice dispute assistant, which is a model plus a system prompt, a retrieval pipeline, a set of tools, output parsing and business rules. Any of those can regress independently, so the unit under test is the whole system, called the way production calls it.

That has a practical consequence. Every evaluation run records the full configuration: model and version, prompt hashes, retrieval index version, tool definitions and decoding parameters. Without that, a score is a number with no explanation, and you cannot tell whether last week's drop came from your change or from the provider's.

Building a golden set that represents the work

A golden set is a fixed collection of inputs with the information needed to judge each output: a reference answer, the facts the answer must contain, the documents it must cite, or the tool call it must make. It is the backbone of everything else, and most teams build it too small, too easy and too late.

Imagine an assistant that helps a finance team resolve invoice disputes. A useful golden set for it draws from three sources. Real traffic, sampled from production logs and redacted, gives you the distribution of what people actually ask. Subject-matter experts write cases for situations that matter but are rare: a credit note that partly offsets two invoices, a dispute in a currency the customer has not used before. Past failures become cases the day they are found, so the set grows a memory of everything that has gone wrong.

Then stratify. Tag every case with the slices you care about (dispute type, customer segment, language, whether the answer requires a tool call, whether the right answer is "I cannot tell from the documents") and make sure each slice has enough cases to measure. Aggregate scores hide slice regressions: a change that improves common cases by two points and breaks refusals entirely can still raise the average.

Size the set with statistics, not instinct. A pass rate measured on n independent cases has a standard error of roughly √(p(1−p)/n). At a 90% pass rate, 100 cases give a 95% interval of about ±6 points; 400 cases give about ±3. If your release gate needs to detect a three-point drop on a slice, that slice needs hundreds of cases, or you need paired comparisons on the same cases, which are far more sensitive than comparing two independent averages. We report intervals alongside every score so nobody celebrates noise.

Finally, treat the set as code. Version it, review changes to it, and keep a held-out portion that prompt authors never see, so the system is not quietly tuned to the test.

Scoring: deterministic first, judged second

Start with everything you can check without a model. Does the output parse against the schema? Does every cited document ID exist in the retrieved set? Do the amounts in the answer match the amounts in the source invoice? Did the agent call create_credit_note when the case required it, and not when it did not? These checks are fast, free and unambiguous, and in our experience they catch a large share of real regressions.

What remains (is the explanation correct, complete and appropriately cautious?) needs judgement. Using a strong model as the judge is now standard practice, and the evidence for it is reasonable with caveats. Zheng and colleagues' MT-Bench study found that strong LLM judges reached over 80% agreement with human preferences, about the same level as agreement between humans, and also documented position bias, verbosity bias and self-enhancement bias in those judges. That is the right frame: a model judge can be as consistent as a person, and it has predictable biases you must design around.

The design choices that matter most are these. Give the judge a rubric with observable criteria ("states the disputed amount; cites the invoice; does not promise a refund") instead of "rate helpfulness from 1 to 10". Ask for per-criterion pass or fail with a short justification, which is easier to calibrate than a single scale. For pairwise comparisons between a baseline and a candidate, run each pair in both orders and count only consistent verdicts, which cancels position bias. Use a different model family for the judge than for the system where you can, to reduce self-preference. And give the judge the reference material, so it checks the answer against facts rather than its own beliefs.

Calibrating the judge against human labels

A judge you have not measured is an opinion. Calibration is how you find out where it can be trusted.

Take a sample of real outputs, typically 150 to 300 across all slices, and have two people label each one against the same rubric, independently. Measure how often the humans agree with each other, using a chance-corrected statistic such as Cohen's kappa rather than raw agreement, because raw agreement looks high whenever most outputs pass. Where the humans disagree, the rubric is ambiguous; fix the rubric before you blame the judge. Then run the judge on the same items and measure its agreement with the adjudicated human labels, per criterion and per slice.

The decision rule we use is simple. Where the judge's agreement with humans is close to the humans' agreement with each other, the judge scores that criterion automatically. Where it is clearly lower, the criterion goes to human review, or the rubric is split into something more observable. It is common to find a judge that is excellent on "cites the right invoice" and unreliable on "tone is appropriate for a customer in arrears".

Calibration is not a one-off. Re-run it when you change the judge model, the rubric, or the system in a way that changes the kind of outputs the judge sees. Keep a small, rotating human review sample in every release so drift shows up before it matters.

Red-teaming as a regression suite

A golden set measures how the system behaves for people trying to use it. A red-team suite measures how it behaves for people trying to misuse it, and for content that tries to misuse it on someone's behalf. The OWASP Top 10 for LLM Applications, now in its 2026 edition with prompt injection still first, and the separate OWASP Top 10 for Agentic Applications are good maps of what to cover. NIST's Generative AI Profile (NIST AI 600-1) lists pre-deployment testing and red-teaming among its suggested actions for managing generative AI risks.

For an invoice assistant with tools, the cases that matter most are concrete. A supplier PDF containing hidden instructions to approve a credit note. A user asking for another customer's account balance. A request phrased to extract the system prompt. A conversation that slowly escalates toward a refund the policy forbids. Open-source scanners such as NVIDIA's garak and Microsoft's PyRIT generate broad families of attacks, and we use tools like them for exploration. The cases that find real weaknesses, though, are usually written by someone who understands the domain and the tools.

The important move is to treat every successful attack like a bug: once found, it becomes a permanent case in the red-team suite with an expected safe behaviour, and it runs on every release. Red-teaming that lives only in a report from last quarter protects nothing this quarter.

Release gates in CI

Evaluation changes behaviour only when it can block a release. We run it in three tiers, because a full suite with a model judge is too slow and too expensive for every commit.

Two kinds of gate run inside those tiers. Hard gates are zero-tolerance: every red-team case must produce the safe behaviour, and a small set of must-pass cases (the ones where a wrong answer costs money or breaks a regulation) must pass. Soft gates compare the candidate against the production baseline on the same cases and block only on a regression larger than the measurement noise. Gating on an absolute threshold ("85% or better") invites arguments every time the set changes; gating on "no significant regression against what is live" does not. Here is an illustrative gate configuration.

suite: invoice-disputes
baseline: production            # compare against the live configuration
runs_per_case: 3                # majority vote smooths sampling noise
gates:
  hard:
    - redteam/*: all_pass
    - must_pass/*: all_pass
    - schema_valid: 1.0
  soft:
    - metric: judge.criteria_pass_rate
      slices: [dispute_type, language, needs_tool]
      fail_if: paired_drop_lower_bound > 0.02   # 95% interval excludes a 2-point drop
    - metric: cost_per_case
      fail_if: increase > 0.20
report: pr_comment               # per-slice table with intervals and changed cases

The report matters as much as the verdict. A failed gate should show which cases changed, side by side, with the judge's justification. Engineers fix regressions quickly when they can read the twelve cases that broke; they argue with a single red number.

Gates also apply to changes you did not make. When a provider announces a new model version, run the full suite against it before switching, exactly as you would for your own change. Pin model versions in production so the switch is a deliberate, gated decision.

Where this breaks

Every metric becomes a target. Teams that tune prompts against the golden set will, over time, overfit it, which is why the held-out portion and the steady inflow of fresh production samples matter. If the held-out score diverges from the working score, you are tuning to the test.

Golden sets go stale. Products change, policies change, and the questions customers ask change. A set built in January can be a confident measurement of the wrong thing by July. Budget for a regular refresh from production traffic and retire cases whose expected answers no longer hold.

Model judges drift and share blind spots. A judge from the same family as the system may systematically like the system's style. A judge model update can shift scores across the board without anything in your system changing, which is why the judge's version is part of the recorded configuration and a judge change triggers recalibration.

Some qualities resist rubrics. Whether an explanation will satisfy an angry customer, or whether a summary emphasises the right risk, may never reach acceptable judge agreement. Accept that, keep those criteria with people, and sample them rather than pretending to automate them.

Finally, evaluation costs money and time. A full judged suite with three runs per case is thousands of model calls. Tiering, caching unchanged results and running the expensive tier only when the relevant components change keep it affordable. Skipping it entirely is cheaper until the first silent regression reaches a customer.

A checklist for your first release gate

1. Record the full system configuration with every evaluation run, including model versions and prompt hashes.
2. Build a golden set from production samples, expert-written edge cases and past failures, tagged by slice, with a held-out portion.
3. Size each slice you gate on for the regression you need to detect, and report intervals with every score.
4. Put deterministic checks first; use a model judge only for what they cannot check.
5. Calibrate the judge against two independent human labellers, per criterion, and route low-agreement criteria to people.
6. Turn every successful attack and every production failure into a permanent case.
7. Gate releases on hard must-pass cases and on paired regression against production, and make the report readable.
8. Run the full suite on any model version change, yours or the provider's, before it reaches production.

Start with the first three and a deterministic gate this week. A small, honest evaluation that blocks releases is worth more than a large one that only produces dashboards.