LLM Evaluation Harness for Production Systems

The problem

An LLM feature ships because it looked good. Someone tried fifteen prompts by hand, the outputs were reasonable, and the demo went well.

Three weeks later a prompt is edited to fix an unrelated complaint, or a model version is upgraded, or a chunking parameter is changed during a performance push.

Something else gets quietly worse.

Nobody notices, because there is no mechanism that could notice, until a user reports it or a number in a board pack looks wrong.

This failure is structurally different from a normal software bug.

Conventional code fails loudly: an exception, a stack trace, a failed test.

A language model fails by being plausible and wrong. The output is well-formed, confident, correctly formatted, and incorrect in a way that a glance will not catch and that a human reviewer will only catch if they happen to know the right answer already.

The underlying issue is that “working” was never defined numerically.

Without a number, there is no way to detect that the number has moved.

Every subsequent conversation about quality becomes a conversation about impressions, and impressions do not survive contact with a stakeholder who wants to know whether last week’s change helped.

When to use this

• An LLM system is going to production, or is already in production and being changed.
• More than one person can modify prompts, models, retrieval parameters or the underlying data.
• The output feeds a decision that matters — a customer response, a document classification, or a figure someone acts on.
• You operate in a regulated or risk-managed environment where someone will eventually ask for evidence of quality.
• The system will run long enough for its inputs to change, which is nearly always.

When not to use this

A genuine throwaway prototype

A prototype with a known short life and no production path does not necessarily justify an evaluation harness.

Building one for a two-week demo is cost without benefit. Be honest about whether it is actually throwaway; most things described that way are not.

No agreement on what “correct” means

An evaluation harness measures agreement against a definition.

Where the relevant experts disagree about the right answer, building the harness first will produce a precise measurement of the wrong thing.

Resolve the definition, on a sample of real cases, before writing any scoring code.

Fewer than roughly fifty representative examples available

If there are fewer than roughly fifty representative examples available, with no practical route to more, the resulting scores may be too noisy to support confident conclusions.

Synthetic data helps with edge cases but does not substitute for a real distribution.

The task has a single deterministic correct answer

Arithmetic, a database lookup or a schema validation can be checked directly.

Assert the answer.

Adding an LLM-as-judge layer here introduces cost, latency and another source of error without solving a real evaluation problem.

Architecture

The golden dataset is the first deliverable, produced before the system it evaluates.

It is a set of representative inputs with expected outputs or acceptance criteria, built with whoever actually owns the definition of correct.

This ordering is the whole pattern: writing the evaluation set first forces the definition of success into the open at the point where it is cheap to argue about, rather than in month four when the argument is expensive and political.

Scoring runs in layers, cheapest first.

Deterministic checks catch what they can — required fields present, schema valid, citation resolves to a real document, no forbidden content, latency and token count inside budget.

Only what survives those checks goes to a model-based judge, which is slower, costs money, and is itself a component with error characteristics that need to be measured.

In CI, the aggregate score becomes a gate: a change that lowers it beyond the accepted threshold fails the build.

In production, a sample of live traffic runs through the reference-free subset of the same metrics, producing a trend line that makes slow decay visible rather than gradual.

Components

Tradeoffs

Implementation notes

Calibrate the judge before trusting it

Have a human score a sample that the judge has also scored, and measure agreement.

A judge that disagrees with your domain expert is measuring something, but not what you intended, and every downstream decision inherits that error.

Re-calibrate when the judge model is upgraded, because a model upgrade silently changes your measuring instrument.

Test known judge failure modes

Known judge failure modes worth testing for specifically include preference for longer answers regardless of quality, preference for outputs from the same model family as the judge, and sensitivity to the order in which candidates are presented.

Randomise presentation order as a matter of course.

Pin the judge model version

Pin the judge model version explicitly.

An evaluation suite whose judge auto-upgrades produces score movements with no corresponding code change.

That is one of the most confusing possible failures in this system: the metric moved, nothing shipped, and the cause is invisible.

Keep two thresholds, not one

Use a warning threshold that reports and a failure threshold that blocks.

A single hard gate set tightly gets disabled within a month by someone under deadline pressure.

A disabled gate is worse than a warning nobody set up, because the team believes it is protected.

Feed production failures back into the golden dataset

Make this a routine step, not a project.

Every real failure that the offline suite missed is direct evidence of a gap in the suite.

This feedback loop is what stops the dataset ossifying around the failure modes the team imagined at the start.

Budget for the harness honestly

Building the evaluation harness well is a meaningful fraction of the total build effort.

Pretending otherwise leads to it being cut first when the schedule tightens — which is precisely when it becomes most valuable.

References

• DeepEval documentation — an open-source evaluation framework with pytest-style assertions, a large metric library and CI integration. The documentation is a useful practical introduction to the metric landscape regardless of which tool you use.
  DeepEval Documentation
• DeepEval: RAG evaluation guide — maps individual metrics to the system parameters they diagnose, which is the part many teams get wrong.
  DeepEval RAG Evaluation Guide
• LLM evaluation frameworks compared — a breakdown of model benchmarking, application evaluation and production monitoring.
  Machine Learning Mastery
• Ragas vs DeepEval, compared — a technical comparison covering judge design and CI integration, including the case for running both.
  QA Skills