Evaluating AI Systems

Looking good on one example is weaker evidence than it feels like

Prompting as Interface Design ended on this exact problem: a change that looks right on the case in front of you can quietly break three others you weren't looking at. With LLM output specifically, that risk is worse than it is for ordinary software, because a wrong answer rarely looks wrong. It reads fluently, confidently, in the same voice as a correct one.

A traditional bug tends to announce itself, a stack trace, a null pointer, an obviously malformed response. A subtly wrong summary, a hallucinated field, an edge case handled with the wrong tone, all of these pass a glance. "It looks fine" from a person skimming a playground is a real signal and a weak one, exactly proportional to how many cases they actually looked at, which for a quick check is approximately one.

None of this means the instinct to check output before shipping is wrong. It means checking has to survive being repeated automatically, at scale, against more cases than a person will patiently reread every time something changes.

A fixed set of cases, checked the same way every time

A golden dataset is a small, deliberately chosen, versioned set of representative inputs, the ordinary case, the known-tricky edge case, the one a real user actually hit last month, each with an expected output or an expected property of one.

1GOLDEN_SET = [
2    {"input": "How do I reset my password?", "expect_contains": "reset"},
3    {"input": "What's your CEO's salary?", "expect_not_contains": "salary"},
4    {"input": "", "expect_contains": "provide"},  # empty input handled gracefully
5]
6
7def run_eval(prompt_fn):
8    results = []
9    for case in GOLDEN_SET:
10        output = prompt_fn(case["input"])
11        passed = check_case(case, output)
12        results.append({"input": case["input"], "passed": passed, "output": output})
13    return results

Run this after every prompt or pipeline change, the same way a test suite runs after every code change, and "did this break something" stops being a question answered by memory and vibes. It's answered by a report that names exactly which case regressed, the same guarantee a unit test gives a function, just checking a fuzzier kind of output.

For output with no single correct answer, a second model call can grade it

expect_contains works for facts and format. It doesn't work for judging whether a summary is genuinely good, whether a tone is appropriately warm, whether an open-ended answer actually addressed the question, there's no fixed string to check for. The common approach is having another model call score the output against a rubric.

1judge_prompt = f"""Rate this summary from 1-5 on accuracy and conciseness.
2Respond with only a number.
3 
4Original: {original_text}
5Summary: {summary}"""
6
7score = int(client.chat.completions.create(
8    model="gpt-4o", messages=[{"role": "user", "content": judge_prompt}],
9).choices[0].message.content)

Treat the judge's score as a useful signal, not ground truth. LLM judges have measurable biases of their own, a documented tendency to score longer answers higher regardless of actual quality is one of the most common, and a judge prompt inherits the same specification problem as any other prompt in this series: a vague rubric produces inconsistent scores for the same reason a vague summarization prompt produces inconsistent summaries.

Calibrate it before trusting it: run the judge against a handful of outputs a person has already scored by hand, and check that the two rankings actually agree before wiring the judge into an automated pipeline that gates real changes.

An average score can hide the one case that actually regressed

Wiring the golden set and the judge together into an actual workflow: every prompt or pipeline change runs against the full set before shipping, and the resulting scores get compared case by case against the previous version, not just averaged into one number.

An aggregate average is exactly the kind of number that hides the failure this whole article is about. A change that improves nine cases and badly breaks a tenth can still raise the average, and shipping on the strength of that average means shipping a real regression nobody noticed because it was outvoted.

A regression on any individual case should block the change the same way a failing unit test blocks a merge, with a human deciding whether the regression is acceptable, not a script deciding for them by rolling everything into one pass or fail number. The value of the golden set was never the average, it was always the ability to point at the specific case that broke.