How to evaluate an AI feature before you ship it
Evaluation for AI features means a fixed set of real inputs with expected outputs, run on every change. Starting with fifty hand-labelled cases drawn from actual usage, measuring the failures rather than the average, and adding every reported bug to the set builds a harness that makes prompt changes safe.
The usual way an AI feature is assessed before launch is that somebody tries it a few times and it seems good. Then it ships, and the failures arrive from users, one screenshot at a time.
The fix is unglamorous and effective: a fixed set of real inputs with known-good outputs, run on every change.
Build it from real usage
Fifty genuine cases are worth more than a thousand invented ones, because real inputs contain the things nobody would think to invent — the typos, the ambiguity, the question asked halfway through a sentence.
Pull them from logs, support tickets, or a short internal trial. Label the expected outcome by hand. It takes an afternoon and it is the highest-leverage afternoon in the project.
Cover the shape of the distribution
Include the common cases, the known-hard ones, and the ones that should be refused. A set made only of easy questions reports a high score that means nothing.
Read the failures, not the average
"87 percent" tells you almost nothing actionable. The thirteen percent that failed usually cluster — one phrasing, one document type, one edge in your data — and the cluster is the actual finding.
Look at each failure individually at least once. That is where the next prompt change comes from.
Make it grow
Every bug report is a case you did not have. Adding it to the set takes a minute and means that failure is checked forever after.
An eval set that is not growing is slowly becoming irrelevant to what your users actually do.
Run it like a test suite
On every prompt change, every model change, every retrieval change. Prompt edits feel weightless — a sentence here, a word there — and they are not: a small rewording can move accuracy several points in either direction.
Without a harness, nobody knows which direction. With one, a prompt change stops being a leap of faith and becomes an ordinary change with a number attached.
Frequently asked questions
How do I evaluate open-ended output with no single right answer?
Score properties rather than exact text: did it cite a real source, stay within length, keep the required format, avoid the claims it must not make. For subjective quality, a model scoring against a rubric works reasonably — validate its scores against human judgement on a sample first.
Is this not just testing?
Same idea, different tolerance. You are not asserting exact equality but measuring a rate, and a change that moves it from 84 to 79 percent is a regression worth investigating even though nothing threw.
