Evals

Evals, short for evaluations, are a repeatable test suite for a model’s output. You collect a set of inputs paired with the outputs you consider correct, run the system against them, and score how close it lands. That dataset of correct answers is the point: it turns “seems good” into a number you can track across changes.

Without the dataset, a prompt or model change has no repeatable measure of whether quality improved. With it, the definition of done is explicit and owned: whoever writes the expected outputs owns what “correct” means. The number worth tracking against it is cost per accepted outcome: add the full cost of the work, retries and review included, count the outputs that met the bar, divide. That prices one resolved case, not one token.

Write the eval alongside the build, not after, and score each step rather than only the final answer, because a clean end-to-end run that produces the wrong output is the most expensive failure. Where no exact answer exists, an LLM judge scores the output against a written rubric, and the rubric gets the same care as the dataset.