Done Is a Function You Write

The new unit test, for AI that thinks.

Your large language model crushed the benchmark. Then it shipped, and the support tickets started. Sound familiar? You stopped trusting public leaderboards the day a contamination-resistant test dropped your "verified" score by thirty-five points overnight. And you realized the number you'd been steering by was noise. So now you're flying blind: a nondeterministic system that's brilliant on the demo, wrong 30% of the time at scale, and impossible to regression-test the way you test real code.

This is the field manual for LLM evaluation: the discipline that quietly became the center of AI engineering, where you write the tests that decide what a thinking machine is allowed to do. Its core move is a reframe: "done" is not a vibe or a leaderboard rank but a function you write, and once you can write that check, you can delegate to the machine everything that passes it. Treating evals as test-driven development for generative AI, it starts where the real failures are, your own traces, and ends with a suite that blocks silent regressions on every deploy. Along the way you'll align an LLM-as-judge to human labels, debug judge bias, defeat benchmark contamination, and separate capability from reliability.

The payoff is the scarce skill no model upgrade erases: not writing code, but writing the checks that let you delegate with confidence. And finally ship AI you can stand behind.

This is hands-on AI engineering, not theory. If you build with large language models and need to know they work before users do, this is the discipline that gets you there.

Read chapter one free with Look Inside.

Part of the Build Agents You Can Trust series, in The Verifier's Library.