· 9 min read · Noor Haddad — Founding engineer, Trellis
Every agent team we meet has the same origin story. The demo was magic, the pilot was promising, and then week three of production arrived: a prompt tweak fixed one customer's complaint and silently broke two workflows nobody was watching. The team's quality bar at that moment was, honestly, "someone would probably notice." That is not a quality bar. An agent doesn't have a quality bar until it has a regression suite — a set of cases it must pass before any change ships.
Your eval set is already in production
Teams stall on evals because they imagine writing hundreds of test cases from scratch. Don't. Your production traces are the test cases. Every run your agent has ever executed is a recorded scenario: real inputs, real tool calls, a real outcome someone either accepted or escalated. Growing a suite is a triage habit, not a writing project — each week, pull the runs that escalated, the runs users retried, and a random sample of the quiet ones, and promote them into the suite with an expected-outcome label.
Ten cases catch embarrassments. A hundred catch regressions. A thousand, sampled well, catch drift — the slow degradation that never causes a single loud failure. The suite's size matters less than its provenance: cases that came from production defend production. Synthetic cases defend your imagination.
Your LLM judge is lying to you (until you calibrate it)
At agent scale you can't hand-grade every case, so you'll use a model to judge model outputs. Fine — everyone does. The mistake is trusting the judge before measuring it. Uncalibrated judges are systematically generous: they reward confident tone, they miss factual substitutions inside fluent prose, and they grade their own model family's style more kindly. We've watched a judge hand a 94% pass rate to a suite a human grader put at 71%.
Calibration is unglamorous and short: take 50–100 cases, grade them by hand, run the judge on the same set, and measure agreement. Below ~90% agreement, fix the rubric — make it binary per dimension ("did the agent cite the actual account balance: yes/no") instead of holistic ("was this helpful: 1–10"). Re-measure quarterly; judges drift when the underlying models update, and a judge that was honest in March can be flattering you by June.
Make the suite a gate, not a dashboard
A regression suite that publishes a score nobody blocks on is a decoration. The whole point is the gate: prompt changes, tool-schema changes, and model-version bumps run the suite in CI, and a drop past your threshold stops the deploy the same way a failing unit test would. This is also the only honest way to adopt new models — "the new model feels better" is vibes; "the new model passes 96% of the suite the old one passed 93% of, and here are the six cases it newly fails" is engineering.
Two budget notes from the trenches. First, evals cost tokens — a thousand-case suite with an LLM judge is real money per run, so tier it: a smoke slice on every commit, the full suite nightly and before releases. Second, keep failed cases forever. A case your agent once failed is the single most valuable kind: it has already proven it can catch something.
Takeaways
- — Promote production traces into your suite weekly; provenance beats volume.
- — Hand-grade 50–100 cases and measure judge agreement before trusting any score.
- — Binary per-dimension rubrics; holistic 1–10 scores hide substitution errors.
- — Wire the suite into CI as a blocking gate; tier smoke/full runs to control token spend.
- — Never delete a case the agent once failed.