product / 03 · evals
Know a change is better before your customers find out.
Prompts, models, and tools change every week. Without evals, every change is a bet placed on production. Trellis turns real traces into eval datasets, runs graders (exact-match, rubric, and LLM-as-judge with calibration), and gates your CI. A candidate that regresses on the golden set never ships.
reading the figure
What you are looking at
-
The candidate model is faster and 39% cheaper per run, and it wins on 11 of 13 suites.
-
It drops below the threshold on citation accuracy and dosage extraction, the two suites that matter most.
-
The gate exits with code 1, so CI blocks the deploy. Nobody has to remember to check a dashboard.
how it works
Three steps, one trace.
-
step 01
Collect
Flag runs from production traces into a dataset. Escalations and retries are the best place to start.
-
step 02
Grade
Score each case with deterministic checks, a rubric, or a calibrated LLM judge.
-
step 03
Gate
Run the suite in CI on every prompt, tool, or model change. A regression fails the build.
capabilities
What evals gives you
03.1
Traces become test cases
Flag any production run, good or bad, and it lands in a dataset with its inputs, tool results, and expected outcome. Your regression suite grows out of real traffic, not guesswork.
03.2
Three grader families
Deterministic checks (exact match, schema, latency), rubric scoring, and LLM-as-judge. Judges are calibrated against human labels, because an uncalibrated judge is just a second opinion.
03.3
CI gates
trellis eval --gate runs the suite against a candidate config and exits non-zero on a regression beyond your thresholds. Add it to any CI like any other test step.
03.4
Side-by-side diffs
Candidate versus baseline, per case: output diff, score change, token change, latency change. You see what a model upgrade actually changes before customers do.
when teams reach for it
Where it pays off first
- Teams about to swap models or upgrade a model version
- Agents whose prompts change every sprint
- Work where a made-up citation or number is a real liability
- Anyone tired of hearing "it seemed fine in testing"
What it costs
sample pricingNo separate meter for evals
Each eval case runs as a normal run and is priced like one. A 1,000-case suite costs what 1,000 runs cost, so most teams run a small smoke slice on every commit and the full suite nightly.
questions
Evals, answered plainly
How many eval cases do I need to start?
Ten real cases catch embarrassing failures. A hundred catch most regressions. Start with the runs that escalated last week and grow the set every Friday.
Can I trust an LLM judge?
Only after you measure it. Hand-grade 50 to 100 cases, run the judge on the same set, and check how often they agree. Trellis shows that agreement score next to every judged suite.
Does the gate work with my CI?
Yes. It is a normal command that exits with code 0 or 1, so it works anywhere a unit test does. The docs include examples for the common hosted CI services.
$ trellis estimate --honest
Try evals on a real workflow.
Price the stack in the configurator, then bring one of your own workflows to a demo. We'll trace it live and show you this module on your data.