product
One runtime, four modules, one trace model.
Everything below operates on the same run: orchestration executes it, guardrails constrain it, evals score it, observability prices it. No integrations between our own features — it's one system.
01 / 04
Orchestration
Durable runs that survive restarts, retries, and people
An agent run is a long-lived, failure-prone conversation between a model and your systems. Trellis makes each run durable: every step is checkpointed, retries are policy-driven with exponential backoff, and a run can pause for hours waiting on a human approval without holding a connection open — then resume exactly where it stopped.
Checkpointed steps
Every model call and tool call is persisted before it executes. A worker crash mid-run replays from the last checkpoint — no duplicate side effects, no lost context.
Retry policies, not retry loops
Declare max attempts, backoff, and which error classes are retryable per tool. Rate-limit errors back off; validation errors fail fast to the fallback branch.
Human-approval steps
Any step can require sign-off. The run parks (costing nothing), a reviewer approves in the console or via API, and execution resumes with the approval recorded in the trace.
Fan-out and joins
Spawn parallel sub-agents with a token budget each, then join on all/any/quorum. Budgets are enforced by the runtime, not by hoping the prompt behaves.
02 / 04
Guardrails
Policy checks at run and call boundaries, enforced by the runtime
Guardrails are deterministic policy checks that sit outside the model: schema validation on tool arguments, allowlists on which tools an agent may touch, PII screens on inputs and outputs, and hard budget caps on tokens and spend. They run in observe, dry-run, or enforce mode — so you can watch what a policy would block before you let it block.
Tool allowlists & argument schemas
An agent can only call tools on its manifest, and every argument payload is validated against a JSON Schema before the call executes. Out-of-schema writes never reach your systems.
Input/output screens
Fast classifier checks (~40 ms p95, run in parallel with the step) for PII, prompt injection, and off-policy content — priced per check, at cost.
Budget caps that actually stop runs
Per-run and per-workspace ceilings on tokens, spend, and wall-clock. Enforcement is in the runtime: the step that would exceed the cap never executes.
Observe → dry-run → enforce
Ship a policy in observe mode, read the would-have-blocked report for a week, then flip to enforce with a one-line change — and a full audit trail of who flipped it.
03 / 04
Evals
Regression gates for behavior, wired into CI
Prompts, models, and tools change weekly; without evals every change is a bet placed on production. Trellis turns real traces into eval datasets, runs graders (exact-match, rubric, LLM-as-judge with calibration), and gates your CI: a candidate that regresses on the golden set never ships.
Traces become test cases
Flag any production run — good or bad — and it lands in a dataset with inputs, tool results, and expected outcomes. Your regression suite grows out of reality, not imagination.
Three grader families
Deterministic checks (exact match, schema, latency), rubric scoring, and LLM-as-judge — with judge calibration against human labels, because an uncalibrated judge is just a second opinion.
CI gates
`trellis eval --gate` runs the suite against a candidate config and exits non-zero on regression beyond your thresholds. Wire it into any CI like any other test step.
Side-by-side diffs
Candidate vs. baseline, per case: output diff, score delta, token delta, latency delta. You see what a model upgrade actually changes before customers do.
04 / 04
Observability
Every run a trace; every span priced
Each run produces a full trace: a tree of spans for model calls, tool calls, guardrail checks, and approvals — with token counts, cost, and latency on every node. Filter fleets by outcome, cost, or policy hits; replay any run step-by-step; export everything over OpenTelemetry to the stack you already have.
Span-level cost accounting
Input tokens, output tokens, cache reads, and dollars on every model span — rolled up per run, per agent, per tenant, per feature, using your metadata.
Step-through replay
Reconstruct any run: what the model saw, what it said, what the tool returned, what the guardrail decided. The 3 a.m. incident review takes minutes, not a log dig.
Fleet views & alerting
Outcome rates, p95 latency, cost per resolved run — sliced by agent version. Alert when error rates spike or cost per run drifts past a threshold you set.
OpenTelemetry out
Traces export as standard OTel spans to Datadog, Grafana, Honeycomb, or your collector. Trellis is a source of truth, not a data prison.
plays well with your stack
Bring your models, keep your tools.
Any model provider
Bring your own API keys or use our metered pass-through — same trace, same math either way. Swap providers per agent, per step, or per eval candidate.
OpenTelemetry native
Spans export with GenAI semantic conventions to Datadog, Grafana, Honeycomb, or your own collector. Trellis is a source of truth, not a silo.
CI you already run
trellis eval --gate is a normal exit-code step — GitHub Actions, Buildkite, Jenkins, whatever. No new deploy system to adopt.
$ trellis estimate --honest
See the four modules on your own workload.
Price the stack first in the configurator, then bring a real workflow to a 30-minute demo — we'll trace it live.