An agent harness where every run is an eval

The demo, in one screen

Here's an agent answering questions over a knowledge vault, with its whole trajectory judged on the way out:

$ grounded-harness demo
  gadget-colours   tool_correctness=ok  loop_efficiency=ok  groundedness=ok  budget_adherence=ok
  rollback-time    tool_correctness=ok  loop_efficiency=ok  groundedness=ok  budget_adherence=ok
  widget-price     tool_correctness=ok  loop_efficiency=ok  groundedness=ok  budget_adherence=ok

  GATE: PASS

Now I break the agent's reading tool, so every note it fetches comes back empty:

$ grounded-harness demo --sabotage broken-tool
  ! [How long does a rollback take?] claim not grounded: '90 seconds' missing from any tool result
  ! groundedness regressed: 1.0 -> 0.3333
  GATE: FAIL

Exit code 1. And a second sabotage, thinning the sample to a single golden run, gets a different and better answer than a pass or a fail:

$ grounded-harness demo --sabotage thin-evidence
  GATE: FAIL — n=1 goldens, need >=3: refusing to PASS on insufficient evidence

Both failure modes run in CI. A sabotaged demo that passes fails the build, because a green gate only means something if you have watched it go red for the right reasons.

The bet

There are plenty of agent harnesses. There are plenty of eval frameworks. What I could not find, as of November 2026, was a harness where the two share a spine: where a run cannot complete without knowing how well it went, cannot lose money to a crash, and cannot silently drift from its recorded baseline. So grounded-harness is that bet, made small: a Python tool-calling loop with three rules enforced in code rather than in documentation.

Unknown is not zero. Evidence has floors. And every gate ships with the sabotage test that proves it can fail.

If you have read the earlier pieces in this series, you will recognise all three. They came out of building an MCP server that abstains, a judge panel whose gate once passed on zero judgments, and two measurement tools that learned to say "insufficient". The harness is where those lessons stop being anecdotes and become the runtime.

What the loop refuses to do

The run discipline is mostly a list of refusals, and each one was earned.

The budget system stops a run gracefully rather than crashing it: a budget-exhausted run is a valid outcome carrying every step taken and an honest partial summary. But the interesting part is the ordering, because my first version had it wrong and the tests caught it before anyone else could. With a tight step ceiling, the budget check fired after paying for a model turn but before recording it. A paid turn with no record, in a module whose docstring promised no such path existed. The fix is a principle, not a patch: step ceilings check before spend happens, cost ceilings settle after the paid step is recorded. No paid turn goes unrecorded, ever. Anthropic's Claude Agent SDK settles its spend cap the same way: it compares spend with the cap "after model responses arrive", and the response that reaches the cap still completes and counts.

A cousin of that rule: if you set a cost ceiling and the provider stops reporting token counts, the harness refuses to continue on an "unverifiable budget". It will not treat unknown spend as free, because a cost ceiling that cannot fire is decoration.

And resume never re-bills. Completed steps replay from an atomic checkpoint, and resuming against changed inputs is refused outright, because a refused resume costs you a re-run while a wrong resume costs you correctness. That trade was measured the hard way in a production pipeline migration; the harness just inherits the scar tissue.

Trajectory evals, and what they honestly test

A golden is a recorded run plus expectations. Replaying it reconstructs the model's turns from the record and re-executes everything else for real: the loop, the tools, the data behind the tools. Four families judge the result: did the tool-call sequence match and every call succeed, did the run stay within its step allowance, is every claimed fact present in both the final answer and some tool result, did it end with the expected status.

Notice what that does and does not cover. Because the model side is deterministic on replay, golden replay is a regression harness for the harness, the tools and the tool data, not for the model. Model quality needs live runs and judges, and that pattern lives in the judge-panel piece, not here. Writing that boundary down matters more than the feature list, because an eval that quietly claims more coverage than it has is worse than no eval.

The gate on top aggregates across goldens against a committed baseline, and it fails closed below three goldens. One good run is an anecdote. The gate says so instead of passing.

The best test result was in someone else's repo

The demo agent's tools are not stubs. They proxy to a real grounded-mcp server over stdio, answering from a vendored copy of its demo vault. And the first time the bridge's integration test called a real tool over real transport, it failed, with a bug that was not in the harness at all.

grounded-mcp's eval suite is thorough and entirely in-process: single thread, no transport. Version 2 of the MCP Python SDK runs synchronous tool functions on worker threads, a change its migration guide does flag, and the server's SQLite index, created on the main thread, refused the crossing. Every database-touching tool call failed over real MCP transport, on a server whose own gates were green. The fix shipped upstream the same day, with a cross-thread regression test.

One sentence of moral: an eval suite's coverage is bounded by the path it exercises, not by the assertions it makes. The harness's integration test exercised a path its sibling's evals never touched, and that is the whole argument for testing compositions, not just components.

Try it

The repo is github.com/rich-atkins/grounded-harness, MIT licensed, and the demo runs offline with no keys: scripted model turns, real MCP tools, three goldens, two sabotages. Point the same loop at AnthropicProvider() and nothing else changes.

Then do the thing this series keeps arriving at: record one golden run of your own agent, replay it in CI, and sabotage it once to watch the gate refuse. If your harness cannot fail a run, it is not evaluating anything. It is narrating.