⚖️ How to evaluate an AI agent
The short answer: declare, per agent, one thing that scores every deliverable the same way, and pick it before you tune anything else. The rest of this post is which scorer to pick and what each one will miss.
First, the failure that makes evaluation worth a post at all. The most common way an agent goes wrong in production is not a crash. It is a confident report with nothing in it — well-formatted, plausible, empty. A crash pages you. An empty deliverable arrives on schedule, reads fine at a glance, and quietly teaches you to stop reading. Our deterministic eval's very first assertion, before any configured check runs, exists because of exactly this: an empty deliverable fails outright, with the recorded reason "the run took no tangible action." That line is in the code because the failure kept happening.
An agent without an eval is not underperforming. It is unmeasured. "Did that change help?" has no answer unless something scores the result the same way each time — which also makes the eval the instrument every other axis is read against. Change memory, planning, or the model, and the eval is how you find out whether it mattered.
The rubric before the verdicts
Three things worth grading an eval on, in order:
- Reproducibility — does the same deliverable always get the same score?
- Coverage — does it see the fuzzy half of "is this actually useful"?
- Cost — what does scoring itself cost, per run?
No option wins all three. That is the honest shape of this axis, and it is why there are three evals in the registry instead of one.
The options, as they exist here
These are the eval plugins this deployment carries — the same list the evaluation axis page renders live from the registry.
Deterministic checks. Exact assertions on the deliverable: required phrases, a minimum row count, URLs that are real. No LLM anywhere, so scoring is free and byte-reproducible — the same deliverable scores the same way forever. Honest limits: the checks only see what you thought to assert, and the row counter is a stated heuristic (it counts markdown table rows or bullet lines — counts, not meaning). A padded table of junk rows passes a row-count check. Best for: deliverables with a checkable shape — digests, tables, reports with required fields.
Goal judge. The default. An LLM reads the deliverable against the agent's goal and is deliberately strict on counts, format, verifiability and emptiness. It covers what checks cannot: whether the thing is actually an answer. Honest limits: it costs a model call per score, and a single holistic score from a judge will disagree with itself at the margins — treat one score as a signal, not a verdict. Best for: goals whose quality is a judgment call, which is most first agents; it is the default for a reason.
Rubric judge. The same judging machinery, but you declare weighted criteria and each one is scored 0-to-1 separately, then combined as a weighted sum. A grading sheet instead of one gut verdict — and noticeably more stable across runs, because the judge answers five narrow questions instead of one broad one. Honest limits: still an LLM, still not byte-reproducible, and if you declare no rubric it deliberately behaves exactly like the goal judge rather than inventing criteria you never wrote. Best for: agents you are actively tuning, where you need scores you can compare across weeks.
The verdict, such as it is: write deterministic checks for everything checkable, and add a judge for the part that isn't. The trade underneath is reproducibility against coverage, and you don't have to pick a side — the checkable floor and the fuzzy ceiling are different jobs.
Two mistakes that make any eval worthless
A pass bar where everything passes. Whichever eval you pick, the threshold is where the honesty lives. An eval that always says yes tells you nothing about the axes you are trying to compare — it is a rubber stamp with extra steps.
Tuning toward the eval and forgetting you wrote it. An eval that only checks length rewards padding. An eval that requires real, resolvable URLs makes an agent that invents citations fail loudly instead of quietly — which is the behavior you want, but only because you thought to assert it. The uncomfortable compression: your agent is only as good as the thing that scores it, and the thing that scores it is only as good as your last honest look at it.
What a judge's self-disagreement actually costs
How much does an LLM judge disagree with itself on identical input? That is a real number with real consequences for how much weight a single score can bear — and it is a number this post does not have, because we have not run that sweep yet. It is precisely what the test bench exists to measure: same deliverable, same judge, k runs, variance reported. When we run it, the result publishes as a data post, whatever it shows. Until then, "the judge is noisy at the margins" is an observation from use, not a measurement.
Where this leaves you
Situational advice: if your deliverable has a shape, start with checks — they cost nothing and never drift. If your goal is fuzzy, take the default goal judge and set a threshold you would defend. Graduate to the rubric judge when you find yourself re-reading old runs to decide whether a change helped.
What this post doesn't tell you: which eval is right for your agent's actual task, whether your threshold is honest, or how noisy the judge is on your deliverables — those are empirical questions, and each agent's Tuning page exists to answer them against your own eval, since the eval is the one axis a sweep holds fixed. Claims in this post are dated August 2026 and describe the registry as it stands; registries grow, and claims like these have a shelf life.
The reframe: evaluation isn't the report card at the end of the pipeline. It is the first decision, because it defines what every later decision means. Pick it first — then go build the agent it will keep honest, or see what the evaluation axis carries today.