Evaluation
Evaluation is how a run's deliverable is scored. Each agent declares its eval when it is built, and the eval runs against what the agent actually produced.
It changes behavior because it defines what "good" means for that agent, and anything you tune afterwards gets tuned toward it. An eval that only checks length rewards padding. An eval that requires real, resolvable URLs makes an agent that invents citations fail loudly instead of quietly. A judge that is strict about emptiness catches the most common agent failure of all — a confident report with nothing in it.
It is also the measurement instrument for every other axis on this page. Changing memory or planning and asking "did that help?" is only answerable if something scores the result the same way each time. Pick the eval first; it is the axis the others are read against.
Evals in this deployment
Read from the eval-plugin registry at page load.
| Goal judge (LLM) | An LLM judges the deliverable against the agent's goal — strict on counts, format, verifiability and emptiness. | available · default |
| Deterministic checks | Exact assertions on the deliverable — required phrases, minimum row counts, real URLs. No LLM, fully reproducible. | available |
How to choose
The honest trade is reproducibility against coverage. Deterministic checks are exact and repeatable — the same deliverable always scores the same — but they only see what you thought to assert. An LLM judge covers the fuzzy half of "is this a useful answer" and will disagree with itself at the margins.
If your deliverable has a checkable shape — a minimum row count, a required phrase, real links — write the checks. They cost nothing to run and they never drift. Use the judge for goals whose quality is a judgment call, and treat a single judge score as a signal rather than a verdict.
Whichever you pick, the eval is only as honest as its threshold. A pass bar set where everything passes tells you nothing about the axes you are trying to compare.
There is no automated experiment runner yet. Nothing here sweeps this axis for you — changing it means editing the agent and reading the next run. A compare mode that runs one task across several compositions is named on the index as not built, and that is still true.