Evaluation
Evaluation is how a run's deliverable is scored. Each agent declares its eval when it is built, and the eval runs against what the agent actually produced.
It changes behavior because it defines what "good" means for that agent, and anything you tune afterwards gets tuned toward it. An eval that only checks length rewards padding. An eval that requires real, resolvable URLs makes an agent that invents citations fail loudly instead of quietly. A judge that is strict about emptiness catches the most common agent failure of all — a confident report with nothing in it.
It is also the measurement instrument for every other axis on this page. Changing memory or planning and asking "did that help?" is only answerable if something scores the result the same way each time. Pick the eval first; it is the axis the others are read against.
Evals in this deployment
Read from the eval-plugin registry at page load.
| Goal judge (LLM) | An LLM judges the deliverable against the agent's goal — strict on counts, format, verifiability and emptiness. | available · default |
| Deterministic checks | Exact assertions on the deliverable — required phrases, minimum row counts, real URLs. No LLM, fully reproducible. | available |
| Rubric judge (LLM) | Scores each declared criterion separately, 0..1, and reports the weighted sum — a grading sheet instead of one holistic verdict. | available |
How to choose
The honest trade is reproducibility against coverage. Deterministic checks are exact and repeatable — the same deliverable always scores the same — but they only see what you thought to assert. An LLM judge covers the fuzzy half of "is this a useful answer" and will disagree with itself at the margins.
If your deliverable has a checkable shape — a minimum row count, a required phrase, real links — write the checks. They cost nothing to run and they never drift. Use the judge for goals whose quality is a judgment call, and treat a single judge score as a signal rather than a verdict.
Whichever you pick, the eval is only as honest as its threshold. A pass bar set where everything passes tells you nothing about the axes you are trying to compare.
There is an automated experiment runner: each agent's Tuning page (under /u/agents) can sweep the model, memory, planning, retrieval and state axes on one fixed objective — the agent's own declared eval, which is never swept, while guardrails only observes. Its limits are stated plainly: a naive full grid, several runs per cell, stopped cleanly at a hard budget wall with partial results kept. It compares compositions; it does not invent them.