📊 Leaderboards test the wrong thing: isolated skill ≠ agent skill (we measured 14 models both ways)

Model leaderboards score one call at a time: given a question and some tools, did the model pick right? Answer well? On that kind of test, nearly every current model is fine — and nearly indistinguishable. But an agent isn't one call. It's a loop: search, read what came back, decide what's next, hold the plan, finish. So we measured 14 models both ways — the same models, as isolated tool calls and as end-to-end agents — and the two scores barely correlate. That gap is the finding, and no single-call leaderboard can show it to you, because seeing it requires running the second test.

The numbers below were measured June 20, 2026 — the then-current flagships from Anthropic, OpenAI, and Google plus the strongest open-weights models (Llama, DeepSeek, Qwen, Mistral, GLM). We publish them as measured, dates and all, because the finding is the durable part: rosters churn monthly, but "isolated skill doesn't predict agent skill" has held through every model we've added since. (A fresh sweep on the current catalog will land at this same URL — see the end of the post.)

The finding: flat on single calls, a cliff on the loop

The microtests measure one decision in isolation: given a weather question and six tools, pick the right one; given a schema, write correct SQL; given messy HTML, extract clean JSON. We grade each 0–1 — deterministic checks where we can (right tool? right fields?), a panel of three judge models from three different providers for the prose ones, so no single judge's family preference can swing a score.

On those, almost everyone is fine. Across all seven categories, every model landed between 0.82 and 0.95. Picking a tool, building a payload, summarizing — capable models tie, and where they tie, the cheapest competent one wins. (The harder variants we added — a branching plan, a transformed payload, an extraction with a decoy — separated the field a little, but the headline held.)

The agent eval measures whether a model can actually run an agent: a real multi-turn loop where it must search, read what came back, decide what's next, and finish the job — a research brief, or a "find the top news and draft me an email" digest. Same model, but now every step depends on the last. We ran each scenario three times and report the spread, because a single lucky trajectory is a data point, not a ranking. Cost is realized dollars from live OpenRouter pricing × actual tokens, not a sticker price.

Watch what happens to the exact same models:

The microtest column is nearly flat. The agent column ranges from 0.77 to 0.06. DeepSeek scores 0.84 on isolated calls and 0.06 running an agent — it can write a perfect query in a vacuum and then never drive a loop to a finish. Qwen and GLM ace the microtests (0.93, 0.95) and complete fewer than half their agent runs. How a model does on a single tool call barely predicts whether it can run an agent. If you only ever tested isolated calls — which is what most leaderboards do — you'd pick exactly the wrong models.

This is why "which model tops the leaderboard this month" is the wrong question for agent builders. The ranking churns; the gap between the two columns is the structural fact, and it's the one your architecture has to answer for.

Second finding: cheap models aren't worse on the loop, they're erratic

Look at the ± column. Opus completes at 0.77 ±0.08 — steady. The cheap models swing violently: GLM ±0.43, Qwen ±0.42, Llama ±0.37. A model that completes the same task 0% of the time on one run and 90% on the next isn't a cheaper option — it's a coin flip, and a coin-flip orchestrator is worse than a reliably-mediocre one, because you can't build on it. This is the finding the first (single-run) pass hid: GLM looked like a perfect 1.00 until we ran it three times and saw 0.43 ±0.43. Repetition turned a fluke into the real story.

The flip side is just as useful: repeatability itself is a per-model property. On the same harness, same inputs, Claude Haiku returned the identical completion score three times in a row on both scenarios (spread ±0.000), while GPT-5.2 swung ±0.41 on one of them. There is no universal "agents are ±X" number — the spread belongs to the model.

So the rule the architecture bets on, now with evidence: keep a coherent, consistent model on the loop, and route the cheap, leaf work — extraction, summarization, a single call — to whatever's cheapest-competent. Best value driver in this table: Claude Haiku (0.65 at $4.91 per 1k runs). Most reliable: Opus, for the genuinely hard, long-horizon agents.

This isn't us being contrarian — it's where the field is going. Nous Research's Hermes agent ships exactly this in production (one loop model + cheaper swappable slots), 2026 research is converging on per-subtask routing, and the Berkeley Function Calling Leaderboard documents the same multi-turn cliff. Our contribution is measuring it on real agent runs, with metered cost.

What production adds — at its actual scale

A controlled benchmark is one thing, so we also looked at the real thing: 210,840 tool calls metered through the gateway every agent's tool calls flow through, across 94 workspaces (corpus snapshot June 26, 2026). Honest framing up front: the tool-call count is large, but the LLM spend in this corpus is small — the money table below totals about $35 — so read this section as a direction check on the benchmark, not independent proof.

Two numbers from that corpus. First, 85% of calls are reads, 15% writes, at a 97% success rate — most of what agents do is look things up, which is why the leaf categories tie and why a gateway can be permissive on reads and careful on the few writes. Second: of the roughly $35 of agent-LLM spend in the corpus, 89% goes to one surface — running the agent loop:

Read the last two rows. Gemini Flash did 8× the call volume of Claude Sonnet for about 1/13th of the cost. The handful of expensive calls on the loop dominate the bill; the high-volume cheap calls on the leaf surfaces are a rounding error. At $35 of spend that's a small-corpus observation, but it is consistent with the model-by-action thesis the benchmark points to: the loop is where the money concentrates, so the loop model is the one decision worth getting right — and the benchmark says the priciest model is rarely the best buy for it.

We didn't just write it down — we wired it in

The results now drive how the builder picks models:

• Leaf tool-internal calls (web extraction, summarization) stay on the cheap tier — the benchmark and the production spend pattern both point the same way: that's safe.
• The agent loop has an orchestration floor: when an agent needs to run a real multi-step loop, the builder won't hand the wheel to a model the benchmark showed is weak or erratic there, and prefers the proven value drivers — Haiku first, Opus for the hard cases. Every pick carries a benchmark-cited reason in the build reply, so the choice is legible.

In plain terms: the model your agent gets is the one that measured best for the kind of work it'll actually do — cheap where cheap is safe, coherent where the loop demands it.

This table refreshes here, not at a new URL

We ran this on June 20, 2026, and we're publishing the table as measured rather than silently re-dating it. Since then: newer models have shipped that this table doesn't include, and the product has grown its own test bench — a sweep runner that runs your agent, on your eval, across models and every other axis, which this June harness predates. Honest provenance: the June numbers come from a standalone OpenRouter single-key harness with simulated tool backends, not the product runner.

When the model catalog refreshes (#858), we'll re-run the sweep on the current models from the product bench, on real tools, and update this post in place — same URL, both-ways method intact, old table kept for the record. If you're reading a leaderboard dated this month, the rankings here may be stale; the two-column gap is what we'd bet survives the refresh.

If what you actually need is a decision procedure — which model should my agent use — that's a different question with a better answer than any ranking: see Which LLM for AI agents? The cheapest one that passes the eval.

What we still don't trust

• Only two agent scenarios. "Run a real agent" should mean a dozen archetypes — a scout, a coder, a multi-agent crew — not two. Everything above is "on these tasks," not a law of nature.
• Simulated tools. The agent eval used mocked tool backends for reproducibility. The real prize is scoring models on actual outcomes — and because the gateway records the cost and result of every real tool call, that's the next version: an outcome-grounded benchmark on the 210k-and-growing corpus.
• The production corpus is small money. 210k calls, ~$35 of LLM spend. It can agree with the benchmark in direction — and does — but it can't independently confirm it. We say "consistent with," and mean exactly that.
• Judges, even three of them, aren't humans. A cross-provider panel beats one grader, but the prose and completion scores still want a human spot-check before they go on a billboard.

So treat the numbers as a strong v0: de-noised with repeats and a judge panel, already changing how we route, directionally matched by what little real spend we can meter, and honest about its edges. The headline is the part we'd bet on: isolated skill ≠ agent skill, cheap models are erratic on the loop, and the expensive model is rarely the best buy — for the loop, pick coherent; for the leaves, pick cheap.

---

See the models this deployment serves at /plugins/models, build an agent that picks per action at /u/build, or sweep the model axis against your own agent's eval at its Tuning page.