💸 Which LLM for AI agents? A procedure: the cheapest model that passes your eval

You searched for a model recommendation, so here is the honest version: any post that names one model for "agents" in general is answering a question that has no general answer. What exists instead is a procedure, and the procedure is short: fix the eval first, then pick the cheapest model that passes it, then stop.

Notice the ordering does all the work. Without an eval, "which model is better" has no answer, and in the absence of an answer the honest default is the expensive one — which is exactly how bills get built. With an eval, the question becomes empirical and cheap models get a fair trial they otherwise never receive.

(If what you want is numbers — how 14 specific models scored, two ways, with cost per run — that's a different post: Leaderboards test the wrong thing: isolated skill ≠ agent skill. This one is the decision procedure those numbers argue for.)

Why this is a per-agent decision

This deployment's catalog currently carries six models across three providers — Gemini 2.5 Flash and Pro, GPT-4o Mini and GPT-4o, Claude Haiku 4.5 and Sonnet 4.5 — rendered live on the models axis page, and the model is chosen per agent, priced per run. Per-agent matters because the jobs differ more than the models do. A well-scoped job — extract these fields, summarize this page, decide which of three branches applies — is often indistinguishable across tiers. A job that requires noticing a tool result contradicts an assumption, or holding a long chain without dropping a clause, is where the tiers separate. (Measured, that separation is dramatic: in our benchmark, models nearly tied on isolated calls and fanned out wide on full agent loops.)

Best for the small tier: frequent, well-scoped, verifiable steps — which, counted per call, is most of what a scheduled agent does. Best for the large tier: the steps where the specific big-model virtues show up — multi-clause instructions followed completely, plans that don't lose the thread, tool results actually read rather than skimmed.

Where the money actually goes

Two cost mechanics from this codebase's own accounting are worth more than any leaderboard.

First: the expensive step is usually not the frequent step. One hard step can dominate an agent's bill while dozens of routine calls round to nothing. This is why an account-wide model setting is the wrong shape — you would be paying the hard step's rate on every routine call, or starving the hard step to discount the routine ones.

Second, caching is not a discount so much as a different pricing scheme. Our metering splits cached-read tokens (billed at roughly a tenth of the input rate) from cache-creation tokens (billed at roughly 1.25x) precisely because agentic loops re-send their context every round — on a loop, cache economics can matter more than list price. Honest limits: those multipliers are the providers' current terms, cited from our cost accounting as of August 2026; providers reprice, and this deployment's price table is synced from a maintained snapshot rather than hand-edited, because hand-edited price tables rot. Treat every number in this section as dated.

Moving up a tier for a vaguely disappointing answer usually just costs more. Move up when you can name the failure — instructions half-followed, a thread lost, a tool result misread. A vague disappointment is at least as often a memory, retrieval or eval problem wearing a model costume, and no upgrade fixes those.

One more reason "which LLM" resists a general answer: someone else's benchmark victory is weak evidence about your agent, and your own eval's verdict is strong evidence that transfers to nothing else. Both halves of that are worth keeping — even about our own numbers. It's also why the catalog here lists six models across three providers rather than crowning one: the catalog's job is to give the procedure a search space, not to have taste.

The one axis you can actually sweep today

Most of this series ends with "this is what the test bench exists to measure." The model axis is the strongest version of that sentence: it is one of the five the experiment runner sweeps, and cost per run is recorded alongside the eval verdict — so "cheapest model that passes" is not a slogan here, it is a button. An agent's Tuning page prices the same task across every model this workspace can serve, judged by the agent's own eval, as a naive full grid under a hard budget cap. Honest limits: a grid with several runs per cell tells you which model passes this task's eval — it is not a general model ranking, and we have no cross-task numbers to offer until real sweeps accumulate. When a real sweep produces catalog-wide numbers, they publish as an update to the benchmark post — that URL is where our measured model data lives.

Where this leaves you

Situational advice: give every new agent the cheap tier and a real eval on day one. If it passes, you're done, and you found out for pennies. If it fails, read how it failed before upgrading — a failed check names its gap, and the gap tells you whether it's a model gap. Re-run the comparison when the catalog shifts; a model choice is a claim with a shelf life measured in months.

What this doesn't tell you: which model passes your eval — by design. That answer costs one sweep and is worth more than every general-purpose leaderboard row ever published, because it's about your task, scored by your definition of good.

The reframe: model selection for agents isn't taste, and it isn't loyalty — it's a unit-economics question your own eval can answer. Set the eval, build the agent, and let the cheapest model apply for the job.