Model routing
Last verified: June 2026· definition
Model routing classifies each request and dispatches it to a model that fits: a cheap, fast model for easy tasks like classification and a frontier model for work where quality is load-bearing.
Done with quality guards (and validated by evals), routing is the single biggest lever in most cost-optimization efforts — a large share of call volume is low-stakes and can move to a cheaper tier with no measurable quality loss.
Routing is usually implemented in a gateway or router layer that also handles caching, batching, and fallback chains.
Most requests do not need your best model
In a typical production system a substantial share of calls are classification, extraction, routing, formatting, or short summarisation — tasks where a much smaller model performs equivalently. Sending all of them to a frontier model is the most common single source of avoidable spend. Routing is the discipline of matching capability to task, and it usually returns more than caching, batching, or prompt trimming, precisely because the mismatch is so widespread.
Routing needs evals to be safe
Every routing rule is a bet that a cheaper model handles a class of request at acceptable quality. Without evals that bet is unmeasured, which is why teams either avoid routing entirely or adopt it and discover the damage through complaints. With a small eval suite per route the question becomes empirical: send a sample to the cheaper model, compare against the current one, and keep the change only where quality holds. That turns a gamble into a measurement.
Fallback is part of routing
A router is also the natural place to handle provider failures, rate limits, and overload, by falling back to an alternative model rather than surfacing an error. This is often the feature that justifies building the routing layer, independent of cost, because it converts provider incidents from outages into degraded performance. It also makes migration between providers a configuration change rather than a code change.
Common misconceptions
MythRouting means accepting worse quality to save money.
RealityIt means not paying frontier prices for tasks a smaller model completes equally well. Where quality would drop, the route stays on the stronger model — that is what the evals decide.
MythA router should pick the model automatically per request.
RealityDynamic selection is possible and adds unpredictability. Most production value comes from static per-route rules, which are simpler to reason about, cheaper, and far easier to evaluate.
MythRouting is only worth it at large scale.
RealityThe saving is proportional, so it applies at any volume. What scale changes is whether the engineering effort is justified — and a gateway you already run makes that effort small.
Frequently asked questions
How do I decide which model to route a request to?
Start from the task rather than the request. Classification, extraction, routing, and fixed-format generation are usually safe on smaller models; open-ended reasoning, long-horizon agentic work, and anything high-stakes generally are not. Then verify empirically: run a sample of real traffic for that route through the cheaper model and compare against your current one with an eval set. Static per-route rules validated this way are more predictable than dynamic selection.
Should routing be dynamic or static?
Static per-route rules cover most of the value and are far easier to reason about, test, and debug — you know exactly which model handles which work. Dynamic routing, where a classifier chooses per request, adds a call, adds latency, and introduces a new failure mode. It is worth considering when a single route genuinely contains a wide mix of difficulty, but it is rarely the right first step.
Do I need a gateway to do model routing?
Not strictly — routing logic can live in application code. A gateway becomes worthwhile once you want routing, fallbacks, caching, and per-route cost observability in one place rather than scattered across services, and once more than one application needs the same behaviour. Many teams start with routing in code and adopt a gateway when the second consumer appears.
Go deeper
How to reduce LLM latency
LLM latency is two separate problems — the wait for the first token and the time to stream the rest — and each has its own set of fixes.
Hiring & partnersHow to hire an LLM cost optimization consultant
Hire an LLM cost consultant when your bill is large enough to matter but your team is too busy to instrument, route, and eval their way out of it themselves — and only hire one who leaves you owning the result.