Model Routing: How to Send Each Request to the Right Model

Key takeaways

• Model routing inspects each request and picks the model that fits it, instead of billing every task at frontier rates.
• Rule-based routing on task type captures most of the available savings. Most teams should not train a router.
• Measure cost per completed task. Cost per token can fall while cost per task rises.
• Routing can break prompt caching, and a cold cache on a long prompt can cost more than the cheaper model saved.
• The cheapest route is no model call, for steps that already run as deterministic code.

What model routing actually is

Model routing is the layer that inspects an incoming request and decides which model should handle it, rather than sending everything to one default. The decision uses signals carried by the request: task type, input length, expected output, caller identity, or an estimated difficulty score. The router forwards the request to the chosen model and returns that model's response to the caller.

This has nothing to do with network routing. Packets, BGP, and IP routing tables share the word and nothing else, which is worth stating plainly because search results for this phrase still mix the two senses.

Model routing also differs from Mixture-of-Experts. Routing selects between separately deployed models at the request level, and the choice is observable to you, since you can log which model answered. Mixture-of-Experts selects among expert subnetworks inside a single model during inference, which is an architectural property of that model rather than something your application configures.

Microsoft's shipped router is a useful reference for the mechanism, because the documentation describes it in three steps: analyze the full request including system message, tool definitions and conversation history, estimate which model in the pool handles it best, then forward it. Microsoft is explicit that the router itself is not an LLM. It is described as "a lightweight ML model designed to predict which model performs best for a given prompt at minimal latency" (Microsoft Learn).

Why routing showed up now

The spread inside a single vendor's catalog got wide enough to be worth arbitraging. Anthropic publishes Claude Opus 5 at $5 per million input tokens and $25 per million output, with Claude Haiku 4.5 at $1 and $5 (claude.com/pricing). OpenAI publishes gpt-6-astra at $10 input and $50 output per million, and gpt-5-nano at $0.05 and $0.40 (OpenAI pricing). Output prices across one provider's own lineup now span two orders of magnitude.

The second condition is that production traffic is not uniformly hard. Microsoft states the premise directly in its router documentation: "not all coding questions are equally hard." If a meaningful share of your requests are classification, extraction, or short reformatting, you are paying a reasoning-model rate for work a small model finishes correctly. Routing exists to stop that.

Both conditions are about price and difficulty distribution. Neither is about capability. Routing does not raise the quality ceiling of your system, and any argument that it does is a sales argument.

The routing strategies that exist

| Strategy | Signal it routes on | Best used for | What it costs you | | --- | --- | --- | --- | | Rule-based | Task type, caller, input length, explicit tags set by your own code | Pipelines where you already know the step boundaries | A rules table someone maintains every time the model lineup changes | | Classifier | A small model's difficulty score over the prompt | Open-ended traffic you cannot label in advance | A second model to evaluate, monitor, and retrain | | Cascading / escalation | The cheap model's own output, checked before it is returned | Quality-sensitive work where a wrong answer costs more than a second call | Added latency and duplicate spend on every request that escalates | | Learned / preference-based | A trained policy over model-quality features | Large or changing model pools, gateway and research settings | Training data, evaluation infrastructure, and drift when the pool changes | | No route at all | The step is known to produce the same shape of output on every run | Lookups, validation, formatting, writes to a system of record | Engineering time up front to build the app that runs the step |

Rule-based routing

Route on what your own code already knows. A support pipeline knows whether the current step is intent classification or a customer-facing reply. Send the first to a small model and the second to a large one. No inference is required to make the decision, so the routing hop adds no latency and no tokens.

The standard objection is that rules are brittle. They are, in the sense that someone has to revisit the table when a model is deprecated or a cheaper tier ships. That maintenance is a calendar reminder. Maintaining a trained classifier is a project.

Classifier routing

A small model scores the incoming prompt for difficulty and picks a destination. This is the shape of most managed routers, including Microsoft's, which is trained on hundreds of thousands of examples and exposes Balanced, Cost, and Quality modes that shift the same traffic across different distributions of the model pool.

Buying this is reasonable. Building it means you now operate two models, and the router is the one nobody has an evaluation set for. Microsoft is unusually candid about the follow-on work: "Managed routing doesn't remove the need for evaluation."

Cascading and escalation

Try the cheap model, check the result, escalate on failure. This is the strategy that preserves quality most reliably, because the expensive model still sees every request the cheap one got wrong. The cost is that escalated requests are paid for twice and answered late. Cascading works when your verification step is itself cheap and honest, and it degrades into pure overhead when the verifier is another frontier call.

The research inventory at Awesome-LLM-Routing catalogues both families, listing cascade systems such as FrugalGPT and AutoMix next to request-level routers such as RouteLLM, GraphRouter, and RouterDC. It is a good place to check whether a strategy you are about to invent already has a name and a published failure mode.

Learned and preference-based routing

The research frontier is routing across model pools that change. In Universal Model Routing for Efficient LLM Inference, Jitkrittum and colleagues propose UniRoute, which represents each LLM as a feature vector derived from its predictions on representative prompts, so the router can handle models it never saw during training. Their experiments route among more than 30 previously unseen LLMs. A later paper, Towards Generalized Routing by Guo and colleagues, extends the problem to selecting agents as well as models, proposing MoMA, which profiles model capabilities and selects agents through a context-aware state machine. Both abstracts report their methods as effective without publishing headline cost or accuracy figures, and I am citing what the abstracts state rather than reconstructing their benchmark tables here.

This is the least common strategy in production, and it should stay that way for most teams.

No route at all

Every row above answers the question "which model?" with a model. There is a fifth answer. If a step produces the same shape of output every time it runs, the correct destination is deterministic code, and the request never reaches the router.

That sounds like a dodge until you do the arithmetic. A request that never reaches the router costs nothing to route, nothing to classify, and nothing in tokens. Rule-based routing might move a step from a $25-per-million-output model to a $5 one. Moving the step into code removes it from the bill.

Here is what that looks like on a real workload. A support-ticket pipeline with four step types, routed by task rather than by difficulty score:

| Step | Typical difficulty | Route to | Why | | --- | --- | --- | --- | | Classify ticket intent | Low, bounded label set | Small model, or a trained classifier once you have labelled volume | The output space is a short enum. Frontier reasoning buys nothing. | | Extract order ID, account, and dates | Low, and the format is known | No model call. Regex, schema validation, and a lookup against the order table | The extraction is deterministic once the ticket format is known. This step should leave the routing table. | | Draft the customer reply | Medium to high, tone and judgment matter | Large model, cascading to a larger one only on verification failure | This is the judgment call the system exists to make. | | Escalate and write to the system of record | None. It is a state transition | No model call. An app with permissions and an audit log | Routing an irreversible write through a probabilistic system is a governance problem, not a cost problem. |

Two of four steps stop being routing decisions. The remaining two get the model attention they deserve.

How to tell whether your router is working

Instrument these five, in this order:

1. Cost per completed task. Not cost per token. A router that halves token price while doubling retries has made things worse and your token dashboard will show an improvement.
2. Escalation rate. The share of requests that started cheap and ended expensive. A rising escalation rate is the earliest signal that your routing thresholds no longer match your traffic.
3. Quality regression against a held-out set. A fixed set of requests with known-good outputs, replayed after every routing change. Without this you cannot distinguish savings from silent degradation.
4. Added latency at the routing hop. Rule-based routing adds roughly nothing. A classifier adds a full inference. Cascading adds a full inference on every escalation.
5. Cache hit rate before and after. This is the one teams skip, and it is the one most likely to invalidate the whole exercise.

Microsoft's guidance on evaluation is worth stealing whatever router you use: change one routing lever at a time, keep the workload and baseline fixed, and rerun the evaluation so the change in cost and quality is attributable.

Where routing goes wrong

The router becomes a failure mode. You have added a component that sits in front of every request and can be wrong. A classifier router can also be down, slow, or stale relative to a model lineup that shifted last week.

Routing breaks prompt caching, and the arithmetic can invert. Anthropic's documentation is explicit that "cache hits require 100% identical prompt segments," that the default cache lifetime is five minutes, and that cache read tokens are priced at 0.1 times the base input rate, while a five-minute cache write is 1.25 times (prompt caching docs). The docs do not promise cache sharing across models, so treat a model switch as a cold cache. Now run the numbers on published prices. A cached read against Opus 5 input is 0.1 x $5, or $0.50 per million tokens. A cold read against Haiku 4.5 is $1.00 per million. On a long, stable system prompt, routing the request to the cheaper model costs twice as much on input as leaving it where the cache already is. If your prompts carry large fixed prefixes, measure the cache before you route.

Quality drifts quietly. Routing failures do not throw exceptions. The cheap model returns a plausible answer, the pipeline continues, and you find out when a customer replies to a wrong response. This is the fear that keeps most teams from routing at all, and the held-out evaluation set is the only thing that answers it.

Most teams should not build a learned router. Rule-based routing on task type captures the majority of the savings at a fraction of the complexity, because your code usually already knows what kind of step it is executing. A trained router adds a second model to version, evaluate, monitor, and debug, to recover a margin over rules that most workloads never realize. If you have one dominant task type, low volume, or a caching strategy already doing real work, the honest answer is to skip the router. Buy a gateway if you want vendor failover, and spend the sprint elsewhere. This is a specific case of the wider set of cost levers, where routing is one option among several and rarely the largest.

What we're building at Major in response

Routing is real optimization and worth doing well. It also optimizes the wrong variable. A router reduces the unit price of re-reasoning. It does not reduce how often you re-reason. If a step produces the same answer shape every run, the right route is no model.

That is the design Major is built on. When an agent works out how to handle a repeatable part of a task, it builds an app for that part and then runs the app instead of reasoning through the step again. Two consequences follow, and both are the point. The app runs as deterministic code, so its output does not drift between runs the way a re-routed model call can. The app holds its own database, files, and logs, so context that used to be rebuilt in a prompt on every execution now persists, which removes tokens as well as variance.

The cost shape is front-loaded and then flat. There is genuine work in building the app the first time. After that, repeated execution stops scaling token spend with usage, because the model is no longer in the loop for that step. The model still routes and still reasons about the judgment calls. It simply handles fewer requests.

This argument does not cover agent evaluation, which is a harder and separate problem, and it does not apply to steps whose output genuinely varies with the input in ways code cannot express. Those steps should be routed, carefully, using the strategies above. For the rest, the smartest routing decision is the one you only have to make once. That is what coordinating models and the app layer is for, and it is where routing strategies and their limits stop being the interesting question.

If you are evaluating gateways that do this for you and want to see what shrinking the routing table looks like instead, see how Major's agents turn repeatable steps into apps they run.

Related articles

• Multi-Agent AI: What It Costs and When It's Worth It
• LLM Gateway: What It Governs, and What It Cannot
• AI Agent API: What Persists After the Request Ends