LLM Router: Routing Strategies and Their Real Limits

Key takeaways
• A router picks which model answers a request. A gateway is the shared access layer in front of your providers. Most products ship both under one name.
• Four strategies cover production use: rule and metadata, classifier, cascade, and semantic.
• Savings depend on your request mix. RouteLLM's "over 2 times" is a paper result, not a forecast for your traffic.
• Routing changes the price of a call. It does not change how many calls you make.

What are LLM routers?

An LLM router is the layer that sits in front of several language models and picks one of them for each incoming request. The routing happens over prompts and models at the application layer, and it has nothing to do with network hardware or a load balancer shuffling packets between hosts, which is a live source of confusion for anyone who searches this term cold. What gets routed is a request. What it gets routed to is a model.

Decision inputs vary by implementation. Most routers combine some subset of these: prompt content, task metadata the application already holds, a predicted difficulty score, price per token, a latency budget, and provider health.

Two distinctions matter before anything else. A router selects one model and sends the request there, while ensembling runs several models on the same request and combines their outputs, which can raise answer quality and always raises spend. Decagon's glossary entry keeps those apart, and so should your evaluation, because their cost arithmetic runs in opposite directions. A router is also not an orchestrator: routing picks a model for one request, while AI orchestration sequences steps, calls tools, and manages state between them.

Why teams reach for a router

Four motivations, and they pull against each other. Unit cost is the usual entry point, because a frontier model answering a request a small model would have handled correctly is money spent for no gain. Tail latency is second, since smaller models return faster and on interactive surfaces the p95 is what users feel. Provider failover is third, and it survives contact with production most reliably: when a provider degrades or rate-limits you, the router moves traffic and the product stays up. Per-task quality is fourth, because models differ by task in ways stable enough to exploit.

Those four do not stack. Tuning for cost pushes traffic toward weaker models, which costs quality on every request the router misjudges. Tuning for quality pushes traffic up, which costs money. The router is where that trade becomes explicit and reviewable, and that visibility is worth something even when the savings turn out modest.

The four routing strategies that actually get used

Published taxonomies run long. The LLMRouter project out of UIUC groups the research literature into seven router families, spanning global ranking, embedding-based supervised selection, preference learning, cost-aware gating, graph-based routing, LLM-driven multi-round routing, and personalized routing. Vendor explainers stack up to eleven overlapping labels. Four strategies cover what teams actually run, and the rest are variations on these or research directions that have not shipped broadly.

One housekeeping note if you are surveying open implementations. The NVIDIA AI Blueprint for LLM routing still ranks near the top of search results for this term, and its own repository now carries a deprecation notice pointing readers to NVIDIA NeMo Switchyard. None of the explainer pages currently ranking for this keyword mention that.

Rule and metadata routing

A static policy maps a known attribute to a model. Support tickets go to the small model, legal review goes to the frontier model, anything tagged with customer PII goes to the provider covered by your data agreement. The decision is made from data the application already has, before any inference runs.

This is the cheapest strategy to build, the easiest to audit, and the only one where a reviewer can read the routing logic and predict the outcome. Its weakness is that it never learns. A rule written against last quarter's traffic keeps firing against this quarter's, and nothing in the system notices.

Classifier routing

A trained scorer reads the request and predicts which model will answer it acceptably. RouteLLM, from Ong et al. at Berkeley (submitted June 2024, revised February 2025), trains routers on preference data to choose between a strong model and a weak one, and reports that the approach reduced costs "by over 2 times in certain cases" without compromising response quality on widely recognized benchmarks. The abstract does not name those benchmarks, and the figure is a result under the paper's conditions rather than a number you should put in a budget.

The cost here is structural. You now own a second model, with its own training data, its own drift, and its own evaluation problem. When the classifier degrades, the symptom is a slow decline in answer quality that looks like the base model getting worse.

Cascade routing

The cheap model answers first. If confidence falls below a threshold, the request escalates to a stronger model. Decagon describes this as complexity scoring plus confidence escalation, and the two-stage form is also called a model cascade.

Cascades are attractive because the escalation rate is directly observable. They fail on two things. Every escalated request pays twice, in tokens and in latency, so the economics invert once the escalation rate climbs past the point where the cheap first pass is mostly wasted work. And the confidence signal is often miscalibrated, because a small model's self-reported certainty is not a reliable estimate of whether it got the answer right.

Semantic routing

The request is embedded and matched against intent clusters or learned query-model affinities. The vLLM project's implementation describes itself as "a programmable Mixture-of-Models router for heterogeneous LLM inference", selecting model paths from request-level signals, user preferences, and application policy.

Semantic routing adapts to phrasing the rules never anticipated. Its failure mode is quiet: cluster boundaries are invisible until they break, and a genuinely new intent lands in an old cluster and gets routed confidently to the wrong place.

• Rule and metadata · Decision basis: Task type, tenant, sensitivity tag, known metadata · Setup cost: Low · Best fit: Stable traffic with clear policy boundaries · Main failure mode: No adaptivity, and rules go stale silently
• Classifier · Decision basis: Trained prediction of which model will succeed · Setup cost: High · Best fit: High volume with a wide capability spread between models · Main failure mode: You own a second model, and it drifts
• Cascade · Decision basis: Cheap model answers, low confidence escalates · Setup cost: Medium · Best fit: Traffic where most requests are genuinely easy · Main failure mode: Escalated requests pay twice, and confidence is often miscalibrated
• Semantic · Decision basis: Embedding match to intent clusters or query-model affinity · Setup cost: Medium to high · Best fit: Varied natural-language input with recurring intents · Main failure mode: New intents land in old clusters and misroute confidently

LLM router vs AI gateway

These two get used interchangeably and they are different components. The gateway is the shared entry point in front of your model providers: one API surface, key custody, rate limits, retries, request logging, spend controls. The router is the model-selection decision that happens behind that entry point.

Most commercial products ship both and market the pair under whichever word is trending. Inworld's comparison states that "Most 'LLM routers' are proxies" that abstract provider integrations rather than making a selection decision worth the name. That observation is useful, though the page then ranks its own product first in its own table.

• Model selection · Router: Its entire job · Gateway: Passes through, or delegates to a router
• API surface · Router: Consumes whatever the gateway exposes · Gateway: Presents one normalized API across providers
• Key and credential custody · Router: Not its concern · Gateway: Holds provider keys and scopes them per caller
• Rate limits and retries · Router: May prefer a healthy provider · Gateway: Enforces limits, handles retries and backoff
• Observability · Router: Logs the routing decision and its inputs · Gateway: Logs requests, tokens, latency, and spend

The practical question is whether the problem you have is access control or model choice. Those are separate layers, and choosing a layer in the agent stack before you buy saves rebuilding later. A gateway with no router is a normal and defensible architecture. A router with no gateway is usually an accident.

Where routing stops helping

Routing is a real optimization with a bounded ceiling. Four things it does not fix:

• A bad prompt. A misrouted request and a well-routed one with a weak prompt produce the same disappointing output, and the router cannot tell which is which.
• The quality ceiling of your best model. Routing moves work between models you already have. It cannot make the top of that range better.
• Latency, at the margin. The routing decision is itself a hop, and a classifier-based decision may involve its own inference call before the real one starts.
• Evaluation. Router accuracy needs labeled outcomes, and most teams have spend dashboards long before they have a labeled set. Savings are easy to see. Misroutes are not.

One more limit sits underneath all four. Routing changes the price of a call. It leaves the number of calls exactly where it found it.

That matters because a large share of production traffic is the same operation running against new inputs. A router will find the cheapest capable model for the ten thousandth identical entitlement check. No routing strategy will ask why the entitlement check is being performed by a language model at all.

What this article does not cover: routing across self-hosted GPU fleets, where the constraints are capacity and scheduling rather than price per token, and router evaluation methodology, which needs labeled outcome data and deserves its own treatment.

The half a router cannot touch

Take a support-triage pipeline. A ticket arrives. The system classifies it, extracts the order ID and issue type, checks entitlement against the orders database, and drafts a reply. Send all of it through a model and you have four inference calls per ticket, at which point a router looks like the obvious lever.

Split the same pipeline by what needs judgment:

• Steps that need a model: Reading an ambiguous complaint and deciding what the customer is asking for · Steps that do not: Extracting an order ID into a fixed schema and validating its format
• Steps that need a model: Judging whether a case should escalate to a human · Steps that do not: Looking up entitlement and warranty status in the orders database
• Steps that need a model: Drafting a reply in the right tone for the situation · Steps that do not: Writing the outcome back to the ticket record and logging the action

The right column is deterministic work being billed as inference. A lookup, a fixed-schema extraction, a validation, a write. Each has a correct answer that does not vary with sampling temperature, and each is performed by the least predictable and most expensive component in the stack. Routing that column to a cheaper model reduces the bill. It does not change the fact that a probabilistic system is doing arithmetic.

Move those steps into application code and the economics change shape. The extraction becomes a schema and a parser. The entitlement check becomes a query against a table the app owns. The write becomes a write, with its own audit record. The model keeps the left column, where its judgment was worth paying for. This is the decomposition behind most durable agentic workflow patterns: separate the steps that need reasoning from the steps that need to be right every time.

The cost curve that produces is front-loaded and then flat. You pay reasoning cost once to work out the operation, and the app runs afterward at the cost of running code. Ticket volume doubles and the deterministic half does not get more expensive. Under a router, that doubling doubles the token spend on those steps.

Running both layers together

These layers are complements. Route the judgment work, because models genuinely differ on it. Then take the repeatable work out of inference entirely, because the cheapest model call is the one that never happens.

That second move is what Major is built for. An agent works out how to handle a task, and when part of that task is repeatable, it builds an app for that part and runs the app from then on instead of reasoning through it again. The app holds its own data in a managed database, keeps its files, writes its logs, and runs under the same permissions and audit as anything else on the platform. Reason once, run forever. For the triage pipeline above, that is an app the agent builds and then calls: entitlement lookups, schema-validated extraction, and ticket writes running as code, with the model reserved for the ambiguous complaint and the escalation call. We have written separately about how to build an agent in production.

You build on Major across Claude, Kimi, Gemini, Muse Spark, ChatGPT, and Grok, and model choice stays reversible because the deterministic layer does not move when the model underneath it does. Context and memory live in the app, in a database you can query and permission, rather than in a context window that has to be refilled every run and disappears when the conversation ends.

The honest scope: the app layer reduces how often the model works, and it does not remove the model. Judgment still costs tokens, and the agent still reasons when a genuinely new situation arrives. The reasoning-to-code step also has to be governed like any other software change. What it buys is that the repeatable half stops being a variable cost and becomes a fixed asset the whole organization can run.

If you came here to cut model spend, the sequence is straightforward. Adopt routing where the work needs judgment and the models genuinely differ, starting with rule-based routing, because it is cheapest to own and the only strategy a reviewer can read and predict. Then find the steps with one correct answer, and give them an app that holds its own data.

Major gives you $100 in free credits to build your first AI agent or app. If the split above matches something you are running, see how Major turns the repeatable half into an app with its own state.

Related articles

• Grok vs ChatGPT: Key Differences and a Reversible Choice
• What Is AI Orchestration? A Guide for Enterprise Teams
• Agentic Workflows: Definition, Patterns, and When to Use Them