What is an LLM router?

An LLM router is a layer in front of several language models that selects one model per request. It decides using the prompt content, task metadata the application already holds, a predicted difficulty score, price per token, latency budget, or provider health. It selects one model, unlike ensembling, which runs several models on the same request and combines their outputs. Routing assumes every request needs a model and only asks which one. Major changes the input to that question, because an agent that has written the repeatable work into a deployed app sends fewer requests to route in the first place.

What is the difference between an LLM router and an LLM gateway?

A gateway is the shared access and control layer in front of your model providers: one normalized API, provider key custody, rate limits, retries, request logging, and spend controls. The router is the model-selection decision behind it. Most commercial products ship both under one name. A gateway without a router is a common architecture. A router without a gateway usually happens by accident. Major sits at a different point entirely and offers neither, governing what the agent can reach through a credential proxy rather than which model answers.

Does an LLM router actually reduce costs?

Conditionally. Savings scale with the share of requests a cheaper model can handle correctly, so a workload that is mostly hard questions saves little. RouteLLM (Ong et al., 2024) reports cost reductions of over 2 times in certain cases without compromising response quality, which is a paper result under the paper's conditions rather than a guarantee for your request mix. Routing lowers the slope of a bill that still rises with volume. Major targets the other term, front-loading cost into the reasoning that builds the app so the settled steps stay flat as usage grows.

Is an LLM router the same as an AI orchestrator?

No. Routing answers one question for a single request: which model should handle this. Orchestration sequences multiple steps, calls tools, manages state between them, and decides what happens next based on results. A pipeline can use both, with an orchestrator running the steps and a router choosing the model for each step that needs one. Major is neither, and it does not route models. It is where an agent turns the stable part of that pipeline into an app that runs without asking a model anything.

How does an LLM router decide which model to use?

Most routers score the request first, then apply a policy. Classifier-based routing predicts difficulty from the prompt and sends easy requests to a smaller model. Rule-based routing keys off metadata the application already knows, such as customer tier or task type. Cascade routing tries the cheap model first and escalates when a confidence check fails. Cost, latency ceilings, and provider health then filter the shortlist. Rules like those are deterministic, and on Major an agent writes that kind of logic into the app once rather than re-deriving it per run.

Do I need an LLM router?

Not until you are running more than one model in production with a meaningful spread of request difficulty. Below that, a router adds a failure point, a latency hop, and a scoring model of its own to maintain, for savings you can approximate by picking a better default model. Add one when a measurable share of traffic is provably easy. Major is a separate decision from that one, and adding it makes sense when work repeats under stable rules rather than when your model bill needs splitting.

LLM Router: Routing Strategies and Their Real Limits

An LLM router picks which model answers each request. The four routing strategies that work, how routers differ from gateways, and what routing cannot fix.

Rahul RamakrishnanAugust 19, 2026Updated August 28, 2026

Painted lines on a wall splitting into separate paths, illustrating how a router sends each request down one of several model paths

Key takeaways
• A router picks which model answers a request. A gateway is the shared access layer in front of your providers. Most products ship both under one name.
• Four strategies cover production use: rule and metadata, classifier, cascade, and semantic.
• Savings depend on your request mix. RouteLLM's "over 2 times" is a paper result, not a forecast for your traffic.
• Routing changes the price of a call. It does not change how many calls you make.

What are LLM routers?

An LLM router is the layer that sits in front of several language models and picks one of them for each incoming request. The routing happens over prompts and models at the application layer, and it has nothing to do with network hardware or a load balancer shuffling packets between hosts, which is a live source of confusion for anyone who searches this term cold. What gets routed is a request. What it gets routed to is a model.

Decision inputs vary by implementation. Most routers combine some subset of these: prompt content, task metadata the application already holds, a predicted difficulty score, price per token, a latency budget, and provider health.

Two distinctions matter before anything else. A router selects one model and sends the request there, while ensembling runs several models on the same request and combines their outputs, which can raise answer quality and always raises spend. Decagon's glossary entry keeps those apart, and so should your evaluation, because their cost arithmetic runs in opposite directions. A router is also not an orchestrator: routing picks a model for one request, while AI orchestration sequences steps, calls tools, and manages state between them.

Why teams reach for a router

Four motivations, and they pull against each other. Unit cost is the usual entry point, because a frontier model answering a request a small model would have handled correctly is money spent for no gain. Tail latency is second, since smaller models return faster and on interactive surfaces the p95 is what users feel. Provider failover is third, and it survives contact with production most reliably: when a provider degrades or rate-limits you, the router moves traffic and the product stays up. Per-task quality is fourth, because models differ by task in ways stable enough to exploit.

Those four do not stack. Tuning for cost pushes traffic toward weaker models, which costs quality on every request the router misjudges. Tuning for quality pushes traffic up, which costs money. The router is where that trade becomes explicit and reviewable, and that visibility is worth something even when the savings turn out modest.

The four routing strategies that actually get used

Published taxonomies run long. The LLMRouter project out of UIUC groups the research literature into seven router families, spanning global ranking, embedding-based supervised selection, preference learning, cost-aware gating, graph-based routing, LLM-driven multi-round routing, and personalized routing. Vendor explainers stack up to eleven overlapping labels. Four strategies cover what teams actually run, and the rest are variations on these or research directions that have not shipped broadly.

One housekeeping note if you are surveying open implementations. The NVIDIA AI Blueprint for LLM routing still ranks near the top of search results for this term, and its own repository now carries a deprecation notice pointing readers to NVIDIA NeMo Switchyard. None of the explainer pages currently ranking for this keyword mention that.

Rule and metadata routing

A static policy maps a known attribute to a model. Support tickets go to the small model, legal review goes to the frontier model, anything tagged with customer PII goes to the provider covered by your data agreement. The decision is made from data the application already has, before any inference runs.

This is the cheapest strategy to build, the easiest to audit, and the only one where a reviewer can read the routing logic and predict the outcome. Its weakness is that it never learns. A rule written against last quarter's traffic keeps firing against this quarter's, and nothing in the system notices.

Classifier routing

A trained scorer reads the request and predicts which model will answer it acceptably. RouteLLM, from Ong et al. at Berkeley (submitted June 2024, revised February 2025), trains routers on preference data to choose between a strong model and a weak one, and reports that the approach reduced costs "by over 2 times in certain cases" without compromising response quality on widely recognized benchmarks. The abstract does not name those benchmarks, and the figure is a result under the paper's conditions rather than a number you should put in a budget.

The cost here is structural. You now own a second model, with its own training data, its own drift, and its own evaluation problem. When the classifier degrades, the symptom is a slow decline in answer quality that looks like the base model getting worse.

Cascade routing

The cheap model answers first. If confidence falls below a threshold, the request escalates to a stronger model. Decagon describes this as complexity scoring plus confidence escalation, and the two-stage form is also called a model cascade.

Cascades are attractive because the escalation rate is directly observable. They fail on two things. Every escalated request pays twice, in tokens and in latency, so the economics invert once the escalation rate climbs past the point where the cheap first pass is mostly wasted work. And the confidence signal is often miscalibrated, because a small model's self-reported certainty is not a reliable estimate of whether it got the answer right.

Semantic routing

The request is embedded and matched against intent clusters or learned query-model affinities. The vLLM project's implementation describes itself as "a programmable Mixture-of-Models router for heterogeneous LLM inference", selecting model paths from request-level signals, user preferences, and application policy.

Semantic routing adapts to phrasing the rules never anticipated. Its failure mode is quiet: cluster boundaries are invisible until they break, and a genuinely new intent lands in an old cluster and gets routed confidently to the wrong place.

Rule and metadata · Decision basis: Task type, tenant, sensitivity tag, known metadata · Setup cost: Low · Best fit: Stable traffic with clear policy boundaries · Main failure mode: No adaptivity, and rules go stale silently
Classifier · Decision basis: Trained prediction of which model will succeed · Setup cost: High · Best fit: High volume with a wide capability spread between models · Main failure mode: You own a second model, and it drifts
Cascade · Decision basis: Cheap model answers, low confidence escalates · Setup cost: Medium · Best fit: Traffic where most requests are genuinely easy · Main failure mode: Escalated requests pay twice, and confidence is often miscalibrated
Semantic · Decision basis: Embedding match to intent clusters or query-model affinity · Setup cost: Medium to high · Best fit: Varied natural-language input with recurring intents · Main failure mode: New intents land in old clusters and misroute confidently

LLM router vs AI gateway

These two get used interchangeably and they are different components. The gateway is the shared entry point in front of your model providers: one API surface, key custody, rate limits, retries, request logging, spend controls. The router is the model-selection decision that happens behind that entry point.

Most commercial products ship both and market the pair under whichever word is trending. Inworld's comparison states that "Most 'LLM routers' are proxies" that abstract provider integrations rather than making a selection decision worth the name. That observation is useful, though the page then ranks its own product first in its own table.

Model selection · Router: Its entire job · Gateway: Passes through, or delegates to a router
API surface · Router: Consumes whatever the gateway exposes · Gateway: Presents one normalized API across providers
Key and credential custody · Router: Not its concern · Gateway: Holds provider keys and scopes them per caller
Rate limits and retries · Router: May prefer a healthy provider · Gateway: Enforces limits, handles retries and backoff
Observability · Router: Logs the routing decision and its inputs · Gateway: Logs requests, tokens, latency, and spend

The practical question is whether the problem you have is access control or model choice. Those are separate layers, and choosing a layer in the agent stack before you buy saves rebuilding later. A gateway with no router is a normal and defensible architecture. A router with no gateway is usually an accident.

Where routing stops helping

Routing is a real optimization with a bounded ceiling. Four things it does not fix:

A bad prompt. A misrouted request and a well-routed one with a weak prompt produce the same disappointing output, and the router cannot tell which is which.
The quality ceiling of your best model. Routing moves work between models you already have. It cannot make the top of that range better.
Latency, at the margin. The routing decision is itself a hop, and a classifier-based decision may involve its own inference call before the real one starts.
Evaluation. Router accuracy needs labeled outcomes, and most teams have spend dashboards long before they have a labeled set. Savings are easy to see. Misroutes are not.

One more limit sits underneath all four. Routing changes the price of a call. It leaves the number of calls exactly where it found it.

That matters because a large share of production traffic is the same operation running against new inputs. A router will find the cheapest capable model for the ten thousandth identical entitlement check. No routing strategy will ask why the entitlement check is being performed by a language model at all.

What this article does not cover: routing across self-hosted GPU fleets, where the constraints are capacity and scheduling rather than price per token, and router evaluation methodology, which needs labeled outcome data and deserves its own treatment.

The half a router cannot touch

Take a support-triage pipeline. A ticket arrives. The system classifies it, extracts the order ID and issue type, checks entitlement against the orders database, and drafts a reply. Send all of it through a model and you have four inference calls per ticket, at which point a router looks like the obvious lever.

Split the same pipeline by what needs judgment:

Steps that need a model: Reading an ambiguous complaint and deciding what the customer is asking for · Steps that do not: Extracting an order ID into a fixed schema and validating its format
Steps that need a model: Judging whether a case should escalate to a human · Steps that do not: Looking up entitlement and warranty status in the orders database
Steps that need a model: Drafting a reply in the right tone for the situation · Steps that do not: Writing the outcome back to the ticket record and logging the action

The right column is deterministic work being billed as inference. A lookup, a fixed-schema extraction, a validation, a write. Each has a correct answer that does not vary with sampling temperature, and each is performed by the least predictable and most expensive component in the stack. Routing that column to a cheaper model reduces the bill. It does not change the fact that a probabilistic system is doing arithmetic.

Move those steps into application code and the economics change shape. The extraction becomes a schema and a parser. The entitlement check becomes a query against a table the app owns. The write becomes a write, with its own audit record. The model keeps the left column, where its judgment was worth paying for. This is the decomposition behind most durable agentic workflow patterns: separate the steps that need reasoning from the steps that need to be right every time.

The cost curve that produces is front-loaded and then flat. You pay reasoning cost once to work out the operation, and the app runs afterward at the cost of running code. Ticket volume doubles and the deterministic half does not get more expensive. Under a router, that doubling doubles the token spend on those steps.

Running both layers together

These layers are complements. Route the judgment work, because models genuinely differ on it. Then take the repeatable work out of inference entirely, because the cheapest model call is the one that never happens.

That second move is what Major is built for. An agent works out how to handle a task, and when part of that task is repeatable, it builds an app for that part and runs the app from then on instead of reasoning through it again. The app holds its own data in a managed database, keeps its files, writes its logs, and runs under the same permissions and audit as anything else on the platform. Reason once, run forever. For the triage pipeline above, that is an app the agent builds and then calls: entitlement lookups, schema-validated extraction, and ticket writes running as code, with the model reserved for the ambiguous complaint and the escalation call. We have written separately about how to build an agent in production.

You build on Major across Claude, Kimi, Gemini, Muse Spark, ChatGPT, and Grok, and model choice stays reversible because the deterministic layer does not move when the model underneath it does. Context and memory live in the app, in a database you can query and permission, rather than in a context window that has to be refilled every run and disappears when the conversation ends.

The honest scope: the app layer reduces how often the model works, and it does not remove the model. Judgment still costs tokens, and the agent still reasons when a genuinely new situation arrives. The reasoning-to-code step also has to be governed like any other software change. What it buys is that the repeatable half stops being a variable cost and becomes a fixed asset the whole organization can run.

If you came here to cut model spend, the sequence is straightforward. Adopt routing where the work needs judgment and the models genuinely differ, starting with rule-based routing, because it is cheapest to own and the only strategy a reviewer can read and predict. Then find the steps with one correct answer, and give them an app that holds its own data.

Major gives you $100 in free credits to build your first AI agent or app. If the split above matches something you are running, see how Major turns the repeatable half into an app with its own state.

Frequently asked questions

What is an LLM router?: An LLM router is a layer in front of several language models that selects one model per request. It decides using the prompt content, task metadata the application already holds, a predicted difficulty score, price per token, latency budget, or provider health. It selects one model, unlike ensembling, which runs several models on the same request and combines their outputs. Routing assumes every request needs a model and only asks which one. Major changes the input to that question, because an agent that has written the repeatable work into a deployed app sends fewer requests to route in the first place.
What is the difference between an LLM router and an LLM gateway?: A gateway is the shared access and control layer in front of your model providers: one normalized API, provider key custody, rate limits, retries, request logging, and spend controls. The router is the model-selection decision behind it. Most commercial products ship both under one name. A gateway without a router is a common architecture. A router without a gateway usually happens by accident. Major sits at a different point entirely and offers neither, governing what the agent can reach through a credential proxy rather than which model answers.
Does an LLM router actually reduce costs?: Conditionally. Savings scale with the share of requests a cheaper model can handle correctly, so a workload that is mostly hard questions saves little. RouteLLM (Ong et al., 2024) reports cost reductions of over 2 times in certain cases without compromising response quality, which is a paper result under the paper's conditions rather than a guarantee for your request mix. Routing lowers the slope of a bill that still rises with volume. Major targets the other term, front-loading cost into the reasoning that builds the app so the settled steps stay flat as usage grows.
Is an LLM router the same as an AI orchestrator?: No. Routing answers one question for a single request: which model should handle this. Orchestration sequences multiple steps, calls tools, manages state between them, and decides what happens next based on results. A pipeline can use both, with an orchestrator running the steps and a router choosing the model for each step that needs one. Major is neither, and it does not route models. It is where an agent turns the stable part of that pipeline into an app that runs without asking a model anything.
How does an LLM router decide which model to use?: Most routers score the request first, then apply a policy. Classifier-based routing predicts difficulty from the prompt and sends easy requests to a smaller model. Rule-based routing keys off metadata the application already knows, such as customer tier or task type. Cascade routing tries the cheap model first and escalates when a confidence check fails. Cost, latency ceilings, and provider health then filter the shortlist. Rules like those are deterministic, and on Major an agent writes that kind of logic into the app once rather than re-deriving it per run.
Do I need an LLM router?: Not until you are running more than one model in production with a meaningful spread of request difficulty. Below that, a router adds a failure point, a latency hop, and a scoring model of its own to maintain, for savings you can approximate by picking a better default model. Add one when a measurable share of traffic is provably easy. Major is a separate decision from that one, and adding it makes sense when work repeats under stable rules rather than when your model bill needs splitting.