Major

LLM Router: Routing Strategies and Their Real Limits

An LLM router picks which model answers each request. The four routing strategies that work, how routers differ from gateways, and what routing cannot fix.

Rahul RamakrishnanUpdated
Railroad tracks crossing and diverging, a routing junction for LLM request routing

An LLM router is a layer in front of several language models that picks one model for each incoming request, based on task type, predicted difficulty, cost, latency, or provider health. The four strategies teams actually run are rule-based routing, trained classifiers, cascades, and semantic matching. A router changes the price of each model call. It does not change how many calls you make, and that second number is where most of the bill lives. Major attacks it directly: an agent builds the repeatable steps into a deterministic app, so those steps stop being model calls at all.

This is routing of prompts to models at the application layer. It has nothing to do with network routers or load balancers.

What is an LLM router?

A router selects one model and sends the request there. Ensembling is different: it runs several models on the same request and combines the outputs, which can raise quality and always raises spend. A router is also different from an orchestrator. Routing picks a model for one request. Orchestration sequences steps, calls tools, and manages state between them.

Decision inputs vary, but most routers combine some of these: prompt content, task metadata the application already holds, a predicted difficulty score, price per token, a latency budget, and provider health.

Teams adopt routers for four reasons that pull against each other. Unit cost, because a frontier model answering a question a small model would get right is wasted money. Tail latency, because smaller models return faster. Provider failover, because traffic can move when a provider degrades. Per-task quality, because models differ by task in stable ways. Tuning for cost costs quality on every misrouted request, and tuning for quality costs money. The router is where that trade becomes explicit and reviewable.

The four routing strategies that get used

Research taxonomies run long. The UIUC LLMRouter project groups the literature into seven router families, and vendor explainers stack up to eleven labels. Four cover what ships in production.

Rule and metadata routing. A static policy maps a known attribute to a model. Support tickets go to the small model, legal review goes to the frontier model, anything tagged with PII goes to the provider covered by your data agreement. It is the cheapest to build and the only strategy where a reviewer can read the logic and predict the outcome. It never learns, so rules written against last quarter's traffic keep firing against this quarter's.

Classifier routing. A trained scorer predicts which model will answer acceptably. RouteLLM (Ong et al., Berkeley) trains routers on preference data to choose between a strong and a weak model, and reports cost reductions of over 2 times in certain cases without compromising response quality on the benchmarks it tested. That is a result under the paper's conditions, not a budget line. You now own a second model with its own drift and evaluation problem, and when it degrades the symptom looks like the base model getting worse.

Cascade routing. The cheap model answers first. If confidence falls below a threshold, the request escalates to a stronger model. The escalation rate is directly observable, which makes cascades easy to tune. Two things break them. Every escalated request pays twice in tokens and latency, so the economics invert once escalation climbs. And thresholds are fragile because a small model's self-reported confidence is a poor estimate of whether it was right. Set the threshold from labeled outcomes, never from a guess.

Semantic routing. The request is embedded and matched against intent clusters or learned query-model affinities. The vLLM project's implementation describes itself as a programmable Mixture-of-Models router. It adapts to phrasing rules never anticipated, and fails quietly: a genuinely new intent lands in an old cluster and gets routed confidently to the wrong place.

Compare them by failure mode:

  • Rule and metadata: setup cost low, best for stable traffic with clear policy boundaries, fails by going stale silently.
  • Classifier: setup cost high, best for high volume with a wide capability gap between models, fails when the scorer drifts.
  • Cascade: setup cost medium, best when most requests are easy, fails when escalations pay twice or confidence is miscalibrated.
  • Semantic: setup cost medium to high, best for varied language with recurring intents, fails when new intents misroute confidently.

LLM router vs AI gateway

The gateway is the shared entry point in front of your providers: one normalized API, key custody, rate limits, retries, request logging, spend controls. The router is the model-selection decision behind it. Most commercial products ship both under whichever word is trending. A gateway without a router is a defensible architecture. A router without a gateway is usually an accident.

Where routing stops helping

Routing does not fix a weak prompt, because a misrouted request and a badly prompted one produce the same disappointing output. It does not raise the ceiling of your best model. It adds a hop, and a classifier may need its own inference call before the real one. And it needs labeled outcomes to evaluate, which most teams lack when they start. Savings are easy to see. Misroutes are not.

One limit sits underneath all four. Routing changes the price of a call and leaves the number of calls exactly where it found it. A router will find the cheapest capable model for the ten thousandth identical entitlement check. No routing strategy asks why a language model is performing the entitlement check at all.

This piece does not cover routing across self-hosted GPU fleets, where capacity and scheduling dominate, or router evaluation methodology, which deserves its own treatment.

The half a router cannot touch

Take a support-triage pipeline. A ticket arrives. The system classifies it, extracts the order ID, checks entitlement in the orders database, drafts a reply, and writes the outcome back. Run every step through a model and a router looks like the obvious lever.

Split it by what needs judgment. Reading an ambiguous complaint, deciding whether to escalate, and drafting a reply in the right tone need a model. Extracting an order ID into a fixed schema, looking up warranty status, and writing the result back to the ticket have one correct answer each. That second group is deterministic work billed as inference. Routing it to a cheaper model lowers the bill but leaves a probabilistic system doing arithmetic.

Move those steps into code and the cost curve changes shape. You pay reasoning cost once to work out the operation, and the app runs afterward at the cost of running code. Ticket volume doubles and the deterministic half does not get more expensive. Under a router, that doubling doubles the token spend on those steps.

Build the routed-plus-deterministic pipeline on Major

Route the judgment work, then take the repeatable work out of inference. Major is the enterprise platform where agents build the software they run on, so the second step is native. Here is the triage pipeline built there.

  • Connectors: Zendesk for tickets, Postgres for the orders database, Slack for escalations. Credentials are scoped per connector and enforced at the point where the agent acts.
  • The app: the agent builds a triage app with a managed database holding ticket state and decision history, plus deployed endpoints for schema-validated extraction, entitlement lookup, and the ticket write-back. These run as code, the same way every time.
  • The agent: it keeps the judgment calls. It reads the ambiguous complaint, decides escalate or resolve, and drafts the reply. Model choice stays reversible because the app layer does not move when the model underneath it does, and you can still apply a rule-based router to this narrower set of calls.
  • The trigger: a workflow fires on each new Zendesk ticket, runs the app, calls the agent only where judgment is needed, and posts high-risk cases to Slack for a person to approve.
  • Governance: role-based access decides who can change the app or the agent, and every action lands in the audit log with the credentials it used, so you can reconstruct what happened in March.

Start building free and the first version of this app is a prompt away.

The honest scope: the app layer reduces how often the model works, and it does not remove the model. Judgment still costs tokens, and the agent still reasons when a genuinely new situation arrives. A one-step workflow that lives in a single prompt does not need any of this. Major earns its place when the pipeline is long, branches across several systems of record, and someone will eventually have to explain what it did.

Start building on Major

Build the triage pipeline above, or any workflow where a model is doing lookups, extraction, and writes it should not be doing. Major turns the repeatable steps into governed apps with their own state, so the model is left with the judgment.

Related articles

Frequently asked questions

What is an LLM router?
An LLM router is a layer in front of several language models that selects one model per request. It decides using the prompt content, task metadata the application already holds, a predicted difficulty score, price per token, a latency budget, or provider health. Routing assumes every request needs a model and only asks which one. Major changes the input to that question, because an agent that has written the repeatable work into a deployed app sends fewer requests to route.
What is the difference between an LLM router and an LLM gateway?
A gateway is the shared access and control layer in front of your model providers: one normalized API, provider key custody, rate limits, retries, request logging, and spend controls. The router is the model-selection decision behind it. Most commercial products ship both under one name. Major governs a different layer, controlling what the agent can reach through scoped credentials and audit logs rather than which model answers.
Does an LLM router actually reduce costs?
Conditionally. Savings scale with the share of requests a cheaper model can handle correctly, so a workload that is mostly hard questions saves little. RouteLLM (Ong et al., 2024) reports cost reductions of over 2 times in certain cases without compromising response quality, a paper result under the paper's conditions. Routing lowers the slope of a bill that still rises with volume. Major front-loads cost into building the app, so settled steps stay flat as usage grows.
Is an LLM router the same as an AI orchestrator?
No. Routing answers one question for a single request: which model should handle this. Orchestration sequences multiple steps, calls tools, manages state between them, and decides what happens next based on results. A pipeline can use both, with an orchestrator running the steps and a router choosing the model for each step that needs one. On Major, an agent turns the stable part of that pipeline into an app that runs without asking a model anything.
How does an LLM router decide which model to use?
Most routers score the request first, then apply a policy. Classifier-based routing predicts difficulty from the prompt and sends easy requests to a smaller model. Rule-based routing keys off metadata the application already knows, such as customer tier or task type. Cascade routing tries the cheap model first and escalates when a confidence check fails. Cost, latency ceilings, and provider health then filter the shortlist.
Do I need an LLM router?
Not until you run more than one model in production with a meaningful spread of request difficulty. Below that, a router adds a failure point, a latency hop, and a scoring model to maintain, for savings you can approximate by picking a better default model. Add one when a measurable share of traffic is provably easy. Moving repeatable steps into an app on Major is a separate lever, worth pulling when work repeats under stable rules.