Major

LLM Routing: How It Works and Where It Stops (2026)

LLM routing sends each request to the model best suited to it. Here are the four strategies, what the benchmarks show, and why Major cuts the call volume routing cannot.

Rahul RamakrishnanUpdated
Airport departure board listing flights, an image for routing each request to the right destination

LLM routing is a decision made before inference. Given a request and a pool of models at different price and capability tiers, a routing policy predicts which model can answer acceptably at the lowest cost, then sends the request only there. The savings are real in measured cases and bounded everywhere, because routing changes which model answers a request. It has no say in how many requests your system makes.

Major goes after that second number. Major is the enterprise platform where agents build the software they run on, so repeatable steps move into deterministic apps and leave the model entirely. A step that no longer calls a model has nothing left to route. Routing and Major stack: Major shrinks the volume, routing prices the inference that remains.

What is LLM routing?

Three properties of the definition do the work. The decision is per request, so two users in the same session can land on different models. The decision is a prediction, so it can be wrong, and the cost of being wrong is lopsided: a hard query sent to a cheap model produces a bad answer, while an easy query sent to an expensive model only produces a bad invoice. And the decision is made from a pool, so routing exists only after you have accepted the operational cost of depending on more than one model.

Routers split into non-predictive and predictive. Non-predictive routers run inference on several models and pick the best result, which costs more than it saves. Predictive routers decide from information available before any inference happens. Nearly everything sold as a router today is predictive. For a tighter treatment of vendor options, see our guide to the LLM router and the companion piece on model routing.

The four LLM routing strategies

  • Rules on metadata. Hand-written conditions on endpoint, customer tier, prompt length or output format. Cheap to run and the hardest to attack, since no learned parameters exist to manipulate.
  • Classifier. A small trained model predicts whether the cheap model will succeed, then thresholds the score. It needs labelled history and retraining when models change.
  • Cascade. Try the cheap model, check the answer, escalate on failure. Economics depend on escalation rate and on whether your check beats a coin flip. A weak verifier pays twice for the same wrong answer.
  • Semantic similarity. Embed the query and route by nearest neighbour in a curated library. Fast and interpretable when traffic clusters, quietly wrong when a new intent resembles nothing in the library.

Start with rules. If your traffic already splits along known lines such as endpoint or customer tier, a rules router is cheaper, faster and harder to manipulate than a learned one.

Does LLM routing work? What the benchmarks show

RouteLLM, from Ong and colleagues at Berkeley, trained routers on Chatbot Arena preference data. The arXiv abstract claims cost reduction of "over 2 times in certain cases" without compromising quality (arXiv:2406.18665). The widely quoted 85 percent figure comes from the project's GitHub README, not the paper, and vendor pages routinely cite it as though it were the paper's result. Know which artifact your business case rests on.

LLMRouterBench, a Findings of ACL 2026 paper by Li and colleagues, is the largest public evaluation: more than 400,000 instances, 21 datasets, 33 models and 10 routing baselines under one unified evaluation. Its findings should shape any routing project:

  • Different models really do win on different queries, so headroom exists.
  • Under unified evaluation, many routing methods perform similarly.
  • Several recent approaches, commercial routers included, fail to reliably outperform a simple baseline.
  • A large gap to the Oracle router remains, driven mainly by model-recall failures.

Routing earns money in measured cases, and the router you pick is weakly determined by the research. Treat it as an experiment on your own traffic with a control arm. Classification with a fixed label set is the friendliest case, because correctness can be checked against ground truth in your ticket history. Most routing decisions are not that clean.

LLM routing vs an LLM gateway

A gateway gives you one API surface across providers, with key management, logging, retries and fallbacks. It decides nothing about model choice by itself. A gateway is where a router gets deployed, but adopting a gateway does not mean you are routing. If every request still reaches the same model, you gained observability and key hygiene and your bill did not move. The LLM gateway guide covers what a gateway governs and where it stops. If you are weighing hosted aggregators, our OpenRouter alternatives ranking puts Major first and explains why.

Where LLM routing stops helping

A predictive router adds a hop to every request, and a cascade makes escalated queries pay full latency twice. The router can also fail quietly. Quality drift shows up as a rising share of hard queries answered by the cheap model, which looks like nothing on a cost dashboard and like everything to the affected users. Watch a quality metric as closely as spend.

Learned routers add an attack surface. Researchers showed that query-independent token sequences appended to any input can force a router to select the expensive model, against open-source and commercial routers (arXiv:2501.01818). A systematization of threats found real routing systems most vulnerable to cost escalation (arXiv:2601.21380).

The ceiling matters most. A workflow that makes forty model calls per run makes forty routed calls per run. Route perfectly and you have bought a discount on a volume you never questioned. That is why LLM cost optimization starts with eliminating repeated calls and treats routing as the second lever.

How to choose a routing approach

  1. Measure first. Log model, tokens, latency and a quality signal per request for two weeks.
  2. Separate the calls that repeat from the calls that need judgment. Repeats belong in code, not in a router.
  3. Try rules on metadata before a learned router.
  4. Run any learned router against a control arm on your own traffic, and compare quality as well as spend.
  5. Put the router behind a gateway so you can roll back in one change.

Build the cost lever on Major instead of only routing

If your routing project is really a cost project, the bill is usually growing because the same reasoning is repeated across thousands of runs. Here is a ticket-triage workflow on Major:

  • Connectors. Zendesk for tickets, Slack for escalations, Postgres or Snowflake for history, each with scoped credentials.
  • The app. The agent works out the triage rules once, then builds an app that applies them in deterministic code, with a managed database holding labels, outcomes and overrides.
  • The agent's judgment. The model handles only the tickets the app cannot classify with confidence, plus drafting replies that need context.
  • The trigger. A workflow runs on new-ticket events or a schedule. The app does the repeatable step with no model call.
  • Governance. RBAC on who can change rules, audit logs on every classification, and credentials scoped to the systems the app touches.
  • Routing still applies. Where a model call remains, pick the tier with the same control-arm discipline described above.

Start building on Major and see which of your model calls are decisions and which are repetitions. The related guide on AI model orchestration shows how this differs from model-level routing.

The Major take on LLM routing

Routing is a real optimization and a bounded one. It cannot stop a decision made last week from being re-derived from scratch today. It prices repetition more cheaply. Moving repeatable work into apps lowers the amount of inference you need, and routing lowers the price of the inference that remains. Use both, in that order.

The honest scoping: this pays off only where work genuinely repeats. A workflow that is different every run has nothing to push into code, and for that workflow a well-tested router is the better investment. Most enterprise workflows are not that workflow. Reason once. Run forever.

Start building on Major

Turn the repeatable steps in your agent workflows into governed apps with their own state, and keep the model for the judgment calls that need it.

Related articles

Frequently asked questions

What is LLM routing?
LLM routing is a decision made before inference. Given an incoming request and a pool of models at different price and capability tiers, a routing policy predicts which model can answer that specific request acceptably at the lowest cost, then sends the request only to that model. The decision is per request and can be wrong.
What is the difference between LLM routing and an LLM gateway?
Routing decides which model answers a request. A gateway provides one API surface across providers, handling key management, logging, retries and fallbacks. A gateway is where a router gets deployed, but adopting one does not mean you are routing. If every request still reaches the same model, spend does not change.
Does LLM routing actually reduce costs?
In measured cases, yes. The RouteLLM paper reports cost reduction of over two times in certain cases. But LLMRouterBench, which evaluated 10 routing baselines across 400,000-plus instances, found several recent approaches, commercial routers included, fail to reliably beat a simple baseline. Test routing on your own traffic with a control arm.
What are the main LLM routing strategies?
Four approaches dominate: rules on request metadata, a trained classifier that predicts whether a cheap model will succeed, a cascade that tries the cheap model and escalates on failure, and semantic similarity that routes by nearest neighbour. Rules are cheapest and hardest to attack, so start there.
When should I use per-query routing instead of fixed rules?
Per-query routing earns its complexity when traffic is heterogeneous in ways metadata cannot see, volume is high enough that a few percentage points of cost matter, and you have labelled history. When traffic already splits by endpoint or customer tier, rules are cheaper, faster and harder to attack.
What are the risks of LLM routing?
The router adds latency to every request and can misclassify. Quality drift is the quiet risk: more hard queries answered by the cheap model, invisible on a cost dashboard. Learned routers are also an attack surface, with published work showing inputs that force selection of the expensive model.