LLM Gateway: What It Governs, and What It Cannot
Every LLM gateway page sells governance and none of them specify it. Here is what a gateway actually controls, the RBAC and credential-scoping questions to ask before you buy, and the class of risk that no gateway can see.
What is an LLM gateway?
An LLM gateway is a single control point in front of multiple model providers. Applications call one endpoint with one request shape, and the gateway handles routing, failover, caching, key management, cost tracking and request logging behind it. It governs model traffic: which calls are allowed, by whom, at what cost. It cannot govern what an agent does with the answer, because that action happens outside the gateway's view.
That boundary is where Major fits. A gateway controls the model call. Major is the enterprise platform where agents build the software they run on, so the action that follows the model call runs inside a deterministic app with scoped credentials, role-based access and its own audit log. If you run agents in production, you want both layers, and the second one is the one most teams are missing.
LLM gateway vs AI gateway vs API gateway vs MCP gateway
Most vendors use "LLM gateway" and "AI gateway" interchangeably. TrueFoundry says so in its own FAQ, and LiteLLM markets one product under both labels. Some vendors use "AI gateway" as the broader term, covering MCP traffic and agent workflows alongside model calls. Read the capability list, not the noun.
OpenRouter draws a useful four-way line:
- API gateway: HTTP authentication, rate limits, load balancing and TLS for service traffic.
- LLM gateway: models, tokens, cost and provider health. It is model-aware, which an API gateway is not.
- Agent gateway: workflows, tools and handoffs across multi-step execution.
- MCP gateway: access control for MCP tools and data sources.
An LLM gateway and an API gateway solve different problems and are commonly run together.
What does an LLM gateway do?
The near-universal capabilities are a unified API, routing, failover, observability and cost tracking. The variable ones are the ones that matter for governance.
- Routing and failover: choose a provider or model by policy, and switch during an outage. Retry safety varies, and it matters when a request can trigger a side effect.
- Caching: exact-match caching is easy to reason about. Semantic caching can return an answer that fits the topic but misses current context, so set freshness rules before enabling it.
- Key management: LiteLLM documents virtual keys, budgets and integrations with Vault, AWS Secrets Manager and Azure Key Vault. Other products stop at basic key storage.
- Audit logging and RBAC: the least specified feature in the category. Several vendors advertise audit logs without saying what is captured, how long it is kept, or who can read it.
- Guardrails: prompt and response inspection for PII, secrets or injection patterns. These produce false positives and false negatives, so they are a filter, not a security boundary.
Questions to ask before you buy
Governance is the most-claimed and least-specified capability in this category. Test for mechanism:
- Can you issue, rotate and revoke credentials per app, team and environment?
- Can permissions be scoped by model, provider, project and action?
- Which fields appear in each audit record, and how long is it retained and exportable?
- Who can read the logs, and does RBAC map to our identity provider?
- What is redacted per request, and can policy differ by data class?
- How is multi-tenant isolation implemented and tested?
- How are retries made safe for requests with side effects?
- What happens when the gateway is unavailable?
A page that says "control who can access models" without answering these describes intent, not enforcement.
Where gateway governance stops
A gateway sees a model request and a response. Take an agent that reads a customer record, asks a model whether to issue a refund, then updates the system of record. The gateway can log the inference call. It cannot prove which record changed, under whose permission, or why the write was allowed. A model-call log is a record of what the agent asked, not what it did.
Three misconceptions follow from this. A gateway does not make AI usage governed; it governs model traffic. Cost dashboards are operational control, and governance also needs identity, permissions, retention and an accountable action record. And an LLM log is not an agent audit trail, since it omits tool calls, record changes, messages sent and approvals.
Build the governed action layer on Major
The model call is the cheap, narrow part of an agent workflow. The consequential part is the sequence of reads and writes around it. Here is how that looks on Major for the refund example.
- Connectors: the agent reaches Zendesk, Stripe and Slack through Major's connectors, with scoped credentials per connector rather than a shared key in a prompt.
- App that holds state: a refunds app with a managed database stores each case, the policy checked, the amount, the decision and the outcome. State lives in the app, not in a context window.
- Agent and judgment: the agent reasons about ambiguous cases, such as a disputed charge or a partial refund. When it works out the repeatable path, it builds that path into the app so the next run executes as code.
- Trigger: a workflow fires on a new ticket or a schedule, runs the deterministic steps, and calls the agent only where judgment is needed.
- Approval: refunds above a threshold route to a person in Slack before the write lands.
- Governance: role-based access decides who can run, edit or approve the app, and every action is written to its audit log with the identity it ran under.
Run your gateway in front of the model calls and Major around the actions. Start building free and stand up the first governed app in a session.
What this article does not cover
This is not a production gateway configuration or a latency benchmark. Vendor latency claims are not methodologically comparable, and an extra hop adds latency, so measure your own traffic. It also does not rank gateways.
What we believe at Major
Major does not replace a gateway. If three teams call three providers, a gateway solves provider outages, opaque spend and key sprawl, and you should have one. Our position is narrower. Governance has to sit where the agent acts, and the only place that is inspectable by construction is software. When an agent builds an app for a repeatable step, the work runs the same way every time, holds its own data, and writes its own logs. The model does less, and what it does is visible. Reason once. Run forever.
Start building on Major
Put the governed action layer behind your gateway: build the app your agent runs, scope its credentials, and get an audit trail of every record it touches.
Related articles
Frequently asked questions
- What is an LLM gateway?
- An LLM gateway is a single control point in front of multiple model providers. Applications call one endpoint with one request shape, and the gateway handles routing, failover, caching, key management, cost tracking and request logging behind it. The property that distinguishes it from an ordinary API gateway is model-awareness: it understands tokens, per-model cost and provider health, and can make routing decisions on all three.
- What is the difference between an LLM gateway and an AI gateway?
- Most vendors use the terms interchangeably. TrueFoundry's own FAQ states that an LLM gateway and an AI gateway are generally considered the same thing, and LiteLLM markets one product under both labels. Some vendors treat AI gateway as the broader term, covering non-LLM models, MCP traffic and agent workflows alongside model calls. Read the capability list rather than the noun.
- Do I need an LLM gateway?
- Yes, once more than one team is calling more than one provider, or once nobody owns the aggregate spend and keys are living in three places. A gateway solves provider outages, opaque cost attribution and key sprawl directly. A single application on a single provider does not need one, and adding a hop for it buys latency without buying control.
- Does an LLM gateway give me an audit trail?
- It gives you a record of model traffic, which is a different thing. What is captured, how long it is retained, whether it is exportable and who can read it vary widely across vendors, and several advertise audit logging without specifying any of it. The record covers what an agent asked a model, not what the agent then did with the answer. That action usually happens outside the gateway entirely.
- What is the difference between an LLM gateway and an API gateway?
- Model-awareness. An API gateway handles HTTP concerns: authentication, rate limiting, load balancing and SSL termination. OpenRouter notes it has no understanding of tokens, model capabilities or inference cost. An LLM gateway tracks token usage, per-model spend and provider health, and routes on those signals. The two solve different problems and are commonly run together.