LLM Cost Optimization: What Works and What It Misses
Caching, routing, and batch APIs cut the price of a model call. Here is which lever fits which symptom, and why removing repeatable calls beats discounting them.

How do you reduce LLM costs?
You reduce LLM costs by matching one of six levers to the symptom in your usage data: prompt caching, batch processing, model routing, output-token limits, retrieval, and semantic caching. Each one lowers the price of a call your system still makes. The structural fix is to remove repeatable calls by having the agent write that work into a deterministic app. Major is built for that second move, and the rest of this piece shows where each approach fits.
This is written for platform and AI leads whose bill keeps climbing while per-token prices fall.
Why the bill grows while token prices fall
Spend on any model-backed feature resolves to one identity: price per token, times tokens per call, times calls per unit of work. Most cost guides attack the first factor and some of the second. Agentic systems spend on the third.
A support agent that handles one ticket may issue a dozen calls: a classification pass, two tool calls whose results re-enter context, a retry after a malformed function call, a summary, a final draft. One unit of work, twelve billable events, each resending the system prompt.
Both halves of the paradox are measurable. Stanford HAI's 2025 AI Index tracked the cost of querying a model at roughly GPT-3.5 quality falling from $20 per million tokens in November 2022 to $0.07 in October 2024, a decline of more than 280 times. Menlo Ventures put enterprise generative AI spending at $37 billion in 2025, up from $11.5 billion in 2024. Cheaper tokens were spent on more tokens, because an agent loop makes many calls where a chatbot made one.
Six levers and the symptom each one treats
Hold one workload in mind: a ticket-triage agent that reads an inbound message, classifies it, pulls the customer's plan and recent orders, and routes it. Provider prices below were checked against vendor documentation on 21 August 2026. Re-check before you plan against them.
- Same prompt and tool schemas resent every call: prompt caching. Anthropic charges cache reads at 0.1x base input, with writes at 1.25x for five minutes and 2x for one hour, so a five-minute cache pays for itself after one hit. The prefix must be identical and come first. A timestamp near the top of your system prompt quietly disables it.
- Work nobody is waiting on: batch processing. Anthropic's Message Batches API, OpenAI's Batch API, and Google's batch mode each document a 50% discount. OpenAI applies it against a 24-hour window. Nightly classification, backfills, and evaluation runs qualify. Interactive triage does not.
- One expensive model serving easy and hard requests: routing and cascades. A cheap model answers first and escalates on low confidence, as Chen, Zaharia, and Zou described in FrugalGPT in 2023. You need a quality gate you trust, or you ship worse answers cheaply.
- Long, chatty responses: output-token control. Output bills at five times input across the Claude line. Sonnet 5 lists at $2 per million input and $10 per million output. Structured formats and max-token limits move real money, but capping a reasoning model mid-thought produces an answer you paid for and must retry.
- Whole documents pasted into context: retrieval. A targeted fetch replaces the paste. The cost moves into an embedding pipeline and a vector store, and retrieval quality becomes a correctness property of your system.
- Many near-identical requests: semantic caching. A hit skips the call. A loose threshold can collide "cancel my subscription" with "cancel my last order" and return a fluent, wrong answer with no error in your logs.
Prompt caching alone justifies an afternoon on most agent workloads. Start there.
The limit all six share
Five of the six lower the price of a call the system still makes. Only semantic caching skips a call, and it does so by guessing the request was a duplicate.
For a large slice of production work, the call being discounted is the model re-deriving a procedure whose shape stopped changing weeks ago. Your triage agent has classified forty thousand tickets. The routing rules settled in week two. Every ticket since has paid a model, at a discounted rate, to rediscover a path it already knows.
The other move is for the agent to work the procedure out once, write it as a deterministic app, and run the app. Cost front-loads into the reasoning that produces the app, then flattens as the app runs. State lives in the app's database instead of a context window, and the action is code someone can read, permission, and audit. Reason once. Run forever.
What to keep in the model and what to turn into an app
Keep work in the model while its rules are still moving. Turn it into an app when they settle. Work whose rules change weekly costs more to rebuild than prompts cost to run, and a workflow that runs a handful of times a month does not repay the build.
- Keep in the model: novel judgment, open-ended drafting and tone, exploratory analysis, edge cases a person reviews anyway.
- Turn into an app: lookups, joins, and record updates; work that runs hundreds of times a week; rules unchanged for a month; anything an auditor will ask about later.
Build the cost-flat triage agent on Major
Major is the enterprise platform where agents build the software they run on. Here is the ticket-triage workflow built so that cost stops tracking volume.
- Connectors: Zendesk for inbound tickets and updates, Slack for escalations, each with scoped credentials held at the platform layer.
- App that holds state: a triage app with a managed database of tickets, classifications, routing decisions, and outcomes, plus a view where a support lead edits the routing table.
- Deterministic path: the agent writes the classification rules, routing table, and account lookup into the app once. Matching tickets run through it without a model call.
- Agent judgment: the model keeps the ambiguous ticket, the angry customer, and any case the app has not seen. When a new pattern recurs, the agent proposes an app change.
- Trigger: a workflow fires on each new Zendesk ticket and runs the app first.
- Governance: RBAC on who edits routing rules, an audit log of every action with its credentials, and replayable runs.
- Model choice: Major builds with Claude, Kimi, Gemini, Muse Spark, ChatGPT, and Grok, and state lives in the app, so swapping in a cheaper model for the judgment step is a configuration change.
New accounts get $100 in free credits, enough to build and run this triage agent.
What this does not solve
The model does less on Major, not nothing, and judgment calls still cost tokens at whatever rate you negotiate. Rebuilding an app is wasted effort when the rules are volatile. Teams without repeatable, high-volume work get most of their savings from the six levers above, and should take them first. We do not promise a savings percentage. The shape is structural: front-loaded, then flat.
Start building on Major
Build the agent that writes your repeatable steps into governed apps, so your model bill stops climbing with ticket volume. Run it on your own Zendesk and Slack data and read the audit log yourself.
Related articles
Frequently asked questions
- What is the biggest driver of LLM costs?
- Calls per task, in most agentic systems. Spend is price per token multiplied by tokens per call multiplied by calls per unit of work, and model choice only touches the first factor. An agent that plans, calls two tools, retries once, and summarizes bills five times for one ticket, resending the same system prompt each time. Count your calls before you shop for a cheaper model. Major attacks that middle factor directly, because work an agent has written into a deployed app runs without a call at all.
- Does prompt caching reduce output quality?
- No. The cached prefix is the same tokens read back from provider storage instead of reprocessed, so the model sees identical input. Anthropic bills cache reads at 0.1x the base input rate for the same content. Semantic caching is the one that can change output, because it serves a stored answer when a new query only looks similar. Caching still leaves a bill that grows with volume, which is the part Major changes by moving settled steps out of the model and into code.
- Can you cut LLM costs without changing application code?
- Partly, and only up to a point. A gateway adds routing, caching, and budget limits in front of your calls without touching application logic, and that is a genuine reduction. It does not change how much reasoning your application performs. Cutting call volume means changing what the application asks the model to do, which is code. On Major the agent writes that code itself, turning a repeated reasoning step into an app it can then run without re-deriving the procedure.
- How much does batch processing save?
- Anthropic, OpenAI, and Google each document a 50% discount for asynchronous batch requests. Anthropic applies it to input and output on the Message Batches API, OpenAI applies it against a 24-hour completion window, and Google halves eligible Gemini rates. The constraint is turnaround, so batch fits backfills and evaluations rather than anything a person is waiting on. It also halves a number that still doubles when your volume doubles, unlike a step Major has already turned into deterministic code.
- When is it worth building an app instead of optimizing prompts?
- When the work runs often and its rules have stopped changing. High frequency with a stable output contract means the model re-derives the same procedure on every run, and that is the case for writing it as deterministic code once. Cost front-loads into the reasoning that produces the app, then flattens as usage grows. For rules that shift weekly, keep the work in the model, because rebuilding the app each week costs more than the prompts did.