Major

Self-Hosted LLM: What It Costs and When It Pays Off

Every self-hosted LLM guide reprints the same breakeven number, and their own cost tables contradict it. Here is the honest math on GPUs, throughput, and the engineer time nobody counts, plus the cheaper lever almost nobody pulls.

Rahul RamakrishnanUpdated
Open computer case with a graphics card and fans, the hardware behind a self-hosted LLM

Self-hosting an LLM is worth it when prompts cannot leave your network. It is not yet worth it for cost alone, because a fully loaded single-GPU box does not break even against a cheap hosted model until roughly 100 million tokens a day. The cheaper lever sits elsewhere: self-hosting changes the price of a token, and it does nothing about how many tokens your system spends. That second number is the one Major shrinks, by having agents build deterministic apps for repeatable work so the model is called only for judgment.

What counts as a self-hosted LLM

A self-hosted LLM is an open-weight model running on infrastructure your organization controls, either hardware you own or GPU instances in your own cloud account. You hold the weights, you operate the inference server, and you pay for capacity rather than per token.

Local inference is the single-machine case, usually Ollama or llama.cpp with one user at a time. Dedicated or managed inference is a different thing: a provider still runs the GPUs and the operations, and you get isolated tenancy (Modular's handbook draws that line well).

The breakeven number most guides print is wrong

The Prem AI guide published a cost table in February 2026, and the Alpacked guide reproduced it in May with the same four volume rows and the same $850 monthly figure. Both say self-hosting becomes competitive around 2 million tokens a day. Their own table shows $60 for the API against $850 self-hosted at that volume. Their API column scales at $30 a month per million daily tokens, so $850 is crossed near 28 million, not 2 million.

The API column also cannot be reproduced from OpenAI's published GPT-4o mini prices of $0.15 per million input tokens and $0.60 per million output. An 80/20 input/output mix blends to $0.24 per million, or $7.20 a month per million daily tokens.

What a single box really costs

Use Alpacked's figures as inputs. An RTX 5090 at a $3,750 street price plus $1,500 for chassis, CPU, 64 GB of RAM and a 1,200W supply is about $5,250. Over 36 months that is $146 a month. Power at $0.16/kWh is about $85 a month for the whole machine.

Then comes the line no published guide prices: someone has to keep it running. Driver and CUDA drift, model swaps, capacity planning, an on-call rotation for the inference tier. I assume four hours a month at a loaded $120 an hour, which is $480. That is my declared assumption, and you should replace it with your own.

Total: $146 + $85 + $480 = $711 a month, with engineer time as the largest line.

Where the real breakeven lands

Divide $711 by $7.20 per million daily tokens and you get about 99 million tokens a day. Against GPT-4o mini, the API wins at 500K, 2M, 10M and 50M tokens a day.

There is a capacity trap under that number. Alpacked measures vLLM at about 793 tokens per second on an RTX 4090 and says the 5090 runs 67% faster, so one 5090 tops out near 1,320 tokens per second, or about 114 million tokens a day at full utilization. At a realistic 40% it serves about 46 million, less than half of what it needs. You reach the crossover only by adding hardware, and adding hardware moves the crossover.

One qualification flips the answer. Against frontier pricing of $1.25 input and $10.00 output per million, the same mix blends to $3.00 per million, and the single box crosses near 8 million tokens a day. But a 4-bit 32B model that fits on one card substitutes for the cheap tier, not the frontier tier. Which hosted model you are actually replacing decides the whole calculation.

A decision test you can run this week

  • Pull 30 days of real token logs, split into input and output.
  • Name the hosted model you would replace and its current price.
  • Price the box for a model that can do your job, with hardware over 36 months, power, and a stated number of engineer hours.
  • Compute capacity as served tokens per second times 86,400 times honest utilization.
  • Self-host on cost only if sustained volume is 2x the crossover and utilization clears 50%. Otherwise self-host for sovereignty or not at all.

Network isolation is not governance

Keeping inference inside your network answers where data goes. It gives you no permissions model, no audit trail, and no way to say who asked the model what in March. Those come from the application around the model, which is what enterprise-grade AI agents need, with control at the point of action.

Cut how often the model runs on Major

Your bill is price multiplied by quantity. Self-hosting attacks price, and routing between models trims it further. Quantity is where the waste hides: an agent that reasons through the same invoice reconciliation from scratch on every run pays full price for a conclusion it already reached.

Major is the enterprise platform where agents build the software they run on. Here is how a cost-motivated platform or finance-ops lead would set it up:

  • Connectors: NetSuite or Postgres for the system of record, Gmail for vendor email, Slack for escalation, each with scoped credentials rather than shared keys.
  • App layer: the agent builds a reconciliation app once. It holds matching rules, run history and exceptions in a managed database, and runs as code on every later execution.
  • Agent layer: the model is called only for judgment, such as an ambiguous contract term or a collections note, and routes high-risk cases to a person.
  • Trigger: a workflow schedule or an inbound email starts each run, so nothing depends on a chat session staying open.
  • Governance: RBAC on who can run or change the app, and an audit log of every action the agent took and under which credentials.
  • Model choice: the model behind the agent can be a hosted frontier model or an open-weight one you run yourself, so a sovereignty requirement does not change the structure.

Cost is front-loaded and then flat, because the repeatable work no longer passes through the model. Start building free and measure the difference against your own traffic. See also how agentic workflows get structured.

What this does not cover

This piece prices one inference tier on one box. It does not cover model quality differences, multi-region serving, or fine-tuning costs. If you hold regulated data, self-host regardless of any breakeven table. Major is not an inference host and sells no GPUs. The position here is narrower: reducing how often your system reasons is the larger lever, and it applies to any model in any location.

Start building on Major

Run your repeatable workflows as governed apps on Major, with scoped connectors, a managed database and an audit trail, and keep the model for the decisions that need it.

Related articles

Frequently asked questions

Is it worth self-hosting an LLM?
For data sovereignty or regulated workloads where prompts cannot leave your network, yes, and the cost calculation is beside the point. For cost alone, usually not yet. Against cheap hosted models a fully loaded self-hosted box does not break even until roughly 100 million tokens a day. Against frontier-model pricing the crossover falls near 8 million a day, but the open-weight model you can fit on your GPU may not be a fair substitute for a frontier model.
What is the best self-hosted LLM model?
Pick by VRAM budget rather than by leaderboard, since the honest answer is the largest model that fits your card at acceptable throughput. A 16GB card runs a 13B model at Q4 comfortably. A 24GB RTX 4090 runs a 32B model at 4-bit, which is the sweet spot for most single-box deployments. Reaching 70B means about 35 to 40GB, so an RTX 5090 or dual 4090s.
Can I run an LLM on my own computer?
Yes, with VRAM as the ceiling. Budget roughly 0.5GB per billion parameters at 4-bit quantization, so a 24GB card handles a 32B model and stops well short of frontier-scale open weights. Falling back to CPU or system RAM works but costs you 10 to 100 times the speed, dropping from tens of tokens per second to single digits.
How much does it cost to self-host an LLM?
Around $700 to $1,800 a month fully loaded, depending on scale. For a single RTX 5090 box: roughly $146 in hardware amortised over 36 months on a $5,250 build, about $85 in power at $0.16/kWh, and $480 in engineer time assuming four hours a month at $120 an hour. That labour line is the largest of the three and appears in almost no published cost table.
Is self-hosting an LLM more secure?
It changes where your data goes, which is a specific and real benefit for regulated workloads. It does not give you access control, a permissions model, or an audit trail. Knowing that inference happened inside your network does not tell you who asked the model what, under whose credentials, or what it did afterward. Those properties come from the application around the model, not from the network boundary.