Is Self-Hosting an LLM Cheaper Than an API? The Real GPU Math

// In plain English

When you call a hosted model API, you pay per token: a tiny bill for every word in and out. When you self-host an open model, you flip that around. You rent (or buy) a GPU, load an open-weights model like a Llama or a Qwen onto it, and run your own inference server. Now you pay for time, not tokens — the GPU costs the same per hour whether it answers a thousand requests or sits idle.

Here is the mental model. A per-token API is a taxi: you pay only when the meter runs, and you never worry about the car. A self-hosted GPU is a leased car: a flat monthly payment, plus insurance and parking, whether you drive it or leave it in the garage. For a couple of trips a month, the taxi wins easily. Drive every day and the lease gets cheaper per mile. The whole question of self-hosting cost is finding where those two lines cross.

// Why it matters

If you are building anything on top of LLMs, the inference bill is often your single largest variable cost. Getting the self-host-versus-API decision wrong is expensive in both directions: pay per token at huge scale and you burn money you could have saved with your own hardware; rent a GPU for a low-traffic app and you pay for an idle machine that an API would have served for cents.

The trap is that both sides sound obviously right depending on who is talking. "Just use the API, hardware is a nightmare" is true for a weekend project. "Self-host, the open model is free" is true for a company pushing billions of tokens a day. The only way to know which applies to you is to do the break-even math — and that math hinges on one number most people forget: utilization, the fraction of the time your rented GPU is actually doing useful work.

• Cost control at scale. Past a certain volume, a busy self-hosted GPU can serve tokens for a fraction of API list prices. That is real money once you are spending five or six figures a month.
• Privacy and control. Some workloads (health, legal, internal code) can't leave your own infrastructure. Here cost isn't even the deciding factor — but you still need to know what you're paying for that control.
• Latency and customization. Running your own server lets you tune batching, quantization, and which model version you serve. You own the whole serving stack.
• The hidden bill. Self-hosting adds engineer time, idle GPUs, and redundancy that the sticker GPU price hides. Ignoring these is the most common way a "cheaper" self-host turns out more expensive.

// How the cost math actually works

Both options are usually quoted in the same unit so you can compare them directly: dollars per million tokens ($/Mtok). An API just hands you that number on a pricing page. For self-hosting you have to derive it from two things you control: how much your GPU costs per hour, and how many tokens it can produce per hour.

The one formula that decides everything

Your self-hosted cost per million tokens is just hourly GPU cost divided by hourly token output:

cost_per_Mtok = gpu_cost_per_hour / tokens_per_hour

where:
  tokens_per_hour = throughput_tokens_per_sec * 3600 * utilization

  utilization = fraction of time the GPU is actually generating
                (0.0 = always idle, 1.0 = busy 24/7)

Two of these are knobs you tune. Throughput is how many tokens per second your server squeezes out of the GPU — driven by the model size, quantization, and especially continuous batching, which lets one GPU answer many requests at once. Utilization is what fraction of the hour the GPU is busy versus waiting for traffic. The diagram below shows how a fixed hourly cost becomes a per-token cost.

The punchline lives in the last two boxes. The hourly rate is fixed. So the more tokens you actually generate per hour, the lower your cost per token. Double your throughput with batching, or double your utilization with steady traffic, and your $/Mtok is cut in half — without touching the GPU's price tag. This is why an idle self-hosted GPU is so expensive: you are paying full rate for very few tokens.

Why batching changes the answer so much

A single request rarely keeps a big GPU busy — the GPU spends most of its time waiting on memory, not computing. Continuous batching packs many in-flight requests through the GPU together, so total tokens-per-second can climb several times higher under concurrent load than for one lonely request. That means your measured throughput depends entirely on how you test: benchmark with one request and you'll wildly underestimate what the GPU can do, and overestimate your cost per token.

// A worked break-even example

Let's make it concrete with round, illustrative numbers (your real numbers will differ — measure them). Say you rent one GPU for $2.00 per hour and, under healthy batched load, your server sustains 2,000 tokens per second of output. Over an hour that's 2,000 × 3,600 = 7,200,000 tokens — if the GPU stays fully busy.

Look what utilization does. The exact same GPU and model swings from 28 cents to over five dollars per million tokens purely based on how busy you keep it. The hardware didn't change — your traffic did. This single table explains why "is self-hosting cheaper?" has no fixed answer: it depends almost entirely on whether you can keep the machine fed.

Now compare to an API

Suppose a hosted model you'd otherwise use charges, for illustration, $0.60 per million tokens (blended in + out). Drop that line onto the table above and the crossover jumps out:

With these numbers, self-hosting beats the API only above roughly 50% utilization — below that, the idle GPU costs more per token than just paying the API. So the real break-even question isn't "how many requests do I have?" but "can I keep one GPU busy enough?" A rough rule of thumb: you generally need sustained traffic — enough concurrent requests to keep batches full for most of the day — before a single rented GPU pencils out cheaper than a competitively priced API.

// The hidden costs sticker prices hide

The clean formula above assumes the GPU hourly rate is your whole cost. It isn't. The reason teams underestimate self-hosting is that several real costs never show up on the GPU price tag, and every one of them effectively lowers your utilization or raises your hourly bill.

• Idle time between bursts. Traffic isn't flat. Nights, weekends, and quiet hours mean the GPU is paid-for but unused. Unless you can scale down (and pay the cold-start cost of reloading the model), you're buying tokens you never generate.
• Redundancy. One GPU is a single point of failure. Serious uptime means at least two, which roughly doubles your fixed cost while serving the same traffic — halving effective utilization unless load grows to match.
• Engineer time. Someone has to set up the serving stack, keep it patched, monitor it, and get paged when it breaks. That salary cost is invisible in $/Mtok but very real, and it's often larger than the GPU bill for small teams.
• Headroom for spikes. To avoid queueing during peaks you must provision for your busiest moment, which means extra idle capacity during normal load.
• Egress, storage, and the model itself. Storing weights, moving data, and occasionally upgrading the model all add up around the edges.

None of this means self-hosting is a bad idea — at high, steady volume it still wins decisively. It means the honest comparison adds these overheads to the GPU rate before you compute $/Mtok. A useful framing: the API price already bakes in the provider's redundancy, ops team, and idle capacity, spread across thousands of customers. When you self-host, you become that provider — for an audience of one.

// Going deeper

Once the basic break-even clicks, several refinements separate a back-of-envelope guess from a decision you'd bet a budget on.

Rent vs buy. Renting (cloud GPUs by the hour) gives flexibility and turns hardware into a pure operating cost — great while your volume is uncertain. Buying GPUs outright can be dramatically cheaper per token if you run them hot for years, but you now own depreciation, power, cooling, and the risk that a better chip lands next quarter. The break-even there is a separate, longer-horizon calculation.

Input vs output tokens. APIs usually price input (prompt) tokens cheaper than output (generated) tokens, and generation is the slow part on your GPU too. A workload that's mostly long prompts with short answers has a very different economic profile from one that generates long completions. Blend your real input/output ratio into both sides of the comparison instead of using a single flat rate.

Quantization and model choice. Serving a smaller or quantized model raises throughput and fits on cheaper GPUs, slashing your $/Mtok — sometimes at an acceptable quality cost. Choosing the right model size for the job is itself a cost lever, not just a quality one. The serving engine matters too: a throughput-focused server like vLLM will give you a very different cost basis than a single-user tool like Ollama (compare them in Ollama vs vLLM).

Autoscaling and spot capacity. The advanced move is to claw back utilization: autoscale the number of GPUs to follow traffic, batch aggressively, and use cheaper interruptible (spot) instances for non-urgent work. Each technique pushes effective utilization up and your cost per token down — which is the entire game. Everything in self-hosting economics, in the end, reduces to one lever: keep the GPU you're paying for busy generating useful tokens.

Finally, don't treat the decision as permanent. Many teams start on an API to ship fast, watch the monthly bill, and only migrate to self-hosting once volume is high and steady enough that the math clearly flips. Starting on hardware "to save money" before you have the traffic to fill it is the classic, expensive mistake — see local LLM hardware requirements before you commit.