Self-Hosting vs API: When Does Running Your Own LLM Pay Off?

// In plain English

When you build with a large language model, you have two basic ways to run inference. You can call a provider's API — OpenAI, Anthropic, Google — and pay a small amount for every thousand tokens you send and receive. Or you can self-host an open-weight model like Llama, Mistral, or Qwen, by renting GPUs (or buying them) and running the model yourself.

The cleanest analogy is renting a car versus owning one. The API is a taxi: you pay per ride, you never touch the engine, and if you take zero rides this month you pay nothing. Self-hosting is buying a car: there's a big fixed cost whether you drive it or not, you handle the maintenance, but if you drive enough miles the per-mile cost drops below what a taxi would charge.

The whole decision comes down to one question: how much do you drive? A taxi is cheaper for occasional trips. Your own car wins only once you rack up serious mileage — and even then only if you actually keep it on the road instead of parked in the garage.

// Why it matters

Inference is usually the single largest recurring line item in an AI product's bill. Getting the build-vs-buy call wrong doesn't just cost a little extra — it can mean paying for idle GPUs that sit at 5% utilization, or conversely paying a premium per token at a scale where you could have cut your bill by 70%.

The reason this decision is so easy to get wrong is that most cost comparisons only look at the per-token sticker price. They put the API's published rate next to the hourly cost of a GPU and declare a winner. That comparison is misleading, because the real cost of self-hosting includes a stack of expenses that never appear on the GPU rental invoice:

• Idle time. A rented GPU bills by the hour whether it's serving 1,000 requests or zero. The API bills only for tokens you actually use.
• Utilization. You pay for the whole GPU but rarely use all of it. A box that's busy 20% of the time effectively costs 5x its hourly rate per useful request.
• Engineering and ops. Someone has to set up serving, autoscaling, monitoring, model updates, and on-call. That salary is real money the API price already absorbs for you.
• Reliability headroom. To avoid dropping traffic at peak, you over-provision — extra GPUs that mostly sit waiting for the busy hour.

When you fold all of that in, the break-even point sits much higher than the naive per-token math suggests. A builder who understands the full picture can confidently say "the API is right for us at this stage" — and just as confidently know the volume at which switching starts to pay. This is core LLMOps territory: the operational cost of running models in production.

// How the two cost models actually work

The two options have fundamentally different cost shapes. Understanding the shape matters more than any single number, because it tells you which option grows cheaper as you scale.

The API model: pure variable cost

An API charges per token, split into input (the prompt you send) and output (what the model writes back), with output usually priced higher. Your monthly bill is simply tokens × price. There is no floor and no ceiling: send nothing, pay nothing; send ten times more, pay ten times more. The cost is a straight line through the origin.

The self-host model: big fixed cost, then cheap per request

Self-hosting flips the shape. You commit to a chunk of GPU capacity — say one or more accelerators running around the clock — and pay for that capacity regardless of traffic. Within that capacity, each additional request is nearly free. So the cost is flat at a high baseline, and your cost per request falls as you push more requests through the same hardware. That falling-per-unit curve is the entire reason self-hosting can ever win.

Put the two on the same graph and they cross at one point: the break-even volume. Below it, the API's pay-as-you-go line sits under the self-host baseline — the API is cheaper. Above it, the self-host curve dips below the rising API line — self-hosting is cheaper. The whole exercise is figuring out where that crossing point is and which side of it your real traffic lives on.

// A worked example with real numbers

Let's make this concrete with round, illustrative figures. (Prices change constantly — treat these as a method, not a current quote. Plug in today's numbers when you do it for real.)

Say each request uses about 1,000 input tokens and 500 output tokens — 1,500 tokens total. Suppose a mid-size API charges roughly $0.50 per million input tokens and $1.50 per million output tokens. Then one request costs about (1,000 × $0.50 + 500 × $1.50) / 1,000,000 ≈ $0.00125 — just over a tenth of a cent.

Now the self-host side. Suppose you rent a single capable GPU for about $2.00/hour, which is $1,440/month if you run it 24/7. That box, well-tuned, might serve on the order of 2 requests per second sustained for a model of this size — but only if you keep it busy. Here's how the true cost per request depends entirely on utilization:

Read that table carefully, because it's the whole article in one place. At 5% utilization — a GPU mostly idle, waiting for traffic — self-hosting costs more than 4x the API per request. The break-even sits somewhere around 25% sustained utilization, which already implies well over a million requests a month flowing steadily through that one box. Only once you're keeping the GPU genuinely busy does self-hosting pull clearly ahead.

And notice what's still missing from the GPU column: the engineer's salary to run all this, the second GPU you keep on standby for failover, and the over-provisioning to survive traffic spikes. Fold those in and the real break-even volume climbs higher still.

// When each option wins

You don't actually need to memorize break-even formulas to make a good call. A handful of signals point clearly one way or the other.

The clearest pattern: start on the API, switch later if and when the math flips. Almost every product begins below break-even, with traffic that's too low and too spiky to keep a GPU busy. Paying per token there is not a compromise — it's the right answer. See from prototype to production for how this fits the usual maturity path.

Self-hosting earns its keep when all of three things are true at once: volume is genuinely high, traffic is steady enough to keep the hardware busy, and you have the team to operate it. Miss any one and you're likely paying more than the API while doing more work. Compliance is the main exception — if data legally cannot leave your infrastructure, you may self-host below break-even and simply accept the premium as the cost of control.

// Going deeper

Once the basic break-even clicks, several refinements matter in practice — and most of them raise the bar for self-hosting being worth it.

Batching and throughput. Modern serving engines (vLLM, TGI, SGLang) use continuous batching to pack many requests onto a GPU at once, which is what makes high utilization achievable. But batching trades latency for throughput: pack tightly and individual requests wait longer. Your achievable requests-per-second — the number under the divisor in every cost calculation — depends heavily on how you tune this, and on the model size and quantization you choose.

The hybrid answer is often best. It's rarely all-or-nothing. A common production design routes cheap, high-volume, latency-tolerant traffic to a self-hosted open model and sends the hard or rare queries to a frontier API. An LLM gateway with model routing makes this switch transparent to your application, and provider failover lets a self-hosted primary fall back to an API when your GPUs are saturated or down.

Hidden costs of ownership. Self-hosting adds operational surface the API hides entirely: model version upgrades, security patches, GPU driver hell, capacity planning, and being your own on-call when inference falls over at 3am. None of that shows on the rental bill, but all of it consumes engineering time. This is exactly why you need real observability on a self-hosted stack — you're now responsible for the uptime an API provider used to guarantee.

Buying vs renting GPUs. Owning hardware outright can beat cloud rental on raw cost at very large, sustained scale, but it adds capital expense, depreciation, datacenter logistics, and the risk that next year's faster, cheaper accelerator makes your fleet obsolete. For all but the largest operators, renting keeps you flexible and is the sane default.

Where the curve is heading. API prices have fallen steadily as competition and efficiency improve, which keeps pushing the break-even volume upward and shrinking the window where self-hosting clearly wins on cost alone. That trend tilts the default further toward "use the API until you have a concrete, measured reason not to." The durable reasons to own your stack are increasingly about control — data residency, latency tuning, model customization — rather than headline price per token.