Self-hosted LLMs: when it pays off, and how

Hosted API vs self-host: the actual tradeoff

A hosted API charges you per token and absorbs everything behind it: the GPUs, the serving stack, autoscaling, batching, uptime, and every model upgrade. You pay only for what you use, and when traffic is zero the bill is zero. That elasticity is the entire value proposition, and it is exactly what you give up when you self-host.

Self-hosting means downloading an open-weight model and running inference on GPUs you rent in the cloud or own on-prem. The weights cost nothing. The GPUs cost the same whether they serve a million requests an hour or sit idle overnight — that is the crux. You are converting a variable per-token cost into a fixed capacity cost, and fixed cost only wins when you keep the capacity busy.

So the decision is not "which is cheaper" in the abstract — it depends entirely on your utilisation and your constraints. Below a break-even level of steady traffic, the API is both cheaper and dramatically less work. Above it, or when a hard requirement rules the API out, self-hosting starts to make sense. Everything else in this guide is about finding which side of that line you are on.

When self-hosting actually wins

Compliance and data residency. The clearest case has nothing to do with cost. If regulation or contract requires that data never leaves your infrastructure or a specific jurisdiction, a hosted API may simply be off the table. Self-hosting gives you a private LLM where prompts and outputs stay inside your network — sometimes the only reason that matters.

High, steady volume. If you run large, predictable inference load around the clock, per-token pricing adds up while a saturated GPU does not. Past a break-even utilisation, owning or renting the hardware and keeping it busy costs less than metering every token. The keyword is steady: bursty or low-average traffic wastes the capacity you are paying for.

Customisation and control. Self-hosting lets you fine-tune on your own data, pin a specific model version so behaviour never shifts under you, and control latency by placing inference close to your workload rather than depending on a shared endpoint. If deep customisation or predictable latency is core to your product, that control can justify the operational cost on its own.

The real costs and risks

GPU economics. Whether you rent in the cloud or buy on-prem, GPUs are the dominant cost, and you pay for the capacity, not the usage. On-prem is capex plus power, cooling, and hardware refresh; cloud rental is opex that can quietly exceed API spend if utilisation is low. Either way, the meter runs while the GPU is idle.

Idle waste. This is where self-hosting projections go wrong. Break-even math assumes near-full utilisation, but real traffic has peaks and troughs. Provision for the peak and you waste capacity in the trough; provision for the average and you fail at the peak. That gap between provisioned and used capacity is pure loss, and it is the single most common reason a self-hosting business case that looked cheaper on paper isn't.

Inference operations. The API provider was running a serving stack, autoscaling, request batching, and observability on your behalf. Self-host and you own all of it: standing up a serving engine, batching requests for throughput, scaling under load, monitoring latency and failures, and keeping the whole thing up. That is real engineering time, ongoing.

Keeping current. Open-weight models improve constantly. A hosted API upgrades transparently; self-hosted, you carry the work of evaluating new releases, re-testing, and migrating — plus the model and hardware selection decisions every time the landscape moves. Standing still means slowly falling behind.

How to actually deploy one

Pick the model and the hardware together. These are one decision, not two. The model's size in memory determines the GPU you need, and the GPU you can get determines which models are realistic. Choose the smallest open-weight model that clears your quality bar on your actual tasks — capability you do not use is capacity you overpay for.

Use quantization to fit. Quantization stores the model's weights at lower numerical precision, shrinking its memory footprint so it fits on smaller or fewer GPUs, usually with modest quality loss. It is the standard lever for making a model economical to serve — but the tradeoff is real, so validate quality on your own evals after quantizing, not before.

Serve with batching. Do not call the model one request at a time. A proper inference server groups concurrent requests into batches so the GPU processes many at once, which is what turns raw hardware into throughput. Batching, continuous batching, and sensible concurrency limits are the difference between a GPU earning its cost and sitting half-idle under load.

Instrument everything. You now own the numbers the API used to show you: latency, throughput, GPU utilisation, queue depth, and error rates. Without this observability you cannot tell whether you are past break-even or quietly wasting capacity — which is the whole question. Treat monitoring as part of the deployment, not an afterthought.

A decision framework

Start on a hosted API and optimise it first. Routing each request to the cheapest adequate model, caching repeated work, and batching offline jobs will cut a hosted bill substantially — often enough that the self-hosting case evaporates. Do this before you price a single GPU; it is faster, cheaper, and reversible.

Then ask two questions. First: does a compliance or data-residency requirement forbid the API? If yes, self-host — the decision is made for you and the economics are secondary. Second: is your volume high and steady enough to clear break-even utilisation? Estimate honestly, using peak-versus-average, and be skeptical of projections that assume a GPU stays busy.

If neither answer is a firm yes, stay on the API. Self-hosting is a real engineering commitment — hardware, serving, ops, and a treadmill of model upgrades — and it only rewards teams whose volume or constraints genuinely demand it. When both answers are yes, deploy deliberately: right-size the model, quantize to fit, batch to serve, and instrument so you can prove the machine is actually earning its cost.