LLM inference cost
Last verified: June 2026· definition
On a hosted API the formula is simple: input tokens × input price + output tokens × output price, where output is typically 3–5x more expensive than input, and reasoning tokens bill as output. Cost therefore scales with usage, which is why AI line items surprise finance in a way fixed infrastructure does not.
Self-hosting flips the shape: you pay for GPUs whether or not requests arrive, so cost per request falls with utilization. That is why self-hosting only pencils out at genuinely high, steady volume — or when data residency makes it the only option.
The number that matters is never the headline price per million tokens; it is cost per successful task, by route. A “cheap” model that needs three retries and a human fix is expensive. That is why cost work starts with instrumentation, not with switching models.
Why the bill grows faster than usage
Token spend is not linear in requests, because the prompt usually grows too. A chat feature sends the whole conversation on every turn, so a twenty-turn conversation costs far more than twenty single messages. Retrieval adds to the same effect: better recall usually means more retrieved context, and that context is billed on every call. Teams frequently find that usage doubled while cost quadrupled, and the extra came from prompt growth rather than traffic.
Where the money actually goes
Cost concentrates far more than most teams expect. In a typical production system a small number of routes generate the large majority of spend, often a background or batch job nobody has looked at since it shipped. This is why per-route attribution is the first instrumentation to add: without it, optimisation effort gets spread evenly across routes that do not matter, and the one route that does goes untouched.
The levers, roughly in order of return
Right-sizing the model per route usually returns the most, because a large share of calls are classification, extraction, or formatting that a smaller model handles at a fraction of the price. After that: prompt caching on stable prefixes, moving non-interactive work to a batch endpoint, trimming retrieved context to what is actually used, and caching repeated identical calls. Ordering matters — routing changes make the others cheaper to reason about.
Common misconceptions
MythThe cheapest model per token gives you the cheapest system.
RealityPer-token price is only one term. A weaker model that needs longer prompts, more retries, or a verification pass can cost more in total than a stronger model that gets it right first time. Compare cost per successful outcome, not per token.
MythOutput tokens and input tokens cost about the same.
RealityOutput is typically several times more expensive than input on hosted APIs, and reasoning tokens bill at output rates. A change that shortens responses often moves the bill more than one that shortens prompts.
MythSelf-hosting is cheaper.
RealityOnly above sustained high utilisation. You pay for GPUs whether requests arrive or not, and the operational staffing is a real recurring cost that spreadsheets routinely omit. Below that threshold an API is usually cheaper as well as simpler.
Frequently asked questions
How do I work out what an LLM feature costs before building it?
Estimate tokens per call rather than calls per month, because that is where the variance lives. Take a realistic prompt including system instructions and any retrieved context, count its tokens, add expected output length, and multiply by the provider price with output weighted at its own higher rate. Then multiply by expected volume. The common error is estimating from a short test prompt rather than a production-shaped one, which understates cost by a wide margin.
Why did our costs rise without a traffic increase?
Almost always prompt growth. System prompts accumulate instructions, retrieval starts returning more chunks, and conversation histories lengthen. All of it bills on every request. Tracking average input tokens per route alongside spend makes this visible as it happens rather than at the end of the month.
Is it worth optimising cost before scaling?
The instrumentation is, the optimisation usually is not. Add per-route cost and token attribution early because it is cheap to build and impossible to reconstruct retroactively. Defer the actual tuning until traffic patterns are real, since optimising against guessed usage tends to target the wrong routes.