Why are LLMs so expensive?

The pricing model: you pay per token, both ways

LLM providers do not charge per request, per seat, or per feature. They charge per token — the sub-word chunks a model reads and writes — and they charge for two separate streams: the input tokens you send (your prompt plus any context, documents, history, and tool definitions) and the output tokens the model generates in reply. These two are priced differently, and output almost always costs more than input, frequently several times more.

The reason for that asymmetry is structural, not arbitrary. Input tokens are processed in a single parallel pass — the model reads the whole prompt at once. Output tokens are produced one at a time, each generation step depending on everything before it, so producing text is the slower, more compute-hungry half of the job. That is why a short prompt that triggers a long answer can cost more than a long prompt that triggers a short one.

The practical consequence is that a bill is never really about how many API calls you make in the human sense. It is about token volume. A single request that carries a 100,000-token context is not one cheap unit of work; it is a large one. Once you start pricing your own workloads in tokens rather than requests, the surprises on the invoice usually stop being surprises.

The real drivers: context, output, tier, and call count

Four things move an LLM bill, and every cost surprise traces back to one of them. First, context length: everything you put in front of the question is billed as input on every call — system prompts, retrieved documents, conversation history, tool schemas. Stuffing the whole knowledge base into the prompt ‘to be safe’ is one of the most common ways teams quietly triple their input cost.

Second, output length: because output tokens are the expensive half, a model that is allowed to ramble, explain its reasoning at length, or return verbose structured payloads costs materially more than one held to a tight response. Third, model tier: the per-token price gap between a small model and a frontier model is large, and using the top tier for classification, extraction, or routing work that a cheap model handles fine is pure overspend.

Fourth, and easiest to overlook, call count: the same task can be one call or fifty. A retrieval-augmented answer, a multi-step tool-using workflow, or a self-correcting chain all multiply the base cost by the number of round trips. None of these four drivers is exotic — but they combine, and it is the combination, not any single one, that produces a bill nobody can explain.

Why agents blow the bill up

A single completion is linear: you send a prompt, you get an answer, you pay once. An agent is a different shape entirely. It works by looping — call the model, read the result, take an action, feed the result back, call again — and crucially, on each step it re-reads the entire conversation and tool history accumulated so far. The context does not stay fixed; it grows with every step.

That growth is what makes agents expensive out of proportion to the task. If a task takes n steps, and each step re-reads a context that has grown with all the previous steps, total token consumption scales with the square of the task length. Double the length of the task and you can quadruple the cost. This is the superlinear — effectively quadratic — cost curve that catches teams off guard: the demo on a three-step task looks cheap, and the same agent on a thirty-step task is anything but.

This is why swapping to a cheaper model often barely dents an agent's bill. The problem is not the sticker price per token; it is the number of tokens the loop generates by re-reading its own growing history. Understanding agent cost as O(n²) in the length of the task, rather than linear in the number of calls, is the single most important mental model for anyone budgeting agentic systems.

What inference actually costs to run

Per-token prices are not plucked from the air; they are downstream of what it physically costs to run a model. Every token, in or out, is processed on a GPU, and the two hard constraints are memory and compute (FLOPs). The model's weights have to sit in GPU memory, and so does the growing key-value cache that holds the attention state for the current sequence. Larger models need more memory just to load, and longer sequences need more memory to hold their intermediate state.

Compute scales with both model size and sequence length. A bigger model performs more arithmetic per token because there are more parameters to multiply through. And the attention mechanism at the heart of a transformer compares each token against every other token in the sequence, so the work per step grows faster than linearly as sequences get longer. That is the hardware-level reason long contexts are costly: you are paying for the attention computation over the whole sequence, not just the newest token.

Put together, the provider's price per token reflects GPU time, and GPU time reflects how large the model is and how long the sequence is. This is the bridge between the two halves of the cost story: the drivers on your bill — model tier, context length, output length — are exactly the variables that determine how much silicon your request consumes underneath the API.

Will LLMs get cheaper or more expensive?

Both, and that is the honest answer. The per-token price of a given level of capability has fallen steadily and substantially over the last few years, driven by hardware improvements, better serving techniques, quantisation, and competition between providers. If you froze your workload exactly as it is today, it would very likely cost less next year than it does now. On that axis, the trend is clearly downward.

But almost nobody freezes their workload. As models get cheaper and more capable, teams do more with them: longer contexts, more agentic loops, more calls per task, more ambitious automation that would not have been viable at last year's prices. Token consumption is rising faster than per-token prices are falling, especially wherever agents are involved. The unit got cheaper; the number of units exploded.

So the realistic expectation is not ‘LLMs will get cheap and this problem goes away.’ It is that the price of a token keeps drifting down while your total spend keeps drifting up, because the interesting things you can now afford to build consume tokens voraciously. Cost stays an engineering concern precisely because capability keeps expanding what you are willing to spend tokens on — which is exactly why controlling consumption, covered in the sibling guides, matters more over time, not less.