LLM cost optimization
Last verified: June 2026· definition
LLM cost optimization starts with observability: instrumenting every call so you can see cost, latency, and token usage by route and model. From there, the main levers are right-sizing the model per task, caching repeated prompts, batching non-urgent work, and defining cheap, reliable fallback chains.
The discipline is doing all of this behind evals, so each change is a measured trade rather than a quality gamble. How much comes out depends on the workload; the only number worth trusting is one measured against your own baseline.
Because spend usually hides as a single line item, the highest-leverage first step is almost always making it visible by route.
Instrumentation comes first, and it is not optional
Every subsequent decision depends on knowing which routes generate spend, and that attribution cannot be reconstructed after the fact from a provider invoice. The invoice tells you the total; it does not tell you that one background job accounts for most of it. Tagging every call with a route identifier, then tracking tokens, cost, and latency per route, is what turns cost work from guessing into engineering. Teams that skip this step reliably optimise the wrong things.
Right-sizing returns the most, and is the least popular
The largest single saving in most systems comes from not using a frontier model for work that does not need one — classification, extraction, routing, formatting, and simple summarisation. It is unpopular because it feels like a downgrade and because it requires evals to do safely. But the alternative is paying frontier prices for tasks a much smaller model completes at equal quality, which is the most common form of LLM waste.
Every change is a quality trade, so measure it
Routing to a smaller model, trimming context, caching aggressively, and shortening outputs all reduce cost and all can reduce quality. Without evals you cannot tell which ones did, so teams either refuse to optimise or optimise blindly and discover the damage through user complaints. This is why cost work and eval work are the same project in practice: the evals are what make the savings safe to keep.
Common misconceptions
MythCost optimisation means accepting worse output.
RealityIt means matching capability to task. A smaller model that handles a classification route at identical accuracy is not a quality reduction, and the largest savings usually come from exactly these cases where the frontier model was never needed.
MythCaching is the first thing to add.
RealityCaching only pays where requests repeat, and in many systems they rarely do. Without measuring hit rate first you can add a lookup to every request for negligible return. Routing and batching usually return more, sooner.
MythYou should optimise once and move on.
RealitySpend drifts as prompts grow, traffic patterns change, and new routes ship. The durable version of this work is instrumentation and per-route budgets that keep it visible, not a one-off tuning pass.
Frequently asked questions
Where should I start if our LLM bill is growing?
Attribution before optimisation. Instrument every call with a route tag and record tokens, cost, and latency per route. Cost concentrates heavily in most systems, so this alone usually identifies a small number of routes responsible for the bulk of spend — frequently including a background job nobody has reviewed since launch. Optimising without that picture spreads effort across routes that do not matter.
How much can we realistically save?
It depends entirely on how much frontier-model usage is going to work that does not need it, which varies widely between systems. A team already routing carefully has little headroom; a team sending everything to one large model usually has a great deal. The honest answer is that the instrumentation tells you before you commit to a target, which is a good reason to build it first.
Does cost optimisation require changing models?
Not necessarily. Moving non-interactive work to a batch endpoint, caching stable prompt prefixes, and trimming retrieved context that the model never uses all reduce spend with the same models in place. Model routing tends to be the largest lever, but it is not the only one and it is not always the first one worth pulling.