How to reduce LLM API costs

Where LLM costs actually arise

Every API call is priced on tokens in and tokens out, and output tokens usually cost several times more than input. So your bill is driven by three multipliers: how many tokens you send, the per-token price of the model you send them to, and how many times you call. Optimising cost means attacking those three, in that order of leverage.

The surprise for most teams is the second-order effect of agents. A single-shot completion is linear. An agent that re-reads its growing context on every step is not — cost grows with the square of the task length. A task that looks twice as long can cost four times as much. This is why "just use a cheaper model" rarely moves the bill: the shape of the workload, not the sticker price, is the problem.

Before optimising anything, instrument. You cannot cut what you cannot see. Capture tokens, latency, cost, and a quality signal per route — per prompt template, per model, per feature — so the next decisions are made on numbers instead of vibes.

Model routing: the biggest lever

Not every request needs your most capable model. Classification, extraction, short rewrites, and routing decisions often run just as well on a small, cheap model; only the genuinely hard prompts need the frontier tier. Cost-aware routing sends each request to the cheapest model that still clears your quality bar for that task, with the expensive model as a fallback when the cheap one is not confident.

The trap is doing this by feel and silently degrading quality. The discipline is to gate routing on evals: you only downgrade a route once a regression suite proves the cheaper model holds quality on that specific prompt. Routing and evaluation are two halves of the same system — which is why teams that cut cost sustainably build both together.

A counterintuitive but real case: sometimes the more capable model is cheaper per finished task, because it needs fewer retries and shorter agent loops. The number that matters is cost per successful task by route, not the headline per-token price.

Caching, batching, and prompt structure

A large share of production traffic is repetitive: the same system prompt, the same documents, the same FAQ-shaped questions. Prompt caching (provider-side or your own) lets you pay once for a stable prefix instead of on every call — structure prompts so the fixed, cacheable part comes first and the variable part last. Semantic caching goes further, serving a stored answer when a new question is close enough to one you have already answered.

For any workload that is not latency-critical — evals, backfills, bulk enrichment, offline classification — use the provider batch APIs, which are typically priced well below real-time. And trim the input itself: retrieve only the context a prompt actually needs rather than stuffing the whole document, and drop dead conversation history that no longer changes the answer.

Controlling output and agent loops

Because output tokens are the expensive ones, constraining what comes back is pure margin. Ask for structured, minimal responses; set sensible max-token limits; and prefer schemas over prose when you are going to parse the result anyway. "Think step by step" is worth it when it changes the answer and wasteful when it does not — measure, do not assume.

For agents, the highest-leverage fix is context management: summarise or prune the working context between steps so the model is not re-reading (and re-paying for) the entire history every turn. Add fallback chains so a transient failure retries on a cheaper path instead of looping expensively on the frontier model.

When self-hosting is (and is not) cheaper

Self-hosting open-weight models can undercut API pricing — but only past a genuine utilisation threshold. Below it, idle GPUs cost more than tokens would have, and you have taken on inference ops, scaling, and observability as new problems. Self-hosting earns its keep when volume is high and steady, when data residency or compliance forecloses the hosted path, or when a fine-tuned smaller model beats a general one on your task. For most teams the first wins come from routing and caching on hosted APIs, with self-hosting evaluated later as a deliberate step.