LLM cost optimization playbook: 7 levers, in order

Before any of the levers, do the read. Most teams running production AI pay more than they need to, and can't say where — not because they made bad choices, but because the bill grew faster than anyone's ability to see inside it.

The standard instrument-and-audit looks like this:

1. Stand up a per-call LLM observability layer (Helicone, LiteLLM + a metrics layer, Maxim Bifrost, or open-source fast-litellm).
2. Per-route, per-model, per-user: tokens, latency, cost, and (if you have it) per-business-KPI.
3. Sort routes by cost. Spend is usually concentrated in a handful of routes; find out how concentrated yours is.
4. For each top route, ask: could a smaller model do this with a quality gate?

Once you can see it, the rest is the 7 levers below.

Model routing is the single biggest lever — and the one most teams under-use.

The pattern: a small model (Haiku, Flash, GPT-4o-mini, DeepSeek-V3) for high-volume, low-stakes calls; a capable model (Sonnet, Opus, GPT-4o, Claude Fable) for the hard reasoning. A quality gate on the boundary decides which calls escalate.

Common splits:

• Classification, extraction, simple Q&A → small model (often most of the call volume)
• Summarization, rephrasing, formatting → small model with structured outputs
• Multi-step reasoning, planning, code generation → capable model (a small share of calls that can carry most of the cost)

To size it: the share of cost on routes a smaller model can serve, times the price gap between tiers. The quality gate is the only tricky part: LLM-as-judge on a small held-out set, sampled online, gives you confidence that the small model isn't silently degrading quality.

Prompt caching is the second-biggest lever — and the one most teams haven't turned on.

Most LLM APIs (Anthropic, OpenAI, Google, DeepSeek) support caching of the prompt prefix. The rule: anything in the prompt that is stable across requests should be cached. Common cached prefixes:

• System prompt (often 500–2,000 tokens of instructions + tools)
• Retrieved documents / RAG context (often 2,000–20,000 tokens; the same doc is sent to many requests)
• Tool definitions (every tool description, every parameter schema)
• Examples / few-shot prompts (stable across calls)

To size it: cached prefix reads bill at a fraction of the base input price (check your provider’s cache pricing), so the saving is the cacheable share of input tokens times that discount. The cost is cache-control plumbing: which part of the prompt is the stable prefix, and which part is the request-specific part.

For deterministic routes, the entire response can be cached.

Safe to fully cache:

• Classification (label the request, cache the label)
• Extraction (same input → same output, unless the model is non-deterministic)
• FAQ / RAG over a known doc set (cache by doc-set version + query)

Not safe to cache (or cache only the prefix):

• Code generation (output is highly variable)
• Multi-step reasoning (the path matters)
• Any route that depends on real-time data

To size it: the duplicate-request rate on safe routes, from a week of request fingerprints, times those routes’ share of the bill.

Most providers offer a batch API at 50% the synchronous price. The trade-off is latency: batches typically complete in minutes to hours, not seconds.

Routes that fit batching:

• Nightly evaluation runs
• Bulk document processing (RAG ingestion, batch summarization)
• Reports that don't need real-time

Routes that don't fit:

• Anything user-facing in a chat flow
• Anything that gates a downstream action in <5 seconds

The saving is the provider’s batch discount (50% at the major providers) on the routes that fit; their share of the bill decides what that is worth.

The cheapest prompts to cut are the ones nobody reads.

Common bloat:

• Retrieved documents that don't change the answer. Measure: drop one retrieved doc at a time, measure the answer quality, keep the docs that move the needle.
• Examples that don't generalize. 3 examples > 20 examples if the 3 are sharp.
• Conversation history that doesn't matter. Summarize or drop turns after the 5th; the answer rarely depends on the full thread.
• Output cap that's too high. Most teams set max_tokens to 4,000 by default. Most routes don't need more than 500–1,000.

To size it: input tokens you can remove per call without moving eval scores. Hard to find without an eval that proves the compression isn't degrading quality.

The most expensive lever, and the most often missed.

Most RAG systems over-fetch. They send 20 chunks of context to the model, when 3 would have answered the question. The cost is:

• The 20 chunks of input token cost
• The latency (longer prompts = slower responses)
• The quality (too much context confuses the model)

The fix: re-rank with a smaller model, retrieve fewer high-quality chunks, and use compression / summarization on the retrieved docs before they reach the model.

To size it: chunks sent per query today versus the fewest that hold answer quality on your eval set.

The hardest lever to pull — and often the biggest absolute saving.

Look at your top 3 routes by cost. For each:

• Is the use case defensible? Could you defend it in front of a board? Or is it a feature nobody uses that someone added in 2024?
• Is the cost per business KPI reasonable? If the route costs $0.50 per call and drives $0.10 of revenue, the unit economics are broken. Either raise the price, lower the cost, or kill it.
• Is it a demo that became production? A demo that became production is a feature that was never reviewed. Review it now.

If one of your top routes can't be defended, killing it is the cleanest reduction you'll ever get — and it needs no quality gate.

None of this sticks without cost observability.

The minimum viable cost observability:

1. Per-call observability: tokens, latency, cost, model, route, user.
2. Per-business-KPI rollup: cost per resolution, cost per conversion, cost per user per day.
3. Alerting on cost anomalies: per-route spend > 2x baseline = page someone.
4. Cost in the dashboard next to latency: not in a separate spreadsheet, not in a finance team's monthly close.

Without this, the wins from week 1 creep back by month 3 — because nobody can see them creeping.

When to do this yourself, when to hire: