LLM Cost & Performance Optimization
Last verified: October 2026· engagement
Production LLM spend usually grows faster than anyone’s ability to see inside it. We baseline it per route, audit prompts, model routing, caching, batching, and fallback chains, then ship cost-aware routing and observability.
- Outcome
- A measured cost reduction against a recorded baseline, quality held
- Timeline
- 4–6 weeks
- Pricing
- $15–40k, or outcome-priced (25% of first-year savings)
- Buyer
- CTO, VP Eng, Head of Infra
The problem
You shipped AI features and the bill kept climbing. Nobody can say which prompts, models, or call patterns are driving spend — so every proposed cut is a guess, and latency complaints pile up on top.
What we do
- Instrument every LLM call: tokens, latency, cost, and quality, per route.
- Right-size models per task and add cost-aware routing with quality guards.
- Add prompt + response caching, request batching, and graceful fallback chains.
- Wire cost and latency dashboards so the savings stay visible after we leave.
How it fits together
What you get
Built on our open source
fast-litellm — Rust acceleration for LiteLLM — faster connection pooling, rate limiting, and memory-intensive workloads.
How the cost work actually runs
Week one is measurement, not savings. Before anything is changed we tag every model call with a stable route name, split input from output tokens, separate cached reads from uncached ones, and record the finish reason. Almost every engagement produces the same surprise at this point: spend is far more concentrated than anyone expected, with a small number of routes carrying most of the bill, and at least one of those routes turns out to be an internal or batch job nobody would have defended if they had seen the number.
With attribution in place the levers get applied in a fixed order, cheapest first. Right-sizing the model per route comes first because it is reversible and the effect is immediate. Prompt-cache prefixes come next, and the usual finding is that a cache was technically enabled but never hitting, because a timestamp or a reordered list sat near the top of a prompt and invalidated the prefix on every call. Then batching for anything not on a user's critical path, then prompt compression where the context has accreted instructions that no longer do anything, then fallback chains so a provider timeout does not turn into an expensive retry storm.
Every change ships behind an eval, and that constraint is what makes the exercise defensible rather than a gamble. A route is only moved to a cheaper model once there is a regression suite built from real traffic on that route and a scored comparison showing quality held. That sequencing is also why we insist on the observability week: the same instrumentation that finds the money is what proves the savings were free. When we leave, the dashboards and the runbook stay, which is the difference between a cost cut and a cost discipline.
One thing worth setting expectations on early: the number we report at the end is a measured change against a recorded baseline, not a modelled projection. That means it is smaller than the figure a spreadsheet would give you and considerably easier to defend when finance asks where it came from. It also means the baseline week is not optional — without it there is nothing to compare against, and a saving nobody can evidence tends to be treated as one that did not happen.
When this is not the right engagement
Cost work pays for itself or it does not, and that is knowable before anyone signs. We will say so on the call when the arithmetic does not work:
- Your eligible LLM spend is under roughly $10k a month. At that size even a good result is worth less over a year than the engagement costs; the honest advice is a day of instrumentation and our reduction guide.
- The bill is dominated by a single fixed-price commitment or a reserved-capacity deal, where the lever is commercial rather than technical.
- Nobody can give us read access to traffic and prompts. An audit without data is a workshop, and we would rather tell you that up front.
What “done” means
These are written into the agreement before work starts, so you, your finance team and procurement can see the finish line up front.
Acceptance
- Cost per accepted task on the in-scope routes is measured against the week-one baseline over the agreed window.
- Eval scores on those routes are at or above the agreed threshold — a cheaper answer that fails them does not count.
- p95 latency stays within the limit agreed at kickoff.
- Routing, caching and dashboards run in your environment, with a runbook your team has used once without us.
If you choose outcome pricing
- The workload in scope is named in the agreement.
- Baseline: cost per accepted task, measured read-only in week one.
- Quality threshold: agreed eval scores; a cheaper completion that fails them does not count as a saving.
- Measurement window and method are fixed before any change ships.
- Savings are normalised for changes in traffic volume and task mix.
- Provider list-price cuts are excluded — we only take credit for our changes.
- New operating costs we introduce (gateway, cache, infrastructure) are netted off.
Not included
- Renegotiating provider contracts or reserved-capacity commitments.
- Rewriting product features or changing what the product does.
- Routes outside the agreed scope, unless added by a written change.
What we need from you
- Read-only access to LLM traffic logs or a gateway export for the baseline week.
- A named engineer who can review and merge, about an hour a week.
- Mutual NDA before any data access.
Common questions about this engagement
Do you work under NDA?
Yes — we sign your mutual NDA before any data or repo access. For audits we prefer read-only access to start; for builds we work in a clean repo under your ownership.
Will we own what you ship?
Always. You own the code, the runbooks, the dashboards. We are explicitly set up to hand off and transition out, not to create dependency.
Vendor-agnostic — what does that mean in practice?
We integrate with what you already run — OpenAI, Anthropic, open-weight models on your cloud, your CI/CD, your observability stack. If a hosted vendor solves it, we will not reinvent it; if a self-hosted tool is the right answer, we will not pretend the hosted one is.
How is this different from a Big Four consulting deck?
We are the engineers doing the work, not analysts handing recommendations to a different team. The deliverable is working software in your repo, not a slide deck.
Can you work with our in-house AI team instead of replacing them?
Yes — most of our engagements pair with an internal owner and ramp them up to run the system after we leave. Many of our best engagements start with "we hired an AI team, help us get them productive."
Can you really cut costs without our team stopping work?
Yes, because most of the engagement happens alongside your roadmap rather than inside it. The measurement week is read-only and needs almost nothing from you beyond access. The changes that follow are small, individually reversible, and land behind evals — a routing change on one route, a cache prefix fix, a batch job moved off the interactive API. The only thing we genuinely need is someone who can review and merge, and roughly an hour a week of their attention.
What if the audit finds the spend is already efficient?
Then we tell you that and you keep the report, which is a better outcome than a build nobody needed. It happens — usually with teams who instrumented early and route deliberately already. In that case the residual value is the attribution layer and the eval suite, both of which you keep, and we would rather have a short honest engagement on the record than manufacture savings that were not there.
Guides for this work
How to reduce LLM API costs
Most production LLM spend is avoidable: it comes from sending the wrong model too many tokens, too many times. Cutting it is an engineering problem — routing, caching, batching, and context control — not a matter of waiting for prices to drop.
LLM costWhy are LLMs so expensive?
LLMs are expensive because you pay per token on every call, output usually costs more than input, and agents re-read a growing context on every step — so cost compounds far faster than the work appears to grow.
LLM evalsLLM observability: what to instrument in production
LLM observability is the practice of recording enough about every model call that you can answer, after the fact, what it cost, how long it took, whether the output was any good, and what the model was actually shown. Standard APM answers none of those, because a call that returns HTTP 200 in 900ms can still be a complete failure.
LLM costHow to reduce LLM latency
LLM latency is two separate problems — the wait for the first token and the time to stream the rest — and each has its own set of fixes.
Hiring & partnersHow to hire an LLM cost optimization consultant
Hire an LLM cost consultant when your bill is large enough to matter but your team is too busy to instrument, route, and eval their way out of it themselves — and only hire one who leaves you owning the result.
Let’s scope it on a call
Thirty minutes with an engineer. We’ll tell you straight whether this is the right first move for your team.