How to fix: LLM responses are too slow / high latency
Last verified: June 2026· Production AI features
Where this shows up
Production AI features
How it shows up
The same root cause surfaces in a few different ways. If any of these match, you're in the right place.
- Users describe the feature as "hanging" — usually a blocking call with no streaming and no progress indicator.
- Median latency is acceptable but the p95 and p99 are far worse, so the average hides the real experience.
- Latency grew gradually without a deploy — typically prompt or retrieved-context creep.
- One route is much slower than the rest, and it is using the same model as everything else.
- Latency rises with conversation length, pointing at input size rather than the model.
Diagnose it first
Confirm which cause you actually have before changing anything — most wasted time here comes from fixing the wrong one.
| Check | What the result tells you |
|---|---|
| Separate time-to-first-token from total generation time. | High time-to-first-token points at input size, queueing, or reasoning. High total time with a fast first token is simply a long output — different problems, different fixes. |
| Measure per route rather than in aggregate. | A single global latency number averages a fast classification route with a slow generation route and hides both. Nothing actionable comes out of the aggregate. |
| Count the tokens actually being sent, including system prompt and retrieved context. | Input that has quietly grown over time is one of the most common causes of latency regression without a code change. |
| Check whether calls that could run concurrently are running in sequence. | Chained awaits on independent calls add their latencies. This is often the single largest and cheapest win available. |
| Look at p95 and p99, not the mean. | Perceived slowness is driven by the tail. A good average with a bad tail still reads as an unreliable feature to users. |
The fix
- 1Stream responses so users see output immediately instead of waiting for the full completion.
- 2Right-size the model and reasoning effort per route — frontier models and high effort cost latency you may not need.
- 3Reduce input size with retrieval and caching; smaller prompts process faster.
- 4Cache repeated prompts and prewarm caches for hot paths.
- 5Parallelize independent calls instead of chaining them sequentially.
Provider-specific behaviour
The error means roughly the same thing everywhere, but each provider signals it differently.
- Model size
- Smaller models in the same family are materially faster and are frequently sufficient for classification, extraction, routing, and formatting. Reserving the frontier model for the routes that genuinely need reasoning is usually the highest-leverage latency change available.
- Reasoning effort
- Where a model exposes an effort or thinking setting, it directly trades latency for answer quality. Setting it per route rather than globally means a simple lookup is not paying the reasoning cost that a complex analysis needs.
- Prompt caching
- Providers that cache a stable prompt prefix can substantially reduce time-to-first-token on repeated calls. It only works if the prefix is byte-identical, so anything varying near the start of the prompt — a timestamp, a session id — defeats it entirely.
- Region and networking
- Calling an endpoint in a distant region adds round-trip time to every request. For chatty multi-call flows this compounds, and it is invisible in any measurement taken from a developer machine sitting somewhere else entirely.
What doesn't work
Common fixes for this error that look right and aren't.
Adding a cache in front of everything as the first move.
Without measuring first you do not know the hit rate. Low-hit-rate caches add a lookup to every request and complexity to the system while moving the number very little.
Upgrading to a larger model to improve a slow route.
Larger models are generally slower. If the route is slow because the prompt is bloated or calls are serialised, a bigger model makes latency worse while raising cost.
Optimising the average.
Users experience the tail. Halving the mean while leaving p99 untouched produces a dashboard improvement that nobody using the product can feel.
Prevent it
Measure latency by route and set per-route model/effort budgets, the same way you manage cost.
Common variations and related errors
You'll usually hit this same root cause under a few different names. Same fix.
- "high LLM latency"
- "time to first token too high"
- "p99 latency spike"
- "streaming is slow"
- "TTFT regression"
Frequently asked questions
What causes “LLM responses are too slow / high latency”?
Latency is driven by model choice, large input/output, high reasoning effort, and lack of streaming.
How do I prevent “LLM responses are too slow / high latency” from recurring?
Measure latency by route and set per-route model/effort budgets, the same way you manage cost.
What is a reasonable latency target for an LLM feature?
It depends entirely on where the call sits. Interactive features are governed by time-to-first-token, because streaming lets a user start reading while generation continues — total time matters much less. Background and batch work should be measured on total completion time and can tolerate far more. Set the target per route from the interaction it supports rather than adopting one number site-wide.
Does streaming reduce latency?
It does not reduce total generation time, but it changes what the user experiences from a wait into a response that begins immediately. For any user-facing route that is the difference between a feature that feels broken and one that feels responsive, which is why it is usually the first change worth making.
Why did latency increase without a code change?
Almost always growth in input size. Retrieved context returning more chunks, conversation histories getting longer, or a system prompt that several people have each added a paragraph to. Track input token count per route as a metric alongside latency and this becomes visible as it happens rather than after users complain.