How to reduce LLM latency

Where LLM latency actually comes from

Before you can reduce latency you have to know which latency you are reducing. Total response time is made of two distinct phases. The first is time-to-first-token (TTFT): the interval between sending your request and receiving the first token back. TTFT is dominated by how much input the model has to process — the full prompt, system instructions, retrieved context, and conversation history — plus any time the request spends queued behind other traffic before it starts running. A long prompt means a long wait before anything comes back, regardless of how short the answer is.

The second phase is generation time, sometimes measured as inter-token latency: how quickly the model emits each subsequent token once it has started. Because tokens are produced sequentially, this phase scales with the length of the output and with the size of the model — a larger model does more computation per token. A request that writes a long response will spend most of its wall-clock time here, in generation, not in TTFT.

This split is the single most important mental model in the guide. If your users complain about the spinner before anything appears, you have a TTFT problem and should look at prompt size, context, and queueing. If the answer starts quickly but takes a long time to finish, you have a generation problem and should look at output length and model size. Measure both separately in production — an averaged "total latency" number hides which half is hurting you and sends you optimising blind.

Streaming and perceived latency

There are two kinds of latency and only one of them is measured by a clock. Real latency is how long the full response takes to complete. Perceived latency is how long the user feels they waited — and for interactive features, perceived latency is what determines whether the product feels fast. Streaming is the technique that pulls these two apart.

When you stream, you display each token as it arrives rather than buffering the entire response and rendering it at the end. The user sees text begin to appear the moment the first token is generated, so their felt wait collapses down to roughly TTFT even though total generation time is unchanged. Nothing about the model got faster; you simply stopped hiding the output behind a spinner. For any feature where a human is reading the result — chat, drafting, explanations — streaming is the highest-leverage change you can make and should almost always be the first thing you ship.

Streaming has limits worth naming. It does nothing for perceived latency when the consumer is a machine that needs the whole response before it can act — for example a call whose output is parsed as JSON and fed into the next step. It also does not reduce real latency or cost. Treat streaming as the fix for the human-facing perception problem, and treat the techniques in the following sections as the fixes for the underlying clock time.

Shrinking the prompt, the context, and the output

Once streaming is in place, the biggest real-latency wins come from sending less in and asking for less out. On the input side, trim the prompt and context. Every token of system prompt, few-shot example, retrieved document, and chat history has to be processed before the first output token appears, so bloated prompts directly inflate TTFT. Audit what you actually send: drop redundant instructions, retrieve fewer and more relevant chunks instead of stuffing the context window, and summarise or truncate long conversation histories rather than replaying them verbatim.

On the output side, cap the output length. Because generation time scales with the number of tokens produced, an unbounded response is an unbounded wait. Set a sensible maximum-tokens limit, and shape the prompt to ask for concise answers — request a short summary rather than an exhaustive essay when the use case allows. If you only need a label, a number, or a structured field, instruct the model to return exactly that and nothing else.

These changes compound with everything downstream. A shorter prompt is cheaper as well as faster, and it makes prompt caching (below) more effective because there is less to cache and less variance. Reducing input and output tokens is the most portable latency lever you have — it works on any model, from any provider, without new infrastructure.

Model choice, caching, and speculative decoding

The model you pick sets the floor on generation speed. Larger models do more work per token, so using a smaller or faster model — or a distilled model — where quality allows is often the largest single latency win available. Not every request needs your most capable model; classification, extraction, routing, and simple rewrites frequently run well on a lighter model. Test the smaller model on your real workload and keep the big one only for the requests that genuinely need it.

Prompt caching attacks TTFT directly. Many applications send a large, stable prefix on every request — a long system prompt, a fixed set of instructions, a tool schema, or a shared document. Caching lets the provider skip re-processing that unchanged prefix on subsequent calls, cutting the time before the first token appears. To benefit, structure your prompts so the stable content sits at the front and the variable, request-specific content comes last; that keeps the cacheable prefix as long and as reusable as possible.

Finally, speculative decoding is a generation-time technique some providers and serving stacks apply: a small, fast draft model proposes several tokens ahead and the main model verifies them in a single step, so more tokens can be confirmed per pass. You typically enable it as a serving feature rather than implement it yourself, but it is worth knowing the term when you evaluate providers or self-hosted inference — it is one of the main levers for reducing inter-token latency without changing the answer.

Architectural moves: parallelize, route, and go async

Beyond a single call, the shape of your system determines how much latency the user actually experiences. First, parallelize independent calls. If a feature makes several model calls that do not depend on each other — scoring three candidates, extracting fields from separate documents, running a fan-out of sub-questions — issue them concurrently rather than in sequence. The user waits for the slowest call instead of the sum of all of them.

Second, route latency-sensitive requests to faster models. Not all traffic has the same urgency. A model-routing layer can send interactive, user-facing requests to a fast model while leaving heavier or offline work on a more capable one. Routing lets you tune the latency-versus-quality trade-off per request type instead of paying the same cost everywhere.

Third, move non-urgent work off the hot path. Anything that does not have to finish before the user gets their answer should not block the response. Enrichment, logging, summarisation for later, secondary analysis — push these to background jobs, batch them, or run them asynchronously. The request path should contain only the work the user is actually waiting on. Combined with streaming, a tight prompt, and the right model, this is how a feature goes from sluggish to responsive without sacrificing what it can do.