Prompt caching
Last verified: June 2026· definition
Prompt caching lets a provider store the processed form of a stable prompt prefix — a large system prompt, tool definitions, or shared context — so repeated requests pay a steep discount on those tokens instead of full price.
It’s a prefix match: any change inside the cached region invalidates everything after it. The practical rule is to keep stable content at the front and put volatile content (timestamps, per-request data) at the end.
For system-prompt-heavy or multi-turn workloads, correct caching is one of the easiest large cost reductions available — often a major part of a cost-optimization engagement.
The prefix must be byte-identical
Caching works by recognising that the beginning of a prompt matches something already processed. Any difference breaks the match from that point on — a timestamp, a session identifier, a user name, even a changed space. This is the single most common reason teams enable caching and see no benefit: something variable sits near the start of the prompt, so the cacheable prefix ends immediately. Ordering the prompt so all stable content precedes anything that varies is the whole technique.
It changes how you structure prompts
Once caching is in play, prompt layout becomes an economic decision rather than purely a clarity one. Large stable material — system instructions, few-shot examples, reference documents, tool definitions — belongs at the front where it can be cached across requests. Per-request content belongs at the end. This sometimes conflicts with what reads most naturally, and it interacts with context engineering, since instructions placed far from the point of use compete with everything between.
Where it pays and where it does not
The benefit scales with how much stable content each request carries and how often requests repeat that content. A chat feature sending a long system prompt on every turn benefits substantially. A system where each request is largely unique has little to cache and will see negligible return. Cached content also typically expires after a period of inactivity, so low-traffic routes may find the cache cold more often than warm.
Cache lifetime shapes the benefit
Cached prefixes typically expire after a period without use, which means the benefit depends on request frequency as well as prompt structure. A high-traffic route keeps its cache continuously warm; a route called a few times an hour may find it cold on most requests and see little return. Where a route matters and traffic is sparse, some teams keep the cache warm deliberately, though whether that is worth the extra calls depends entirely on the size of the prefix being preserved.
Common misconceptions
MythPrompt caching caches the response.
RealityIt caches the processing of the input prefix, not the output. The model still generates a fresh response each time — what is saved is the work of reading the repeated preamble.
MythEnabling caching is enough to see savings.
RealityThe prompt must be structured so a substantial stable prefix exists. Anything variable near the start truncates the cacheable region to almost nothing, which is why enabling it often appears to do nothing.
MythCaching helps every workload.
RealityIt helps workloads with large repeated preambles. Where requests share little stable content, or arrive too infrequently to keep a cache warm, the return is minimal.
Frequently asked questions
Why is prompt caching not reducing our costs?
Almost always because something variable appears near the start of the prompt, ending the cacheable prefix before it contains anything substantial. Common culprits are timestamps, request identifiers, user names, and dynamically assembled instructions. Print the exact prompt for two consecutive requests and compare them character by character — the point where they first differ is where your cache stops, and moving that content later usually fixes it.
What should go in the cached portion of a prompt?
Everything large and stable: system instructions, tool and function definitions, few-shot examples, and any reference material shared across requests. Then the variable content — the user message, retrieved context specific to this query, session details. The larger the stable prefix and the more requests that share it, the more caching returns.
Does prompt caching affect output quality?
No. It changes how the input is processed, not what the model produces. The model sees the same content either way, so responses are unaffected. This makes it one of the few cost optimisations with no quality trade to evaluate, which is why it is usually worth doing before optimisations that do carry one.
Go deeper
How to reduce LLM latency
LLM latency is two separate problems — the wait for the first token and the time to stream the rest — and each has its own set of fixes.
Hiring & partnersHow to hire an LLM cost optimization consultant
Hire an LLM cost consultant when your bill is large enough to matter but your team is too busy to instrument, route, and eval their way out of it themselves — and only hire one who leaves you owning the result.
LLM evalsLLM observability: what to instrument in production
LLM observability is the practice of recording enough about every model call that you can answer, after the fact, what it cost, how long it took, whether the output was any good, and what the model was actually shown. Standard APM answers none of those, because a call that returns HTTP 200 in 900ms can still be a complete failure.
Agentic codingContext engineering for coding agents
Context engineering is the work of deciding what a model sees on each turn: what goes in, in what order, and — the part teams skip — what gets left out. On coding agents it is usually the difference between an agent that lands a clean diff and one that thrashes, because almost every agent failure that looks like a reasoning failure is really a context failure.