Prompt compression
Last verified: June 2026· definition
Because input tokens are billed on every single call, a bloated system prompt is a tax you pay forever. Teams accumulate instructions defensively — a rule added for one edge case in March, three redundant restatements of the same constraint, a dozen few-shot examples where two would do.
The practical moves: consolidate overlapping instructions, drop examples that no longer earn their place, inject few-shot examples conditionally rather than always, retrieve fewer and better chunks instead of stuffing context, and ask for structured output so the model stops narrating.
Compression pairs with caching rather than competing with it: cache the stable prefix you cannot remove, and delete the parts you never needed. Both belong behind an eval, since a prompt is a program and deleting lines can change behavior.
Start by measuring what is actually sent
Most compression opportunities are found by printing the resolved prompt for a real request and reading it. What appears is usually familiar: a system prompt several people have each added a paragraph to, examples that duplicate one another, retrieved chunks the model never references, and instructions repeated in two places. None of this is visible from the application code, which is why compression work almost always begins with inspection rather than with technique.
Retrieved context is usually the largest term
In retrieval-based systems the retrieved passages typically dwarf the instructions, and the number of chunks retrieved is often set once and never revisited. Retrieving fewer, better-ranked chunks — by adding a reranker rather than simply lowering the count — usually reduces tokens while improving answers, because relevant material stops competing with marginal material. This is the rare optimisation that cuts cost and raises quality together.
It interacts with caching
Compression and prompt caching pull in different directions if applied carelessly. A large stable preamble is expensive per request but cacheable, so trimming it may save less than expected while restructuring it can save a great deal. The productive order is to arrange the prompt so stable content sits at the front and cacheable, then compress the variable portion, rather than shortening everything uniformly.
Verify with evals, not by inspection
Removing instructions that look redundant occasionally removes the one holding a behaviour in place, and the resulting regression is subtle rather than obvious. Because compression changes what the model sees on every request, each removal deserves checking against an eval set. Treating it as an obviously safe cleanup is how teams end up with a cheaper system that quietly performs worse.
Common misconceptions
MythCompression means summarising the prompt with a model.
RealityThat is one technique and rarely the first. Most gains come from removing duplication, trimming unused retrieved context, and cutting instructions that no longer do anything.
MythShorter prompts always perform worse.
RealityFrequently they perform better, because relevant material stops competing with filler. Dilution is a real effect, and reducing it is part of why compression sometimes improves quality.
MythIt is a safe change that does not need testing.
RealityRemoving an instruction that quietly held a behaviour in place produces a subtle regression. Every removal is a change to what the model sees on every request.
Frequently asked questions
Where do I find the easiest token savings?
Print the fully resolved prompt for a real production request and read it. The recurring findings are a system prompt that has accumulated overlapping instructions, few-shot examples demonstrating the same pattern more than once, and retrieved context far larger than the answer needed. Retrieved context is usually the largest single component in a RAG system and the least often revisited.
Does compressing prompts hurt quality?
It can, and it can also help. Removing genuine filler tends to improve results because relevant content stops competing for attention. Removing an instruction that was doing real work causes a regression that is easy to miss because the output still looks reasonable. The distinction is not reliably visible by reading, which is why each change should be validated against an eval set rather than assumed safe.
Should I compress or cache?
Usually both, in the right order. Restructure first so the large stable content sits at the front where caching applies, then compress what remains variable. Trimming a preamble that was already being cached saves less than it appears to, while shortening the uncached variable portion saves on every request. Getting the order wrong means doing work that the cache was already handling.