Context window
Last verified: June 2026· definition
The context window bounds how much the model can “see” at once — the system prompt, conversation history, retrieved documents, and the response all draw from it. Exceeding it forces truncation, summarization, or compaction.
Larger context windows (now up to ~1M tokens on frontier models) enable whole-codebase reasoning and long documents, but using more context costs more tokens and can dilute focus, so it’s not always better.
Managing the context window well — via retrieval, compaction, and caching — is central to building cost-effective, reliable LLM systems.
Everything shares the same budget
The window is not just your prompt. System instructions, conversation history, retrieved documents, tool definitions, tool results, and the model’s own response all draw from the same allowance. This is why a request that worked in testing can fail in production: the prompt is identical, but real conversation history and real retrieved context consume space that the test never used. Budgeting by component rather than by prompt length is what makes this predictable.
Filling the window is rarely the right goal
Larger windows removed a hard constraint and introduced a softer one. Cost scales with tokens, latency scales with tokens, and relevant information competes with everything else present. A carefully retrieved ten thousand tokens usually outperforms a million-token dump of everything remotely related, and costs a fraction as much. The window is a ceiling, not a target.
What happens when you exceed it
Behaviour varies and matters. Some systems return an explicit error, which is the good case because it is visible. Others silently truncate, which is considerably worse: the request succeeds, the answer looks fine, and content the model needed was quietly dropped. Knowing which behaviour your stack has, and monitoring input token counts before you approach the limit, is the difference between a caught error and a silent quality regression.
Reasoning tokens consume it too
On models that reason before answering, the intermediate reasoning draws from the same allowance and bills at output rates. A request can therefore be comfortably within the window on visible input and still run out of room mid-answer, because the reasoning consumed more than expected. Where a model exposes an effort setting, raising it increases this consumption, which is another reason to set effort per route rather than globally.
Budget by component, not by total
The practical technique is allocating the window before you fill it: a ceiling for system instructions, a ceiling for retrieved context, a reserve for conversation history, and a guaranteed floor for the response. Enforcing those ceilings means growth in one component cannot silently starve another. Without allocation, retrieved context tends to expand until it crowds out the response, which surfaces as answers that stop abruptly rather than as an obvious error.
Common misconceptions
MythA bigger context window means better answers.
RealityIt means more can fit. Quality depends on what you put in it, and irrelevant material actively competes with relevant material for the model’s attention.
MythThe context window is the input limit.
RealityIt covers input and output together. A long prompt reduces the room available for the response, which is a common cause of answers that stop unexpectedly early.
MythStaying under the limit means you are fine.
RealityCost and latency both scale with usage well before you hit the ceiling, and dilution degrades quality gradually rather than at a threshold. The limit is the last constraint you encounter, not the first.
Frequently asked questions
What counts toward the context window?
Everything in the request and the response: system prompt, full conversation history, retrieved documents, tool definitions, tool results, and the generated output. Teams often account only for the user message and are surprised when a request fails, because in a mature system the accumulated history and retrieved context are usually much larger than anything the user typed.
What happens if I exceed the context window?
It depends on the provider and framework. Some reject the request with an explicit error, which is preferable because the failure is visible. Others truncate silently, dropping content to make it fit — the call succeeds and the model answers without information it needed. Since silent truncation is invisible at the API level, tracking input token counts per route is how you catch it.
Should I use a model with a larger context window?
Only if you genuinely need to reason over more material at once than a smaller window allows. If the requirement is finding relevant information in a large corpus, retrieval solves that more cheaply and usually more accurately than loading everything. The large window is valuable for tasks needing whole-document or whole-codebase reasoning, where the relevant context genuinely is the entire thing.
Go deeper
Why are LLMs so expensive?
LLMs are expensive because you pay per token on every call, output usually costs more than input, and agents re-read a growing context on every step — so cost compounds far faster than the work appears to grow.
Agentic codingContext engineering for coding agents
Context engineering is the work of deciding what a model sees on each turn: what goes in, in what order, and — the part teams skip — what gets left out. On coding agents it is usually the difference between an agent that lands a clean diff and one that thrashes, because almost every agent failure that looks like a reasoning failure is really a context failure.