Token budget
Last verified: June 2026· definition
Autonomous agents fail in an expensive direction. A loop that cannot make progress will keep trying — re-reading files, re-running a failing test, re-planning — and each iteration costs money. Without a ceiling, the failure mode is a surprise invoice.
Budgets work at two levels. A hard cap (max tokens per response, max iterations per loop) is enforced by your code and the model is unaware of it. A softer task budget tells the model how much it has for the whole job so it can prioritize and wrap up gracefully rather than being cut off mid-thought.
Budgets pair with observability: the cap stops the bleeding, the trace tells you which route bled and why. Both belong in any agent harness before it runs unattended.
A budget bounds cost, it does not fix behaviour
Capping spend stops a stuck agent consuming resources indefinitely, which is necessary. It does not address why the agent was stuck, and a budget applied alone converts an unbounded problem into a silent one: the run stops without completing and without explaining itself. Budgets belong alongside informative tool errors and a way for the agent to report that it cannot proceed, not as a substitute for them.
Set it from observed successful runs
A limit chosen arbitrarily either fails to catch runaway loops or truncates legitimate long tasks, and the second failure is worse because it produces partial results that look complete. Measuring the token distribution across runs that genuinely succeeded, then setting the ceiling above the tail, means the budget only ever engages on runs that were not going to finish anyway.
Hitting the budget should be a visible event
A run terminated by its budget should be distinguishable from one that finished, both in what the caller receives and in your monitoring. If the two look the same, truncated work enters downstream systems as though it were complete. Tracking how often budgets are hit, and on which routes, also turns the budget into a diagnostic — a rising rate usually indicates a tool degrading or a task type the agent handles poorly.
Budget the task, not the call
Per-call limits are a weak control for agents, since an agent makes many calls and each can be individually reasonable while the total is not. The meaningful unit is the whole task: total tokens across every turn, tied to one trace. That is also the number worth attributing cost against, which is why budgeting and trace-level observability tend to be built together.
Common misconceptions
MythA token budget prevents runaway costs.
RealityIt bounds a single task. Many tasks each within budget still add up, so per-task limits need pairing with aggregate monitoring.
MythHitting the budget means the limit is too low.
RealityFrequently it means the agent was stuck. Raising the ceiling in response usually buys a more expensive version of the same failure.
MythPer-call limits are sufficient.
RealityAgents make many calls per task, each plausibly sized. Only a task-level budget captures what the work actually consumed.
Frequently asked questions
How do I choose a token budget for an agent?
Derive it from data rather than picking a number. Measure total tokens consumed across runs that completed successfully, look at the distribution, and set the ceiling above the tail so that legitimate long tasks are not cut off. A budget that truncates real work produces partial results that appear complete, which is considerably harder to detect than an obvious runaway loop.
What should happen when an agent hits its budget?
It should stop and report clearly that it stopped because of the limit, including what it had accomplished. That must be distinguishable from normal completion in both the response and your monitoring, so truncated work does not flow downstream as though finished. The report is also useful diagnostically — what the agent was attempting when it ran out usually indicates why it was stuck.
Should budgets be per task or per user?
Both, for different purposes. Per-task budgets stop an individual run consuming resources without limit and are the control that catches loops. Per-user or per-tenant limits stop many individually reasonable tasks aggregating into an unreasonable bill, and are what protect you from abuse or an unexpected usage pattern. Neither substitutes for the other.