Context budget

Context Budget

Treat the context window as a finite budget: see where the tokens go, then cut or compress the lowest-value ones first.

When to use

• The window is near full and you need headroom to keep working.
• Runs feel slow, expensive, or the model seems to be "losing track" (context rot).
• A prompt, tool set, or set of attachments has grown bloated over time.
• The user asks what's consuming context or to optimize token usage.

When NOT to use

• The context is small and well within budget — don't optimize prematurely.
• Cutting would remove genuinely load-bearing information (then compact or scratchpad instead — relocate, don't delete).
• The real problem is a missing summary, not bloat — reach for compact.

The method (numbered, concrete — the heart)

1. Inventory the consumers. Account for each category and roughly how much it costs: system prompt, tool/function definitions, conversation history, attached/read files, retrieved docs, and prior tool outputs.
2. Rank by value density. For each chunk ask: how likely is this to be needed for the remaining work, per token it costs? Low-likelihood, high-token chunks are the first targets.
3. Cut dead weight. Remove tool definitions you won't use, files read once and finished with, stale retrieved docs, and tool outputs already acted upon.
4. Compress the rest. Replace full files with the relevant excerpt; replace verbose history with a compact summary; replace big results with their conclusion plus a pointer.
5. Right-size the toolset. Fewer, well-chosen tools beat a giant menu — every definition costs tokens and dilutes attention. Load tools on demand rather than all upfront.
6. Push bulk out of the window. Move reference material to files and pull it back on demand (see scratchpad) instead of keeping it resident.
7. Tighten the system prompt. Aim for the "right altitude" — specific enough to guide, not a bloated rulebook. Delete redundant, contradictory, or never-triggered instructions.
8. Re-measure. Confirm headroom was reclaimed and nothing load-bearing was lost, then continue.

What good looks like

• You can name the top 3 token consumers and justify each one's keep/cut/compress call.
• The biggest savings came from the highest-token, lowest-value chunks — not from nibbling.
• Nothing essential was deleted; it was relocated to a file or a summary instead.
• The toolset is minimal-sufficient; the system prompt is lean and non-redundant.

Anti-patterns

• Trimming the cheap stuff — shaving a short prompt while a 5,000-line file sits ignored in context.
• Deleting instead of relocating — losing information that's needed later instead of offloading it.
• Tool sprawl — loading dozens of tool definitions "just in case", taxing every single turn.
• Pasting whole files — when one function or section was all that mattered.
• Optimizing once and forgetting — budget is a running concern; re-check as the session grows.

Example (short, vivid)

"Context is full and it's crawling." Quick audit:

Consumer	Tokens	Verdict
3 large files read earlier	~18k	Cut — task moved on; keep paths only
22 tool definitions	~6k	Trim — only 4 used this session
Raw stack-trace dumps ×5	~4k	Compress — keep the one root cause
Conversation history	~12k	Compact — summarize to ~2k
System prompt	~1k	Keep

Result: ~40k reclaimed by attacking the few fat, low-value chunks — the small system prompt was left alone.