Token optimization

Token Optimization

The method behind the "Token Optimizer" role and the /optimize-tokens and /evolve commands. It cuts the tokens the AI spends — always-loaded context, repeated instructions, over-reading, bloated reports — while preserving every quality safeguard. This is a compression-and-cleanup discipline, not a feature-removal one.

When to use

• The OS or a project's AI config feels heavy: long prompts, duplicated rules, agents that repeat each other, reports that restate the diff.
• During /evolve (self-improvement) or a periodic cleanup.
• After several real tasks reveal the same context being re-loaded or the same guidance re-stated.

When NOT to use

• To make runtime application code faster or cheaper (that is the performance skill / performance-engineer).
• As an excuse to delete quality gates, safety rules (§8), verification steps, or the exclusion clauses that keep agent routing accurate. If a cut would reduce coverage, reliability, or decision accuracy, do not make it.

What it targets (in priority order)

1. Always-loaded overhead — the highest-leverage tokens, paid every session: CLAUDE.md, agent description: fields (all injected into every prompt), skill description: fields. Trim these first.
2. Duplicated instructions — the same rule restated across many files instead of stated once and referenced.
3. Bloated bodies — motivational/generic prose that carries no execution value; examples beyond the one that teaches.
4. Overlap — two agents/commands/skills/docs that produce the same output; merge or cross-exclude.
5. Unused files — templates/checklists/matrices/commands nothing references (orphans).
6. Reading waste — patterns that read whole files/dirs where a map + targeted grep would do (fix the rules, e.g. wire map-first into the offending command).
7. Report waste — final reports that restate the diff or exceed the task's budget.

Steps

1. Measure first. Quantify the always-loaded payload and find the heaviest contributors:
   • Sum CLAUDE.md chars; sum every agent/skill description:; list the longest.
   • Grep for repeated rule phrases (e.g. a safety line) to count restatements.
   • Find orphans: for each template/checklist/matrix, grep the tree for a reference; zero refs = orphan candidate. Record the baseline numbers — you will report the delta.
2. Rank by leverage = (tokens saved) × (how often loaded). Always-loaded beats per-invocation beats one-off. A 200-char cut to an agent description (every session) outranks a 2,000-char cut to a rarely-read doc.
3. Compress, don't amputate. For each target: keep the operational content (what it does, when to use, the distinctive keywords, the "not for X" exclusions, the steps, the safeguards); cut adjectives, repetition, and redundant elaboration. For descriptions, keep trigger + exclusion; drop everything else.
4. Dedupe by reference. When a rule appears in N places, state it once in the canonical file (CLAUDE.md / CONTEXT_DISCIPLINE / a skill) and replace the copies with a one-line pointer.
5. Merge or cross-exclude overlaps. Two things with the same job → merge. Two things with adjacent jobs → add a one-line "not for X (use Y)" boundary to each so neither is loaded by mistake.
6. Retire orphans — only after proving nothing references them (grep the whole tree, including presets, matrices, docs, adapters). If unsure, mark for review rather than delete.
7. Fix reading rules at the source — if a command/agent over-reads, edit it to read the map first and grep before full reads; don't just note it.
8. Verify nothing broke — run the OS's own gates (integrity-check.py, validate.py --strict) and regenerate adapters. Every quality gate, safety rule, and routing exclusion must survive.

What to avoid

• Padding files back up to hit a word floor — if a file is genuinely thin, that's a content problem, not a reason to bloat it.
• Cutting "not for X (use Y)" exclusions to save characters — that trades decision accuracy for pennies.
• Deleting a file you can't prove is unreferenced.
• Silent scope cuts (dropping a checklist item, a gate, a verification) in the name of tokens.

Expected output

A compact report: baseline vs. after (always-loaded token delta, files touched), what was compressed/merged/deleted/rewired, and an explicit line confirming which quality safeguards were preserved (gates, §8, routing exclusions, verification). Numbers first, prose minimal.