Tune skills and agents

Tuning skills, subagents, and their context layout

This skill is about making skills and subagents cheaper per use and more correct, and about not getting fooled by plausible-but-wrong intuitions while doing it. The guidance here is empirically grounded — the headline verdicts below came from A/B tests, and a few of them overturned the "obvious" design. When you apply this skill, you measure before you presume.

The one mental model: hot vs cold

Every piece of context lives in one of two states. Internalize this — every decision flows from it.

state	where it lives	cost
HOT	CLAUDE.md, an agent's .md body, a skills:-preloaded SKILL.md, the system prompt	in context every turn / every invocation — paid N times
COLD	references/*, a blueprint doc, any file reached by Read/Grep	paid only when read — a tool call away

The whole game is putting the right things in the right state. Hot is a standing tax; cold is pay-per-use. Optimizing is mostly moving rarely-needed detail from hot to cold, and keeping hot to {what's needed almost every time + what must never be skipped}.

Hard-won verdicts (measured, often counter-intuitive)

Lead with these. Several contradict the natural guess — that's exactly why they're worth stating.

1. A grep/anchor index over a small, well-structured doc gives ~no retrieval benefit. Markdown headers are already grep targets; an agent greps ## 7. or a keyword on its own. Adding <!-- tag --> anchors + a token index measured worse (more lines read, same tool calls) on docs in the hundreds of lines. Don't add index machinery by reflex. → references/file-indexes.md

2. An index inside HOT content is pointless. If the whole file is already in context, there is nothing to "grep to" — the agent has every line. Anchors in a hot file are dead weight.

3. The real token win is the TRIM, not the index. Moving mid-depth prose out of an always-loaded file (CLAUDE.md / agent body) into a cold reference is what actually saves tokens. The index that often accompanies it is usually ceremony. → references/hot-vs-cold.md

4. Cold references are read reluctantly. On a neutral prompt, a subagent will answer from its hot context + training and not open a relevant cold reference — even one engineered to be needed. It reads when: the fact is clearly project-specific or version-sensitive (a ground truth it knows it lacks), when it senses it can't recall a precise value, or when the prompt induces it. → references/ab-test-harness.md

5. The silent-skip failure is the dangerous one. If a critical fact lives only in a cold reference and the model's training is stale/wrong, the agent never looks and answers confidently wrong — silently. Fixing this is a rules problem, and the fix is layered and measurable. → references/subagent-verification-rules.md

6. Provenance forcing is the cheapest robust fix. Requiring an agent to state where a specific claim came from converts a silent confident-wrong answer into a visibly-flagged estimate, even when it still doesn't read the reference. It fired reliably across tests where the read-trigger only fired sometimes. → references/subagent-verification-rules.md

7. Measure, don't presume — and clean up after yourself. The A/B harness below is how every verdict above was earned. Agent memory contaminates repeat tests; clean it between runs. → references/ab-test-harness.md, references/memory-hygiene.md

When to reach for what

The user wants to…	Do this	Reference
Decide CLAUDE.md vs reference; cut a long hot file	Hot/cold triage + trim	references/hot-vs-cold.md
Add/judge a grep index, anchors, table-of-contents	Apply the index cost test (usually: don't)	references/file-indexes.md
Know if a prompt/rule/skill change actually helped	Run the A/B harness with transcript instrumentation	references/ab-test-harness.md
Fix a subagent that won't read its docs / answers stale	Add the verification + provenance rules	references/subagent-verification-rules.md
Run tests on subagents without poisoning future runs	Clean agent-memory / reflection_store / index	references/memory-hygiene.md

Core workflow

Whatever the specific ask, the shape is the same: characterize → hypothesize → A/B → keep what wins.

1. Characterize. Read the target (skill, agent .md, CLAUDE.md, the doc). For every chunk, ask: hot or cold? Used almost-every-time or occasionally? Load-bearing (must-never-skip) or optional depth? This classification is most of the analysis.

2. Hypothesize a change, and predict its effect in hot/cold terms. "Move §X to a reference" → saves hot tokens. "Add an anchor index" → predict ~no retrieval gain on a small doc (verdict 1); say so. "Add a verify rule" → predict it fires for project/version-specific facts.

3. A/B test it when the effect is non-obvious or the user wants proof. Same prompt, change only the one variable (the rule, the skill, the doc layout) → clean causal attribution. Instrument via the transcript, not self-report: count tool calls, lines read (input proxy), correctness, and provenance honesty (claimed source vs actual tool call). Full protocol in references/ab-test-harness.md.

4. Keep what wins, revert what doesn't, and say what you measured. Don't ship ceremony. If the index didn't help, drop it; if the trim saved tokens, keep it; if a rule fired only partially, report the limit honestly rather than overclaiming.

5. Clean up. If you ran subagent tests, scrub the memory they generated (references/memory-hygiene.md) so it can't contaminate later work.

Anti-patterns this skill exists to stop

• Index-by-reflex. Adding anchors/TOC/grep-tokens to every doc "for navigability." Measure first; on small docs it's cost without benefit (verdict 1).
• Hoarding hot. Letting CLAUDE.md / an agent body accrete mid-depth prose that's needed 5% of the time. That prose is a per-turn tax. Push it cold.
• Burying must-apply rules cold. A non-negotiable, divergent-from-default, or version-pinned fact placed only in a reference will be silently skipped (verdict 5). Critical → hot, or guard it with a verify/provenance rule.
• Presuming instead of measuring. "This is obviously better" is how the grep-index almost shipped as dogma. Run the A/B; let the transcript decide.
• Leaving test memory behind. Subagent runs write agent-memory + reflection_store + index entries that re-inject into later runs. Always clean.