Skill counterfactual audit

Skill Counterfactual Audit

Why

When you ship a skill, pass-rate is the easy metric — and it's blind to almost everything that matters. CTA (arxiv 2605.11946v1) shows a single skill can produce 522 measurable behavioural changes across 49 tasks while pass-rate moves only +0.3%. The interesting effects (surface anchoring on the skill's vocabulary, template copy without adaptation, excess planning loops, recovery from off-task drift, generation of off-task artifacts) live entirely below pass-rate.

The remedy is a paired-trace audit: run the same probe task twice, with and without the skill, and compare the behaviours directly rather than the headline metric.

How

For each skill under audit:

1. Curate a probe-task bank (3-5 small tasks per skill domain). Each task should have:
   • A clear goal (a single instruction)
   • An expected category of behaviour
   • A "phase decomposition" (rough segmentation rules)
2. Run paired — same prompt, same task, run A with skill loaded, run B without.
3. Segment each trace — phases like {plan, retrieve, edit, verify, fix-loop, summary}.
4. Align phases — match B phases to A phases; mark new/dropped/displaced phases.
5. Classify SIP (Skill Influence Pattern) — for each aligned phase, label the effect:
   • surface-anchoring — A copies vocabulary/structure from the skill itself
   • template-copy — A reproduces a skill template literally without adaptation
   • excess-planning — A adds extra planning steps absent in B
   • task-recovery — A recovers from drift that B falls into
   • off-task-artifact — A produces artifacts unrelated to the goal
6. Emit report to .planning/patterns/skill-audits/<skill>-<timestamp>.md with phase comparison table + SIP distribution + retire/refine/keep recommendation.

Recommendation rule of thumb

Dominant SIP	Verdict
task-recovery + correct artifacts	keep (genuine value)
surface-anchoring with no task-recovery	refine (skill is leaking style, not capability)
template-copy on most tasks	refine (lower template specificity)
off-task-artifact on >25% of probes	retire (skill is causing distraction)
excess-planning with no completion delta	refine (skill adds ceremony)

When to skip

• The skill is a workflow guide that doesn't directly shape output (e.g., loop, schedule) — audit dominant SIPs would always be "task-recovery" trivially.
• No probe-task bank exists for the domain (build one first; defer audit).
• The skill was just created and has no historical baseline. Run the audit when the skill has been active for ≥3 sessions.

Integration

• done-retirement — required gate before retiring a skill.
• gsd-cleanup — sweeps audits on a milestone-close cadence.
• Companion to simplify, code-review — those audit code; this audits skill behaviour.

Cross-references

• Rosetta concept #9 (Execution-Grounded Selection) — paired-trace audit is the skill-domain instantiation
• College: agent-systems / integration-evaluation / agent-paired-trace-audit
• Related skills: skill-integration (manages the skill lifecycle that this audits)