Error audit

Error Audit

If running interactively (human present), output "Read Error Audit skill." to acknowledge. If running with --dangerously-skip-permissions (AFK/unattended), skip acknowledgement and proceed directly.

Scan every Claude Code session transcript for errors, cluster by root cause, surface the top offenders with suggested remediations.

---

When to invoke

• Periodically (every 5-10 sessions) as a health check
• When the user notices "this keeps happening"
• After a session with 3+ retry loops on the same error
• When proposing a new hook or rule (to justify it with data)

---

Usage

# Default: all sessions, top 20 clusters, human output
python3 skills/error-audit/error-audit.py

# Last 30 days only
python3 skills/error-audit/error-audit.py --since 30

# Show fewer clusters
python3 skills/error-audit/error-audit.py --top 10

# Machine-readable (for piping into other tools)
python3 skills/error-audit/error-audit.py --json

# Override the projects dir (useful for testing)
python3 skills/error-audit/error-audit.py --projects-dir /path/to/projects

# Override or disable suppressions
python3 skills/error-audit/error-audit.py --suppressions-path /path/to/suppressions.md
python3 skills/error-audit/error-audit.py --no-suppressions
python3 skills/error-audit/error-audit.py --show-suppressed

Steps (when invoking via /error-audit)

1. Run python3 <skill-path>/error-audit.py with any arguments the user provided.
2. Display the output to the user exactly as printed (includes colour-coded counts and suggested remediation tiers).
3. For the top 3 clusters, propose a concrete action:
   • Tier 1 (settings allowlist): show the exact ~/.claude/settings.json entry to add, but do NOT auto-apply — user must review Bash allowlist changes.
   • Tier 2 (hook or script fix): locate the hook/script and propose the edit.
   • Tier 3 (instruction/memory): draft the feedback or mechanism memory entry.
4. Do NOT auto-apply any remediation. This skill surfaces and proposes; the user decides.

Error Classes

Class	Signal	Example
tool_error	is_error:true on a tool result (excluding denials)	Read/Write/Edit failures
validation_error	InputValidationError in tool result	Wrong parameter types
permission_denial	"The user doesn't want to proceed"	Bash command rejected
hook_block	hook_failure attachment type	git-workflow-gate blocks
bash_fail	Non-zero exitCode on Bash attachment	Command failures
retry_storm	3+ overloaded_error records for the same tool on the same day	API overload
read_before_edit	"File has not been read yet"	Edit without prior Read

Remediation Tiers

Tier	Fix type	Example
1	Settings allowlist	Add Bash(cmd:*) to allow list (review first)
2	Hook or script fix	Tune a hook to exclude a known-good pattern
3	Instruction/memory	Add behavioral memory covering the failure mode

Configuration

Env var	Purpose	Default
CLAUDE_ERROR_AUDIT_SUPPRESSIONS	Suppressions file path	<skill-dir>/suppressions.md

Suppressions

The suppressions.md file (ships alongside error-audit.py) marks known "working-as-designed" cluster keys. Default human output hides them; --show-suppressed re-includes them; --json always includes them with a suppressed: true|false field.

Fallback Method (when Python is unavailable)

If python3 is not available, fall back to manual analysis:

Phase 1 — Gather error data

# Recent fix commits
git log --oneline --since="2 weeks ago" --grep="fix" --grep="revert" | head -30

# Find recent session files
find ~/.claude/projects/ -name "*.jsonl" -mtime -14 2>/dev/null | head -10

# HUD events
if [[ -f "$HOME/dotfiles/working/logs/events.jsonl" ]]; then
  grep -i "error\|fail\|block" "$HOME/dotfiles/working/logs/events.jsonl" | tail -30
fi

Phase 2 — Classify

Group errors into: Tooling, Convention, Environment, Architecture, External.

Phase 3 — Score

Frequency × Cost × Fixability. Minimum 3 occurrences before calling "systemic."

Phase 4 — Recommend actions

One specific action per pattern. For each high-priority pattern, recommend ONE specific action:

Action Type	When to use	Example
New hook	Repeated mechanical error that can be detected before it happens	"Add a hook that checks Node version on SessionStart"
New rule	Convention violation that the model keeps forgetting	"Add a rule that requires test file for every new module"
Architecture fix	Structural issue causing cascading failures	"Extract shared types to a package"
Documentation	Misunderstanding of how something works	"Document the auth flow in README"
Ignore	External/transient issue not worth automating	"GH API rate limits during peak hours"

---

Report Format

## Error Audit Report

**Period**: Last 2 weeks (N sessions analyzed)
**Patterns found**: M

### High Priority

| # | Pattern | Freq | Cost | Action |
|---|---------|------|------|--------|
| 1 | TypeScript path alias misconfigured | 4 sessions | ~10 min each | Rule: verify tsconfig paths on project start |
| 2 | Commit to main without PR | 3 sessions | 5 min each | Hook: ✅ already fixed (git-workflow-gate) |

### Medium Priority
...

### Resolved by existing mechanisms
- [pattern] → resolved by [hook/rule name]

### Recommendations
1. [Specific action with a one-liner justification]
2. ...

---

Integration

After producing recommendations:

1. If a new hook is recommended: Use the hooks/PROPOSAL_TEMPLATE.md to draft it
2. If a new rule is recommended: Create in rules/ following existing patterns
3. If resolved: Update hooks/COVERAGE_MATRIX.md to reflect new coverage

---

Constraints

• Only analyze data that actually exists — do NOT fabricate patterns from thin air
• If session data is unavailable or incomplete, say so explicitly
• Minimum 3 occurrences before calling something "systemic"
• Prefer conservative recommendations — one well-placed hook over five speculative ones