Failure archaeology

Failure Archaeology

Purpose

Git history is the only honest record of what actually broke; commit messages and diffs encode incidents nobody wrote up. This skill mines that record — reverts, fix-chains, hardening arcs, pinning incidents — and converts each finding into a guardrail. The goal is never "assign blame"; it is "make the past failure impossible to repeat silently."

When to Use / When NOT to Use

Use when:

• Code looks wrong or over-specific (an odd pin, a strange guard, a deleted feature) and you need the why before changing it.
• Asked to extract lessons, write a postmortem, or justify a guardrail.
• Before removing any constraint (version pin, size limit, blocklist) — the constraint may be a scar.
• Onboarding surfaced a suspicious history arc worth excavating.

Do NOT use when:

• You need a first map of an unfamiliar repo → load codebase-onboarding (do it first; archaeology needs the map).
• You are debugging a live failure, not a historical one → load debugging-playbook.
• You are deciding dependency policy going forward → load dependency-management (archaeology supplies its evidence).
• You are about to make the change itself → load change-control.

Core Methodology

1. Size the dig. git log --oneline | wc -l. Under ~50 commits: read every subject line. Over that: use keyword scans (step 2) and file-scoped logs.
2. Scan for signal keywords. Search subjects for revert, fix, security, pin, bump, harden, remove, back out. Each hit is a candidate incident.
3. Classify the pattern. Assign each candidate to one of four incident shapes:
   • Revert: a commit undone. Lesson lives in why the original seemed fine.
   • Fix-after-fix chain: 2+ consecutive fixes to the same area. First fix treated a symptom.
   • Hardening arc: security/robustness commits clustered in time. Reveals the threat model as enacted, not as documented.
   • Dependency-pinning incident: a pin or downgrade commit. Something upstream broke; the pin is the scar.
4. Reconstruct the incident. For each: git show <hash> and read the full diff, not just the message. Answer: What was the failure? What was the trigger? Why didn't existing defenses catch it? Quote exact lines. If the diff doesn't say why, the sequence of commits often does — read neighbors with git log --oneline around the hash.
5. Extract the lesson at the right altitude. "chromadb broke" is too specific; "never trust anything" is too general. The right altitude is a rule you can check mechanically: "when a dependency does not declare its own transitive constraints, pin them yourself."
6. Install a guardrail. Every lesson becomes exactly one artifact: a test, a version pin, an in-code comment at the scar site, a checklist line, or a CI check. A lesson with no artifact will be relearned the hard way.
7. Record provenance. Note hash, date, and repo next to the guardrail so the next archaeologist can re-verify instead of re-excavating.

Decision tree: classifying a suspicious commit

Commit message contains "revert" / "back out"?
├─ yes → Revert incident: diff the original commit too; the lesson is in the
│        gap between "looked fine" and "was wrong"
└─ no → Same files fixed in 2+ nearby commits?
    ├─ yes → Fix-after-fix chain: the FIRST fix's diff shows the symptom-level
    │        thinking; the LAST fix shows the root cause
    └─ no → Message pins/downgrades/constrains a dependency?
        ├─ yes → Pinning incident: identify what upstream change broke what
        │        runtime behavior; check if the pin has an expiry plan
        └─ no → Message says security/harden/validate?
            ├─ yes → Hardening arc: list every defense added in the commit;
            │        each one names a feared attack
            └─ no → Probably routine; log it and move on

Reconstruction checklist (per incident)

• Hash, date, and one-line summary recorded.
• Full diff read (git show <hash>), not just the message.
• Failure trigger identified or explicitly marked "(hypothesis — requires verification)".
• Why prior defenses missed it: answered.
• Lesson stated as a mechanically checkable rule.
• Guardrail artifact named (test / pin / comment / checklist / CI) and located.
• Checked whether the same class of failure exists elsewhere in the repo right now.

Discovery Commands

All repo-agnostic; run from the repo root.

# Size the dig
git log --oneline | wc -l
git log --oneline            # small repos: read it all

# Keyword scans (run each; -i = case-insensitive)
git log --oneline -i --grep="revert"
git log --oneline -i --grep="fix"
git log --oneline -i --grep="security"
git log --oneline -i --grep="pin"
git log --oneline -i --grep="remove"

# Fix-after-fix chains: history of one file or directory
git log --oneline -- path/to/file
git log --follow --oneline -- path/to/file   # survives renames

# Reconstruct: full diff of one commit
git show <hash>
git show --stat <hash>                        # files touched, quick triage

# Who/when context around a scar line
git log -p -S "suspicious_string" -- .        # commits that added/removed a string
git log --format="%h %ad %s" --date=short | head -20

# Dependency-pin scars in build files
git log --oneline -- pyproject.toml package.json Cargo.toml requirements.txt
git log -p -- pyproject.toml | grep -n "^\+.*[<>=]" | head -30

# Reverts recorded by git itself
git log --oneline --grep="Revert"             # git's auto-generated revert subjects

Ecosystem variants: for lockfile incidents inspect git log --oneline -- package-lock.json pnpm-lock.yaml poetry.lock Cargo.lock; for monorepos scope with git log --oneline -- packages/<name>.

Failure Modes & Anti-patterns

Symptom	Mistake	Correction
"History is clean, nothing to learn"	Reading only commit subjects	Read diffs; the lesson is in the code delta, not the message
Removed a pin, runtime broke	Treating an old constraint as cruft	Every pin/limit/blocklist is presumed a scar until archaeology clears it
Confident but wrong incident story	Inventing causality from a message alone	Quote diff lines; mark unproven causes "(hypothesis — requires verification)"
Same bug returns in 6 months	Lesson written in a doc nobody reads	Lesson must become an artifact: test, pin, in-code comment, or checklist
Lesson too vague to act on	Extracting at the wrong altitude ("be careful with deps")	Restate as a mechanically checkable rule
Excavation never ends	Digging the whole history of a large repo	Scope by keyword and by files the current task touches
Blaming the author	Treating incidents as personal failures	Incidents are system information; extract the rule, drop the name

Repository Examples

RAGIT — dependency-pinning incident (~/ragit, as of 2026-07-04). History: 4eed67ea initial → 3f88ce8d remove generated metadata → 57f95afd packaging keywords → fc3f8a43 switch to pypdf → a1ad63ed "Pin chromadb to 0.4.x" → 0a06ae9b "Pin numpy for chromadb 0.4 and add demo GIF". The two pin commits in sequence are the fingerprint of a real incident: chromadb 0.4 does not declare a numpy<2 constraint of its own, so numpy 2.0 installed cleanly and broke chromadb at runtime. The fix was to pin the transitive dependency yourself (numpy<2.0 in pyproject.toml alongside chromadb>=0.4.0,<0.5.0). Extracted rule: when an upstream package omits a constraint it actually needs, you own that constraint. Guardrail artifact: the pins themselves. Note also fc3f8a43 "switch to pypdf": a dependency replaced, not patched — minimal-dependency doctrine in action.

PRISM — hardening arc (~/prism, as of 2026-07-04). 51b92c9 "security: add payload size limit, concurrency cap, repo name validation, Ollama timeout" followed immediately by b011239 "tests: add security path coverage for signature, repo validation, size limit". A single commit enumerates the feared attacks (oversized payloads, resource exhaustion, injected repo names, hung backend), and the very next commit pins each defense with a test — the complete lesson→guardrail loop inside two commits. The enacted threat model matches the code: HMAC-SHA256 webhook verification with hmac.compare_digest runs BEFORE JSON parsing, and the payload-size check runs before json.loads. Verified against live history 2026-07-04 (git -C ~/prism log --oneline).

RUFLO — process-slip incident (~/ruflo, as of 2026-07-04). Commit "fix: @claude-flow/browser peer dep, dist-tags, bump to alpha.3" records a real release slip in a pnpm monorepo that publishes three synchronized packages (@claude-flow/cli, claude-flow, ruflo) at identical versions: post-publish dist-tags must be set on ALL aliases, and internal docs explicitly mark the ruflo dist-tag "EASY TO FORGET". The guardrail here is a documented checklist line — an example of a lesson artifact that is prose because the failure is a human release step, not code.

AGENTIX — hardening with rationale in-code (~/agentix, as of 2026-07-04). Last commit: "Security review: harden embedding load + document threat model". The diff's key scar: np.load(..., allow_pickle=False) with the in-code comment "a tampered DB blob must not be able to execute code on load" — the incident rationale stored at the scar site, which is the strongest guardrail placement. Same commit family documents the README threat model ("the security boundary is you and the model you point it at, not the code"), showing security effort proportional to a written threat model rather than generic maximalism.

ORPHY / ASVER — small scars, marked honestly (as of 2026-07-04). Orphy commit "fix(frontend): replace Tailwind centering with reliable inline styles" — the causal reading, "when a framework utility fails under edge cases, drop to CSS primitives," is (hypothesis — requires verification). Asver commit "Remove server build from package.json for static deployment" — lesson (partly inferred): declare deployment constraints early and strip incompatible build steps rather than carrying them.

Validation Criteria

You applied this skill correctly if:

• Every incident you report cites a hash and quotes the diff or exact message, or is explicitly marked "(hypothesis — requires verification)".
• Each lesson is stated as a mechanically checkable rule, not a platitude.
• Each lesson has exactly one named guardrail artifact and its location.
• You checked whether the failure class recurs elsewhere in the current code, not just in history.
• No existing pin, limit, or blocklist was removed without an excavation clearing it first.

Provenance & Maintenance

• Sources: prism, agentix, ragit, orphy, asver, ruflo at ~/<repo>, investigated 2026-07-04. Prism history re-verified live via git -C ~/prism log --oneline on 2026-07-04; ragit/ruflo/agentix/orphy/asver facts from the same-day investigation fact pack.
• Doctrine encoded: local-first, minimal dependencies, security proportional to a written threat model. In such repos, sparse histories are dense — nearly every commit is a decision, which makes archaeology cheap and high-yield.
• Assumptions: commit messages in these repos are honest and descriptive (they are, as of 2026-07-04); repos remain at the listed paths. The orphy causal reading and part of the asver lesson are inferred, and are labeled as such above.
• Re-verification commands: git -C ~/ragit log --oneline; git -C ~/ragit show a1ad63ed; git -C ~/prism show 51b92c9; git -C ~/agentix log --oneline | head -3; git -C ~/ruflo log --oneline -i --grep="dist-tag".
• Likely to drift: "last commit" claims (agentix) drift with any new commit; ragit may eventually unpin chromadb/numpy when upstream fixes its constraints; ruflo release process may gain automation that retires the checklist lesson.
• Maintenance checklist: quarterly — re-run re-verification commands; re-stamp examples; check whether any documented pin has been lifted (if so, record the lifting commit as the incident's closure); confirm cross-referenced skills (codebase-onboarding, debugging-playbook, dependency-management, change-control) still exist under ~/.claude/skills/.