agentsclimarketplace

Harness audit

Skill treylom/tofu-harness/skills/harness-audit

Use when auditing an AI-agent harness (CLAUDE.md / AGENTS.md / SKILL.md / rules files) for drift — dead links, stale model names, tool mismatches, and cross-document conflicts — then repairing only independently verified findings. Deadline-bounded, parallel-census based, with built-in guards against audit overrun. Especially useful right after a new model release, when official prompting guidance changes underneath your harness.From its SKILL.md

Install
npx -y skills add treylom/tofu-harness --skill harness-audit

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

  • no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
  • 3 stars3 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

7.3 KB, as published. Nobody here has run it

/harness-audit — census, verify, repair your harness documents

Origin: a real overnight audit (2026-07-25; a single run, one harness, 252 documents) — full census → 52 suspected defects → independent re-verification → 49 confirmed → all repaired (one commit spanning 47 files + 2 sibling fixes). These aggregates carry their scope label deliberately — per-document verdicts are the reliable unit (STEP 2). The seven guards below were reverse-engineered from that night's inefficiencies (over-verification, purpose drift, no deadline), so that running this skill never costs what the first run did.

When to run this

  • A new model just shipped. Model vendors now publish model-specific prompting guidance (e.g. Anthropic's Prompting Claude Opus 5, released with the model — see the dated source in this repo's README). Every release quietly invalidates some of your harness: model names pinned in docs, thresholds benchmarked on the old generation, instructions the new model no longer needs — or newly needs. This skill is the systematic "does my harness still tell the truth?" pass.
  • Quarterly maintenance, or whenever a doc's claim visibly disagrees with reality.
  • Before publishing/porting your harness anywhere else.

Fixed deliverables (declared at start — guard ②)

Exactly three outputs. Declare them up front; wanting to produce anything else mid-run is your drift alarm.

  1. Census table — every document × 4 axes, verdict OK / FINDING / UNREADABLE
  2. Repair commit(s) — CONFIRMED findings only, applied centrally
  3. Disposition list — one line each for what a text edit can't fix (structural calls, files you don't own, deferred judgment)

The seven guards

#GuardRule
①Deadline requiredAsk for an end time before starting (refuse to start open-ended). Reaching it = report what exists. Extensions are the user's call, not yours
②Deliverable frame is fixedThe three outputs above, nothing else — "while I'm at it" is how audits eat the night
③Verification rounds capped at 2Re-verifying or re-wording the same target more than twice = escalate to the user instead
④Label fixes only when behavior changesWording/label defects in files you don't own: intervene only if an agent would act differently; otherwise record and move on
⑤Parallel by default, workers judge-only30+ documents = parallel worker census (not sequential reading). Workers never edit anything
⑥Measure capacity before fan-outCheck the machine's concurrent capacity (panes/process headroom) before spawning workers; verify leftover cleanup after
⑦Runtime checks are opt-inDocument truthfulness is the default scope. Executing skills to verify them is a separate, costlier stage — offer it as an upgrade question and run it only if the user says yes

Procedure

STEP 0 — Deadline & frame (guards ①②)

Ask for: the deadline, and the target scope (default: project + user-home CLAUDE.md / AGENTS.md / SKILL.md / rules files). Declare the three deliverables.

STEP 1 — Fix the denominator first

Build an independent file list with find before scanning — never count while scanning (if existence becomes the entry condition, absences silently fall out of the denominator). Save the list as a JSON array.

STEP 2 — Four-axis census (guards ⑤⑥)

Parallel workers (~12 docs each) judge every document on four axes:

  • tool-mismatch — do the tools/paths/servers the doc references exist in the current environment?
  • stale — are model names, versions, dates, and retired-tool references current?
  • dead-link — do relative references (reference/, scripts/, examples/) resolve?
  • conflict-dup — does it contradict another loaded document?

Worker schema: {path, verdict: OK|FINDING|UNREADABLE, axis, finding}. No minimizers in the prompt ("only report serious ones" shrinks output literally) — collect everything, filter downstream. Include a unit-definition example in the worker prompt. Without "here is what counts as one finding", extraction criteria diverge per worker and aggregate counts don't reproduce (measured: verdict agreement 86–88%, extraction agreement 26–41%). Trust per-document verdicts; never quote aggregate counts standalone.

STEP 3 — Independent re-verification + fix proposals (two-stage)

Feed FINDINGs to different workers who re-verify from scratch: {verdict: CONFIRMED|REFUTED|UNCERTAIN, evidence (one measured line), fix_type: edit|policy|none, old_string?, new_string?}. Workers stay read-only; old_string must be verbatim and uniquely matchable. Don't punish UNCERTAIN — a wrong fix is worse than no fix. (In the origin run this stage overturned 3 of 52 first-pass findings.)

STEP 4 — Central batch apply

Apply CONFIRMED+edit through one applier (replace only when the match count is exactly 1; report failures) → stage files by name → commit. policy items go to the disposition list (banners, retirement notices, owner notifications — prefer reversible moves).

STEP 5 — Report

Census table (per-group accounting + findings) + repair commit hashes + disposition list + THREATS (why not to over-trust this result: skim-level verdicts, sample verification rate, denominator scope). If the deadline arrives first, report what exists.

STEP 6 (opt-in, guard ⑦) — Runtime verification of top skills

Only with explicit user consent: pick the N most-used skills and run non-destructive smokes (--help, --dry-run, status modes, dependency version checks) into a PASS/FAIL table. Never execute destructive paths (paid APIs, outbound sends, deletions).

Safety boundaries

  • Workers never modify files — all edits are applied centrally by the main agent.
  • Files owned by someone else (another agent, another person): notify via the disposition list instead of editing.
  • In shared trees, git add by explicit filename only (never bulk-add).
  • Process cleanup only after per-process tree inspection — never kill by name pattern (the origin run nearly killed three live agent sessions that matched a worker-looking pattern).

Composes with

  • The gates in this repo (hooks/): the audit repairs what documents say; the gates enforce what agents do. A repaired rule that never fires is decoration — wire it; a firing gate built on a stale rule enforces yesterday's truth — audit it. Run both.
  • A planning pass first (Superpowers brainstorming, Ouroboros interview, or any spec tool): decide scope/deadline/exclusions before touching ~250 documents — the audit's guards assume a plan exists.
  • /doctor (Claude Code built-in): doctor checks the installation (settings parse, duplicate installs, unused extensions); harness-audit checks the content (whether your documents tell the truth). Different layers, same morning.

Keep looking

Skills are one crate of 609,856. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.