Workflow audit

workflow-audit — Mine the agent-chat corpus for optimization targets

Find 10–15 high-leverage interventions in HOW the user works with agents, ranked by (bucket_coverage × cost), output as a single PII-scrubbed markdown report.

This skill is intended to be dispatched as a sub-agent so the caller's context stays clean and the analysis is reproducible across machines.

When to use

• User asks to "audit my chat history", "find optimization targets", "what am I correcting agents on repeatedly", or similar.
• User explicitly invokes /workflow-audit.
• Periodically (e.g. monthly) as a maintenance ritual on the user's CC setup.

Arguments

$ARGUMENTS — optional, positional:

1. Output path. Default: ~/workflow-audit-<YYYYMMDD>.md.
2. Weeks of corpus to scan. Default: 12.

Prerequisites

The agent must have:

1. mache binary for the chat-corpus ingest. If missing:

   git clone https://github.com/agentic-research/mache ~/.cache/workflow-audit/mache
   cd ~/.cache/workflow-audit/mache && task install
   

2. chat-embed binary from ley-line-open. If missing:

   git clone https://github.com/agentic-research/ley-line-open ~/.cache/workflow-audit/llo
   cd ~/.cache/workflow-audit/llo && cargo install --path rs/ll-open/chat-embed
   

3. Python 3 with stdlib only (used for JSONL parsing + scoring). No third-party deps.

If any prereq install fails, bail with a clear error, do not fabricate findings.

Load-bearing rules

Three rules that MUST hold for the report to be trustworthy. If you find yourself relaxing any of them, stop and report why instead of soldiering on.

Rule 1 — Anti-temporal-bias

The user dispatches a lot of agents; recent weeks are dense and biased.

1. Bucket sessions by ISO week (YYYY-Www).
2. Exclude the 2 most recent calendar weeks from aggregation.
3. For each finding, compute bucket_coverage = weeks_appearing / weeks_scanned.
4. Rank by bucket_coverage × estimated_cost, NOT by raw count.
5. Drop any finding appearing in fewer than 3 distinct buckets.
6. If a single week contributes >40% of a finding's occurrences, either drop the finding or flag the concentration in the report.

Rule 2 — PII scrub before anything leaves the process

This must run BEFORE any tokens reach the report, the user-facing transcript, or any tool call's input. Including intermediate thinking and scratch buffers.

Build a deterministic substitution table, then apply it to ALL output. The table itself never appears in the report.

Real form	Placeholder
GitHub user names (mine + colleagues)	USER_N
Org names (agentic-research, employer, …)	ORG_A, ORG_B, …
Repo names	REPO_N
Branch names	BRANCH_N
/Users/<me>/... and /home/<me>/...	$HOME/...
Email addresses	EMAIL_N
Phone numbers	PHONE_N
Slack / Discord / Notion handles	HANDLE_N
Internal URLs, S3 buckets, DB connection strings	INTERNAL_URL_N
External-tracker ticket IDs (JIRA, Linear, etc.)	TICKET_N
Customer / project codenames	PROJECT_X

Rules:

• Deterministic. Same input → same placeholder. Patterns must stay visible across sessions.
• Pessimistic on name-shaped tokens. If it looks like a proper noun and isn't a common English word, scrub. False positives are cheap; leaked PII is expensive.
• Pass-through. Bead IDs, commit SHAs, opaque tokens (UUIDs, hashes) pass through unchanged. They're traceable but not identifying.

Rule 3 — Don't fabricate

• If the corpus is too small (<50 sessions in scope), return a stats-only report. Do not invent patterns.
• If a finding hits Rule 1's coverage threshold but the agent isn't sure what it means, surface it as a "needs human interpretation" finding.
• Every example snippet in the report must be a real (scrubbed) quote. No paraphrases.
• Contradictions between findings must be reported, not collapsed.

Step 1 — Locate the corpus

The agent-chat corpus lives at:

~/.claude/projects/<sanitized-cwd>/

Each sub-directory is one project; each *.jsonl file is one session. Skip projects matching $WORKFLOW_AUDIT_SKIP_PATHS (env var, glob list) — this is how the user excludes work projects on the work machine.

ls -la ~/.claude/projects/ | head -20

Step 2 — Ingest into a single sqlite db

Use mache to consolidate sessions into a queryable shape:

mache ingest claude-chats \
  --src ~/.claude/projects \
  --out ~/.cache/workflow-audit/chats.db \
  --since "$(date -v-${WEEKS}w +%Y-%m-%d)"   # macOS; Linux: --date "${WEEKS} weeks ago"

If --since isn't supported, ingest everything and filter at scoring time.

Step 3 — Embed for semantic clustering

chat-embed index ~/.cache/workflow-audit/chats.db
chat-embed stats ~/.cache/workflow-audit/chats.db

Embeddings drive Rule 1's pattern persistence detection. Without them you can only do exact-phrase matching, which under-counts paraphrased recurrences (the user says "no" in 30 ways; the embedding model sees them as one cluster).

If chat-embed fails or the user's machine can't run fastembed, fall back to substring matching with explicit phrase lists. Note the degradation in the report.

Step 4 — Sample with anti-temporal-bias

# Walk the corpus chronologically, bucketing by ISO week.
# Exclude the 2 most recent weeks. Cap at $WEEKS buckets total.
python3 - <<'PY'
import json, glob, os, datetime, collections
import pathlib

BUCKETS = collections.defaultdict(list)
WEEKS = int(os.environ.get("WEEKS", 12))
SKIP = os.environ.get("WORKFLOW_AUDIT_SKIP_PATHS", "").split(":")

for path in glob.glob(os.path.expanduser("~/.claude/projects/*/*.jsonl")):
    if any(s and s in path for s in SKIP):
        continue
    with open(path) as f:
        for line in f:
            try:
                evt = json.loads(line)
            except Exception:
                continue
            ts = evt.get("timestamp") or evt.get("created_at")
            if not ts: continue
            d = datetime.datetime.fromisoformat(ts.rstrip("Z"))
            iso_year, iso_week, _ = d.isocalendar()
            BUCKETS[(iso_year, iso_week)].append((path, evt))

# Drop the 2 most recent buckets.
recent = sorted(BUCKETS.keys())[-2:]
for k in recent: BUCKETS.pop(k, None)

# Cap to the last $WEEKS buckets.
kept = sorted(BUCKETS.keys())[-WEEKS:]
for k in list(BUCKETS.keys()):
    if k not in kept: BUCKETS.pop(k)

print(f"buckets={len(BUCKETS)} events={sum(len(v) for v in BUCKETS.values())}")
PY

Step 5 — Pattern detection

Eight pattern shapes the report ranks. Each is a separate pass over the bucketed corpus. For every match: record (bucket, session_id, snippet, estimated_cost).

1. Repeated user corrections. User reply ≤30 tokens following an agent turn ≥500 tokens, matching one of: no, wait, ugh, stop, ew, wrong, lying, you didn't, that's not, actually.
2. Tool-use thrash. Sessions where ≥4 tool calls precede the first Edit/Write, OR where the same file is Read more than twice without intervening edits.
3. Pre-action over-asking. Agent turn ends with a ?; next user turn ≤20 tokens contains the answer derivable from prior context (heuristic: the answer's keywords appear earlier in the same session).
4. Post-action under-asking. Tool calls matching git push, git push --force, gh pr create, rm -rf, git reset --hard, gh pr merge followed by user objection in the next 3 turns.
5. Dropped threads. Sessions ending without git commit, gh pr ..., or a final Stop/success marker AND lasting >10 turns.
6. Restated constraints. User instructions (≥3 tokens) appearing in $N sessions where N ≥ 3 AND the agent demonstrably violated the instruction at least once in a session AFTER the first one. Semantic match via chat-embed query.
7. Long-cycle recurrence. Same semantic cluster (chat-embed query) appearing in ≥4 buckets separated by ≥2 weeks. Surface as "design debt the user keeps re-touching."
8. Phrasing-shift markers. Register shifts (honest, lowkey, tbh, wait, ya feel, srsly) followed within 3 turns by a correction. Map to what the agent had just done.

Step 6 — Score and rank

cost_proxy = (tokens_wasted_in_correction_recovery
            + tokens_in_dropped_thread_dead_weight
            + 0.5 × user_turns_in_correction_burst)

bucket_coverage = unique_buckets_appearing / total_buckets_scanned

score = bucket_coverage × cost_proxy

Sort findings descending by score. Apply Rule 1 step 5 (drop coverage < 3 buckets). Apply Rule 1 step 6 (flag concentration > 40%). Take top 15.

Step 7 — Scrub

Run the PII scrub pass over every snippet, finding title, intervention text, and example. Build the mapping table on first encounter; reuse on subsequent ones. Verify NO raw forms appear in the output before writing.

# Pseudo:
def scrub(text, table):
    for real, placeholder in table.items():
        text = re.sub(re.escape(real), placeholder, text, flags=re.IGNORECASE)
    return text

Step 8 — Write the report

Output to $OUTPUT_PATH (arg 1) using this exact skeleton:

# Workflow audit — YYYY-MM-DD
# Buckets scanned: N weeks (recent 2 excluded)
# Sessions: N across M projects
# PII scrub: applied (mapping table not included)
# Corpus health: <one line>

## Top findings (ranked by bucket_coverage × cost)

### F1. <one-line title>
- **Pattern**: <2 sentences, scrubbed>
- **Bucket coverage**: x/N weeks  |  **Concentration**: passed | flagged
- **Estimated cost per occurrence**: <tokens or qualitative>
- **Example (scrubbed)**: > <real quote, scrubbed>
- **Intervention**: <one concrete line>
  - WHERE: `CLAUDE.md` | `~/.claude/settings.json` | new skill | new hook | dispatch pattern
  - WHAT (copy-paste):
    ```
    <fragment>
    ```

### F2. … (repeat through F10–F15)

## Findings dropped (low coverage; for transparency)
- F-D1. <one line>
- F-D2. …

## Corpus health stats
- Sessions / week (last 12 weeks, with the 2 recent included for context)
- Median tool calls / session
- Median user turns / session
- % sessions ending in a `git commit`
- % sessions ending in a user objection within 3 turns
- Top 5 most-edited files (across the corpus, scrubbed to REPO_N pattern)

## Ready-to-apply (3 hand-picked)
For each: WHERE, copy-paste fragment, expected effect, contradiction check
against existing `CLAUDE.md` / `~/.claude/CLAUDE.md` memory entries.

### A1. <title>
### A2. <title>
### A3. <title>

## Notes
- Contradictions surfaced (if any): <list>
- Findings flagged for human interpretation: <list>
- PII scrub: <count> unique forms mapped to placeholders. Mapping lives
  at `~/.cache/workflow-audit/scrub-table.json` and is NOT in this file.

Step 9 — Hand-off

Print the absolute path of the report. Do NOT cat the file (it may be large). Do NOT post the report to any external system — the user reviews it locally first.

If invoked by a dispatcher agent, return:

report=<absolute path>
buckets=<N>
sessions=<N>
findings_top=<count>
findings_dropped=<count>
prereq_status=<ok|degraded|failed>

Running on work machines

Two safeties:

1. Skip work projects via env var: WORKFLOW_AUDIT_SKIP_PATHS is a colon-separated glob list. Default empty; on work machines set to your work-project path patterns.
2. Scrub-table local-only: the scrub mapping never leaves the machine. If the user wants to compare audits across machines, they compare the reports (which are placeholder-only); the mappings stay home.

When the report finishes, leave a brief one-liner explaining what's in ~/.cache/workflow-audit/ so the user knows what to delete vs keep.

Why this skill exists separately from self-audit

Skill	Surface	Trigger
self-audit	PR code	Before tagging a reviewer
workflow-audit	Chat corpus	Maintenance ritual, or when the user feels friction

Don't confuse them. self-audit reviews what you wrote. workflow-audit reviews how you wrote it.