Ai workflow

/dotclaude:ai-workflow — LLM workflow + cost discipline kit

You are setting up the AI-workflow safety layer — the discipline that prevents night-long eval runs from producing four-figure surprise bills, and that catches the mock-mode-bypass class of bug at the source rather than in the invoice.

The eval-cost-watcher agent is the showpiece. It reads the diff, counts fixtures, projects cost ranges, and suggests cheaper alternatives — all BEFORE the eval runs.

Phase 1 — Read the project's AI shape

Before any question:

1. Detect AI SDK presence — does the project actually have LLM calls?

   grep -rln "@anthropic-ai/sdk\|openai\|@ai-sdk\|@google/generative-ai\|gemini\|claude" \
     package.json Cargo.toml pyproject.toml requirements.txt 2>/dev/null | head
   grep -rn "from anthropic\|from openai\|import.*@anthropic" \
     --include="*.py" --include="*.ts" --include="*.js" . 2>/dev/null | head -10
   

   If NO AI dependencies, STOP. Report applicability check failed. Tell the user this skill doesn't apply.

2. AI workflow directories — find where prompts / configs / fixtures live:

   find . -type d \( -name "ai" -o -name "llm" -o -name "prompts" -o -name "evals" \
     -o -name "workflows" -o -name "agents" -o -name "fixtures" \) \
     -not -path "*/node_modules/*" 2>/dev/null | head
   find . -path ./node_modules -prune -o \
     \( -name "*.prompt.*" -o -name "*system-prompt*" -o -name "*instruction*" \) \
     -print 2>/dev/null | head
   

3. Eval suite — find the eval entry points:

   cat package.json 2>/dev/null | grep -A 2 '"scripts"' | grep -E "eval|test:ai|test:llm"
   find . -name "*.eval.*" -o -name "*eval*.py" -o -name "*eval*.ts" \
     -not -path "*/node_modules/*" 2>/dev/null | head
   ls evals/ tests/eval/ tests/llm/ 2>/dev/null
   

4. Model identifiers — read the config:

   grep -rEn "MODEL_NAME|model.*=.*\"|model: ['\"]|MODEL_ID|model_id" \
     --include="*.ts" --include="*.py" --include="*.json" --include="*.yaml" . \
     2>/dev/null | head -30
   

   Note which model tiers the project uses (frontier-large vs frontier-mid vs frontier-fast). Each tier swap is a 5-10× cost multiplier.

5. Mock mode — does one exist?

   grep -rn "MOCK_MODE\|MOCK_AI\|mockAi\|dryRun\|mock_response\|AI_MOCK\|FIXTURE_MODE" \
     --include="*.ts" --include="*.py" --include="*.js" . 2>/dev/null | head
   

   If yes, note WHERE the mock-mode check fires. Then trace whether ALL entry points to the LLM workflow pass through the mock-aware layer, or whether some bypass it (the war-story pattern at ../examples/the-test-passed-for-the-wrong-reason.md).

6. Fixture count + cascade depth — gauge the cost scale:

   find . -path "*fixtures*" -type f 2>/dev/null | wc -l
   # Read 1-2 workflow files to count cascade stages
   find . -name "*workflow*.ts" -o -name "*pipeline*.ts" -o -name "*cascade*.ts" \
     -not -path "*/node_modules/*" 2>/dev/null | head
   

7. Recent AI-cost-related commits:

   git log --oneline --grep="cost\|token\|eval\|fixture\|model\|mock" -20
   

Build mental model of: what AI architecture (single call / multi-stage cascade / agent loop), what models per stage, what mock-mode coverage looks like, what the eval cost scales with.

Phase 2 — Interview

Open interview.md (same directory). 3-4 questions. Adaptive — skip what Phase 1 already answered.

• Workflow shape — single call / multi-stage / eval harness with fixtures? Production or dev-only?
• Cost ceiling — what's a "this eval got too expensive" threshold per iteration?
• Models in use — providers + tiers per stage (often discoverable from Phase 1; confirm).
• Past cost surprises — drives the watcher's safety thresholds.

Phase 3 — Read the principles

Read these from ../../principles/ SELECTIVELY:

Always read (this kit's core):

• ai-cost-monitoring.md — the five-step projection methodology

Cross-reference (link from the watcher's body, do not re-author):

• code-review.md — mock-mode bypass is a code-review pattern; the watcher's mock-coverage check overlaps
• pre-flight.md — pre-flight should ask about cost impact for AI-workflow changes

Read the war-story example — load-bearing for this skill:

• ../examples/the-test-passed-for-the-wrong-reason.md — the canonical mock-mode-bypass failure. The watcher's mock-coverage check exists because of this class of bug; the AI-workflow-discipline rule references it.

Phase 4 — Author the kit

Agents (in .claude-staging/agents/)

• eval-cost-watcher.md — the pre-eval cost projector
  • Frontmatter: description: — derived from ai-cost-monitoring.md principle, tuned to THE user's stack
  • Tool surface: Read, Grep, Glob, Bash (for git diff, file counting, fixture counting). NO Edit, NO Write, NO ability to actually run the eval — projection is the role, NOT execution.
  • Body sections:
    • The five-step methodology (classify-diff / read-model+temperature / count-fixtures / estimate-tokens / produce-projection)
    • Project-specific model-tier map — concrete model identifiers from Phase 1 + their tier classification (frontier-large = <model-id>, frontier-mid = <model-id>, frontier-fast = <model-id>). With a "verify pricing at provider's docs — drifts" note.
    • Project-specific cascade architecture — list the stages, the passes per stage, the fixture sets per stage. If the workflow has shape "Stage A (research, 1 pass) → Stage B (synthesis, 2 passes) → Stage C (critic, 1 pass) per fixture × 20 fixtures = 80 calls per full eval," encode that.
    • Mock-mode coverage map — for each entry point to the workflow, note whether it respects mock mode. Explicitly flag any that BYPASS it (the war-story pattern). This is the most-load-bearing section: a bypass shipped to production is a four-figure surprise bill waiting to happen.
    • Cost ceiling + alarm threshold — derived from the user's interview answer to AI2. The watcher recommends NOT running the eval if the projection exceeds the ceiling without explicit override.
    • Projection-format template (affected stages / model changes / cost range / cheaper alternatives / recommendation)
    • "I do not run the eval" constraint explicit
  • Model: mid-tier reasoning model. The work is diff-classification + arithmetic; doesn't require the heaviest reasoning. Save top-tier for code-review / pre-flight.

Rules (in .claude-staging/rules/)

• ai-workflow-discipline.md (small rule; author only if helpful for the project's shape)
  • Mock-mode placement — single rule: "mock-mode flag is a property of the WORKFLOW, not the route." Cross-reference the war story. Specifically prohibit the pattern where the mock check lives only in the HTTP layer; require the workflow's entry function to check.
  • Fixture freshness — when prompts change, fixtures captured against old prompts are stale and produce false-positive eval results. Encode the convention for re-capturing fixtures (manual? scripted? what's the command?).
  • Multi-stage cost accumulation — cascade depth × passes × fixtures × output tokens. The cost-watcher does the math; the rule reminds the user that this math compounds.
  • Model-swap discipline — tier swaps (frontier-fast → frontier-large) need explicit justification + cost projection. The rule says: "Before swapping the model on any stage, run the watcher."
  • Skip authoring this rule entirely if the project's AI usage is minimal (single call, no cascade, no eval suite) — the watcher carries the load alone.

Skills (in .claude-staging/skills/)

• eval-regression/SKILL.md (stub — author only if the project has a meaningful eval suite)
  • User-invocable (/eval-regression)
  • Wraps the watcher + the actual eval-run command + the result-comparison step
  • Workflow: (1) run cost projection via watcher, (2) prompt user to confirm, (3) run eval against fixture set, (4) compare results to baseline, (5) flag regressions
  • This is a stub if the project doesn't yet have a structured baseline-comparison; the user fills it in as the eval suite matures.
  • Skip entirely if the project has no eval suite (just inline AI calls without a regression harness).

Hooks

The AI-workflow kit doesn't typically ship hooks. Edit-time hooks would have to read the diff and project cost on EVERY edit; the agent doing that on demand is the right shape. One exception: a pre-eval-run wrapper script could invoke the watcher automatically; that's the eval-regression skill's job, not a hook.

Phase 5 — Stage + present + commit

Staging

Write everything to .claude-staging/ first, organized by artifact type.

Present

Walk the user through:

1. The kit overview — what landed (typically 1 agent + maybe 1 rule + maybe 1 skill)
2. Top 2-3 highlight artifacts — concrete reasoning. NOT "I added a cost-watcher" but: "Your workflow has <N> stages × <M> passes × <K> fixtures = <total> calls per full eval. Model tier per stage: <list>. Currently your bill for one full eval projects at $<low>-$<high>. The watcher will run pre-eval and bail if any single run exceeds your $<ceiling> threshold."
3. Mock-mode coverage — call out explicitly which entry points respect mock mode and which BYPASS it. If you found a bypass, surface it: "Your dev cron at <path> calls the workflow directly, bypassing the mock check at <other-path>. The war-story pattern. The watcher will flag any change that propagates work through the bypassed path."
4. What got SKIPPED — and why. "Skipped eval-regression skill because no eval suite found — re-run this skill when you build one."
5. Model + token-cost note — eval-cost-watcher uses mid-tier reasoning model and is cheap to invoke (compared to the costs it projects). Worst case it costs $0.10 to prevent a $40 surprise eval — favorable asymmetry.

Approve → commit

After explicit user approval, move .claude-staging/ → .claude/ and commit with structured message:

feat(.claude): AI workflow discipline (dotclaude:ai-workflow)

Authored:
- agents:  eval-cost-watcher
- rules:   [ai-workflow-discipline]
- skills:  [eval-regression]

Workflow shape: <single-call / cascade / agent loop>
Models in use: <tier 1 IDs, tier 2 IDs, tier 3 IDs>
Mock-mode coverage: <fully covered / N bypasses flagged>
Per-eval cost projection: $<low>-$<high>
Alarm ceiling: $<user-supplied>

Non-negotiable rules for this flow

1. Applicability gate is HARD. If Phase 1 shows no AI SDK dependencies, this skill DOES NOT APPLY. Don't ship a phantom cost-watcher for a project with no AI workflow. Report applicability failure cleanly and stop. The user can re-invoke once AI lands in the codebase.

2. The watcher does NOT run the eval. Projection from source files, never from actual model calls. The whole point is to project BEFORE spending; if the watcher itself burns calls to estimate, the asymmetry collapses. Tool surface excludes anything that could invoke the workflow.

3. Mock-mode coverage is the highest-priority finding. A workflow that bypasses mock mode is a four-figure surprise bill waiting to fire. If Phase 1 surfaces a bypass, the watcher's body should treat it as an explicit ALWAYS-FLAG finding on any change affecting the bypassed path. Don't bury it; surface it.

4. Pricing is in a maintained config, not hardcoded. LLM provider prices change. The watcher's body should reference a maintained config (project-local ai-pricing.json or similar) with a "verify against provider's docs" note. NEVER hardcode numbers — they go stale and the watcher's projections become silently wrong.

5. Projections are ranges, not points. Different fixtures produce different output lengths. Different inputs produce different cache-hit behavior. The watcher reports $low - $high and acknowledges the gap. Single-point estimates that turn out to be 3× off train users to distrust the agent.

6. Cheaper alternatives are mandatory output. A projection alone says "this will cost $X." A projection + alternatives says "this will cost $X; here are three ways to get the same signal for less." Always provide the alternatives — subset of fixtures, mock mode (if applicable), skip a cascade stage, run only the affected stage. The user needs the option list to make the run/skip decision.

7. Tier swaps are called out separately. Swapping frontier-fast for frontier-large is a 5-10× multiplier on its own — the highest-impact category of change. Don't bury it inside a general projection; the watcher should have a dedicated "Model changes" section that flags swaps explicitly with the multiplier number.

8. The watcher is cheap; the eval is expensive. The watcher's own model is mid-tier and its context is bounded. If the watcher itself approaches the cost of the eval it's projecting, the whole abstraction is broken. Watch the watcher's spend during authoring — keep its prompt focused, its context tight, and its model choice modest.