Harness writing

🌐 繁體中文（canonical） · English mirror

Harness Writing — Generate a Repository-Specific Evidence-Driven Workflow

Core Model

The Harness is the sole orchestration kernel: risk routing and claim→evidence gates control state transitions; profiles select required states; adapters describe available capabilities; contracts retain project judgments; domain playbooks provide techniques inside a state.

Do not reinvent invariants or invent project parameters. File count is only a supporting signal; route primarily by reversibility, blast radius, ambiguity, external side effects, and verifiability.

When to Use

• A new project needs a Harness, contract index, and mechanical-check baseline.
• An existing project needs AI workflow adoption or recalibration.
• An existing Harness needs an audit for dual sources of truth, fake evidence, review ceremony, or contract pollution.
• Not for adding one already-decided project contract; edit it through the existing lifecycle.

Generation Workflow

1. Discover (read-only)

• Inventory structure, CI, existing rules, integration points, public interfaces, production resources, and release paths.
• Assess side effects before build/test/lint; run each safe command once and record duration, output, and baseline failures. Do not run commands that may write production resources; record the limitation.
• Inventory capabilities rather than product names: isolated writes, shell execution, tests, fresh reviewers, persistent artifacts, and access to another model family.
• Derive the five risk signals and absolute escalation triggers from real project evidence; record absent items as "none."
• If a Harness exists, modify it instead of generating a second one.

2. Generate Kernel / Profiles / Adapters

Read all of harness-core-template.md and fill every {{SLOT}}. Do not silently remove these invariants:

1. Contracts and instruction precedence come first.
2. Mechanical checks precede AI review.
3. PASS requires claim→evidence; reviewer self-report is not evidence.
4. Spend user attention only on costly ambiguity, irreversible choices, or external side effects.
5. Preserve regression baselines and scope checks.
6. Use the candidate → active → hardened → deprecated contract lifecycle.
7. Degrade honestly when independent reviewers are unavailable; never pretend two reviews occurred.
8. Review convergence: only blocking findings trigger re-review; re-reviews continue the prior reviewer scoped to the delta; a PASSed claim with unchanged evidence cannot be reopened; the loop caps at two rounds, then escalates to the user.

List any proposed exception and its reason for user adjudication.

Write the parameters as JSON, then run scripts/render_harness.py --template harness-core-template.md --config <config.json> --output <target-repo> to produce the three artifacts. The renderer proves only field completeness, full slot resolution, and overwrite refusal; discovery evidence must still prove every parameter.

3. Seed Contracts

• Existing explicit rules may start as active; include provenance and a test/check.
• Rules inferred from code conventions start as candidate, never active from one observation.
• New projects get only an index, schema, and maintenance rules—not imagined best practices.
• Promote a candidate after repeated observation, an explicit user ruling, or concrete incident evidence; promote to hardened only after low-cost mechanical enforcement exists.

4. Wire It In

• Generate .claude/skills/harness/SKILL.md, references/pipelines.md, and docs/contracts/README.md.
• Project instructions point to one orchestration entry: Harness. Process skills such as feature-flow become profiles/state strategies, not a second state machine.
• Store artifacts under docs/plans/<slug>/; legacy paths may remain as compatibility entries only when one canonical source is named.

5. Validate Delivery

• Smoke-route one quick, standard, and high-risk scenario; verify route reasons and required evidence.
• Run rg '\{\{[^}]+\}\}' .claude/skills/harness docs/contracts against rendered output; any remaining slot blocks delivery.
• Confirm mechanical commands are safe and executable and adapters reference real capabilities.
• When independent review exists, separate semantic and executable review; add adversarial review only for high risk. Otherwise use fresh read-back/real execution and disclose the limitation.
• Deliver the file list, discovery evidence, baseline, risk router, adapter capabilities, initial contract states, and unverified items.

Red Lines

• ❌ File count as the only risk signal; a one-file migration can still be high-risk.
• ❌ Calling identical prompts and evidence paths an independent double review.
• ❌ PASS/REJECT verdicts without claim→evidence.
• ❌ Promoting a one-off observation directly to an active contract.
• ❌ Keeping Harness and another complete process source of truth in parallel.

References Index

• harness-core-template.md: read in full before generation; never reconstruct from this summary.
• references/evaluation-protocol.md: read when changing models, calibrating routing, or measuring process cost/effectiveness.
• references/evaluation-scenarios.json: canonical scenario ID/star mapping for the behavioral gate.
• references/test-scenarios.md: run in a fresh session after model swaps or changes to risk routing, evidence, or contract lifecycle.
• scripts/render_harness.py: validate slots and safely create the three canonical artifacts.
• scripts/evaluate_runs.py: validate JSONL behavioral records, summarize effectiveness metrics, and enforce the gate.