Cad sim quality harness

CAD Sim Quality Harness

Create the smallest falsifiable evidence contract that prevents polished packets from hiding weak engineering. This is an internal helper, not a user-facing workflow entrypoint.

Contract

Write or refresh these files under the active run root:

quality/quality_harness.yaml
quality/quality_harness_results.json
quality/quality_harness_trace.md
quality/quality_reset_seed.yaml   # only for Quality Reset / clean_restart

quality_harness.yaml is the source of truth. Keep rows compact and current:

schema_version: 2
harness_id: "<stable run-local id>"
harness_version: "<date-or-counter>"
status: active
required: true
selected_profile: "physical_prototype | simulation_optimization | robotics_electromechanical | software_connected_hardware | design_review_only | hybrid"
acceptance_profile: "generic_physical | simulation_optimization | robotics_electromechanical | canoe_design_study | garmin_style_mount"
rows:
  - id: "<stable row id>"
    decision_or_risk: "<decision or failure mode>"
    intended_behavior: "<observable behavior the design claims>"
    verifier_check:
      kind: "command | structured_artifact | visual_review | boundary_review"
      target: "<command, structured file, primary-surface state, or boundary>"
      expected: "<falsifiable expected result>"
      reviewer_command: "<required for command kind; side-effect-free check/dry-run command>"
      reviewer_execution_mode: "isolated_material_copy"
      reviewer_side_effects: "none"
    current_result: "pass | fail | missing | boundary | not_relevant"
    not_relevant_reason: "<required only for not_relevant>"
    evidence_artifacts: []
    required_in_primary_surface: true
    primary_surface_location: "<path#section, or empty until promoted>"
    boundary: "none | physical_boundary | user_boundary | stage_approval"
    next_action: "<smallest next move if this row does not pass>"
    route_if_failed: "continue_autonomous | needs_user | ready_for_bench_test | ready_for_review_with_boundary"

Workflow

1. Read only the active run's allowed inputs: intake/project_brief.yaml, intake/active_review_rubric.yaml when present, current review_loop.yaml, current handoff.json, canonical annotation_view/, and latest review or user complaint. Preserve any source-boundary preflight.

2. Select one primary acceptance_profile. Read references/profile_contracts.yaml for the mandatory row IDs and allowed verifier kinds, then references/profile_templates.md for authoring guidance. Every mandatory row must be present. Add at most three run-specific risk rows.

3. Prefer generated checks, plots, CAD/spatial screenshots, simulator exports, or compact calculations over prose. A pass must use an allowed verifier; boundary_review is only a pass basis when the row itself is bounded. Every command verifier must provide a side-effect-free reviewer_command, declare reviewer_execution_mode: isolated_material_copy and reviewer_side_effects: none, and list its local helper/input files in the verifier mapping. Authoring may run a normal result-producing command before freeze; reviewers never run that command directly against the run root.

4. Generate quality_harness_results.json from the YAML source of truth:

   python scripts/harness_results.py <run-root>
   python scripts/harness_results.py <run-root> --check
   

   Never hand-edit the JSON mirror. Write quality_harness_trace.md with the short human trace from row to evidence and boundary.

5. Add pointers, not duplicates: intake/project_brief.yaml.review_rubric.quality_harness.path and intake/active_review_rubric.yaml.quality_harness.path should point to quality/quality_harness.yaml.

6. If the current packet is patchwork, stale-proof driven, or still bad after repeated fuzzy complaints, write quality_reset_seed.yaml and route through existing clean_restart state values. Old artifacts may be benchmarks, anti-patterns, or requirements evidence, never current proof.

Completion

Stop when every required row is pass, not_relevant, or explicitly boundary with physical_boundary, user_boundary, or stage_approval; each readiness-critical row has a primary-surface location when required; and route preflight can enforce the same facts without reading prose.