Autocalib

EDPA Auto-Calibration — Monte Carlo signal-weight optimizer

What this does

Optimizes the three signal weights (commit_author, pr_reviewer, issue_comment) in plugin/edpa/templates/cw_heuristics.yaml.tmpl against a synthetic corpus generated procedurally. The engine consumes those weights directly — there is no role_weights or role_overrides block any more (both were dropped in v1.11; see plugin/edpa/scripts/engine.py:864).

The optimizer is self-contained: it generates its own ground truth via Monte Carlo, evaluates candidate weight vectors against it, and writes the best candidate back into the template when --apply is passed.

Arguments

$ARGUMENTS = optional flags forwarded to calibrate_signals.py. Common forms:

• empty / help → show current calibration metadata, propose a default run
• quick → adds --quick (200 MC samples; ~1 s; smoke test only)
• a positive integer → --scenarios <N> (e.g. 2000); default 1000
• apply → after calibration, write best weights back to the template
• raw flags (--scenarios 2000 --seed 7 --apply --report report.json) → passed verbatim

Argument resolution (when $ARGUMENTS is empty)

1. Read the current calibration: block from plugin/edpa/templates/cw_heuristics.yaml.tmpl and print:

   Last calibration:
     method:        MC random-sample + coordinate descent
     scenarios:     1000   records: 31041
     baseline MAD:  0.0889
     calibrated:    0.0869 (+2.2%)
     version:       1.11.0
     timestamp:     2026-05-08T18:37:24Z
   

2. Suggest defaults:

   Suggested run:
     python3 plugin/edpa/scripts/calibrate_signals.py \
       --scenarios 1000 --seed 42
   
   Apply best weights to the template? [N]
   

3. Wait for user confirmation (run / apply / change scenarios / cancel).

Prerequisites

None for the synthetic path. The MC corpus is generated in-process; no .edpa/data/ground_truth.yaml is needed. The legacy role_weights / role_overrides schema and the evaluate_cw.py autoresearch evaluator were removed in v1.18.2.

If the user explicitly asks to calibrate against real PI data, fall through to "Re-run with real data" below.

Configuration

Target file:    plugin/edpa/templates/cw_heuristics.yaml.tmpl
Script:         plugin/edpa/scripts/calibrate_signals.py (LOCKED)
Metric:         mean_absolute_deviation(predicted_cw, true_cw)
                where predicted_cw = Σ weight × signal_count, per-item normalized
Direction:      lower
Phases:         (1) MC random sampling   default 2000 samples (200 if --quick)
                (2) coordinate descent   refines top-5 candidates
Search space:   3D, each weight ∈ [0.1, 8.0]
Defaults:       commit_author 4.00, pr_reviewer 2.17, issue_comment 1.46

CRITICAL: never edit calibrate_signals.py. The synthetic corpus generator and the MAD cost function are inside the same script intentionally — the locked-vs-tunable separation is preserved by structure: the cost function takes only a candidate weight vector and pure-reads signal_count × weight with per-item normalization (no parameters live inside the cost function itself). If you are tempted to modify the generator to match a particular weight vector, STOP — that gamifies the metric. Add new scenario flavors only when they reflect a real-world contribution pattern the corpus does not yet model, and even then file a separate PR — not inside a calibration run.

Run

Step 1 — Execute

python3 plugin/edpa/scripts/calibrate_signals.py \
  --scenarios "${SCENARIOS:-1000}" \
  --seed "${SEED:-42}" \
  ${QUICK:+--quick} \
  ${APPLY:+--apply} \
  ${REPORT:+--report "$REPORT"}

Expected stdout (abridged):

Generating 1000 synthetic scenarios (seed=42)...
  → 31041 (person, item) records across 1000 scenarios
Baseline (shipped defaults): MAD = 0.0889
Phase 1 — Monte Carlo random sampling (2000 samples)...
  Top 5 candidates by MAD: ...
Phase 2 — Coordinate descent refinement...
  Cand 1: 0.0881 → 0.0869 after refinement
  ...
Best calibrated weights (MAD = 0.0869):
  commit_author: 4.00
  pr_reviewer: 2.17
  issue_comment: 1.46
MAD improvement: 0.0889 → 0.0869 (+2.2%)

Step 2 — Report

Summarize the run to the user:

• baseline MAD, calibrated MAD, % improvement
• which weights moved most (delta from defaults)
• whether --apply was used (template updated or not)

Step 3 — Apply (only if requested)

When the user passed apply, calibrate_signals.py --apply has already rewritten the template signals: block and refreshed the calibration: metadata. Confirm by re-reading the target file's calibration: block and echoing mad_calibrated and calibrated_at.

If not applied, leave the template untouched and tell the user how to apply later:

Re-run with: python3 plugin/edpa/scripts/calibrate_signals.py --apply

Re-run with real data (post-first-PI)

The MC corpus is a prior: it encodes plausible signal/cw mappings under v1.11's procedural model. After a PI closes, capture team-confirmed CW corrections in .edpa/data/calibration_corrections.yaml and run the blended calibration — real corrections are weighted 10× higher than synthetic records.

Step 1 — Add corrections after PI retrospective

# .edpa/data/calibration_corrections.yaml (use project_setup template or create manually)
corrections:
  - iteration: PI-2026-1
    item: S-200
    person: turyna
    actual_cw: 0.70
    note: "Pair session not in commits"
  - iteration: PI-2026-1
    item: S-200
    person: tuma
    actual_cw: 0.30

Each entry: iteration, item (backlog ID), person (people.yaml ID), actual_cw (team-confirmed weight, values per item should sum to ≈ 1.0). Signal counts are derived from contributors[].signals[] in the item YAML when present; otherwise inferred from the as: role field.

Step 2 — Run blended calibration

python3 plugin/edpa/scripts/calibrate_signals.py \
  --real-data \
  [--corrections .edpa/data/calibration_corrections.yaml] \
  [--scenarios 1000] \
  [--seed 42] \
  [--apply]

Or via the skill: /edpa:autocalib --real-data apply

The script:

1. Loads corrections → builds real SyntheticContribution records
2. Generates --scenarios synthetic records as regularisation prior
3. Blends real (×10) + synthetic → runs MC + coordinate-descent
4. Reports: blended MAD vs synthetic-only baseline, real record count
5. --apply writes best weights to cw_heuristics.yaml.tmpl

Argument variants for this skill when $ARGUMENTS includes --real-data:

• --real-data → blended with default corrections path
• --real-data apply → blended + write weights
• --real-data --corrections <path> → custom corrections file
• --real-data --real-weight 20 → stronger real-data influence

Keep corrections across PIs — the file accumulates evidence. The iteration field is for audit; all corrections are used together in each run.

Strategy guidance

• Smoke test / CI gate: --scenarios 200 --quick (~1 s; may not improve over baseline, that's fine).
• Honest calibration: --scenarios 1000 --seed 42 (~10 s; 31 k records).
• Thorough run: --scenarios 2000 --seed 42 --apply (~30 s).
• Stability check: rerun with 3 different --seed values. If best weights agree within ~±0.3, the result is stable. If they diverge, raise --scenarios.

Error handling

• calibrate_signals.py missing → check plugin/edpa/scripts/; do not recreate from template. Tell the user the plugin install is incomplete.
• Template file missing → same; do not synthesize. Point to plugin install state.
• MAD improvement: +0.0% after a full run → expected; the shipped defaults are already near a local optimum on the v1.11 generator. Higher --scenarios rarely changes this.
• Negative improvement → corpus generator was edited; revert that change.