Gutcheck

gutcheck — calibrate on the human, not the proxy

Proxy metrics for subjective quality can silently point the wrong way — optimizing "engagement-style brightness" while the user actually prefers darker, bassier mixes. Ten minutes of blind gut-reaction ratings detects this and replaces the proxy with a scorer fitted to the actual human.

When to reach for this

• You are optimizing anything subjective: audio pleasingness, visual style, text tone, UI feel.
• A metric/judge says A > B but the user says B feels better — stop tuning the metric; calibrate instead.
• You are about to spend significant compute generating/ranking variants with an unvalidated quality signal.

When NOT: the target is objective (tests, latency, accuracy), there are fewer than ~8 items (just ask directly in chat), or the user isn't available to rate.

Choose a mode, tell the user why (one sentence)

• absolute (keyboard 1–5, ~5 s/item): broad first pass over a diverse batch, or N > 40.
• pairwise (A/B duels, arrow keys, Bradley-Terry): similar-quality variants, or re-ranking a top slice where absolute ratings saturated.

Run a session

python3 "${CLAUDE_SKILL_DIR}/scripts/gutcheck.py" "<glob-or-dir>" --mode absolute --open

Rules:

1. Run it with run_in_background: true. Always quote paths (spaces are common).
2. It prints one line: gutcheck: rating UI at http://127.0.0.1:7337 (...). Relay that URL, then yield your turn — tell the user to rate with the keyboard and press f when done.
3. The process exits by itself when the user presses Finish. Do not poll in a loop. When the background task completes (or the user says "done"), confirm calibration/DONE exists, then read the outputs.
4. If the user killed it mid-session, nothing is lost: re-running the same command resumes (rated items are never asked twice), and --analyze-only produces results from a partial session.

Useful flags: --out DIR (session/output dir, default calibration/), --features table.csv (feature table keyed by item id → correlation report), --max-judgments N, --port N, --seed N, --analyze-only.

Read the outputs (in this order)

1. calibration/report.md → Warnings first.
   • saturation:ceiling — ratings maxed out; offer ONE pairwise round over the top slice (--mode pairwise, new --out).
   • confound:* — a coarse group axis (directory / filename prefix) explains most of the variance; the user may be detecting group membership, not the quality you care about. Offer a within-group session before trusting the ranking.
   • graph:disconnected — pairwise order between never-compared groups is prior-driven; a few more duels fix it.
2. Obey the report's "Suggested next step" line.
3. Then use the ranking.

Consume scores.json downstream

calibration/scores.json → items[] sorted by rank with id, score (mean_rating or bt_log_strength), n_judgments. Treat it as the fitness function for subsequent generation/selection:

• Join scores to your candidate features; check the Spearman table before trusting any feature as a proxy — this is exactly where proxies invert.
• Re-aim generation at what the ranking (and variant-factor tables) reward.
• Keep the file; score future variants by their similarity to what won.

Schemas and interpretation details: references/outputs.md. Math (Bradley-Terry MM fit, saturation rule, η² confound guard): references/methodology.md.

Budget etiquette — the human's 15 minutes are precious

• One session ≤ 100 judgments (~10–15 min). Batch every question you have into it; don't run a session for 10 items now and 10 more in an hour.
• Never ask about an item twice. Resume skips rated items by design; don't work around that.
• Before proposing a second round, say in one sentence what it buys (e.g. "a pairwise round would separate the five items that all rated 5").

Limitations

• HTML artifacts render fully sandboxed; relative assets (css/images next to the file) won't load.
• Safari requires the bundled HTTP Range support — already handled; don't swap in another static server.