Statistical reporting

statistical-reporting

Never report a single run as a result. A number from one draw of a stochastic system is an anecdote; a claim is a distribution summarized honestly: center, spread, replication count, and — for comparisons — a paired design with the pairing stated. This skill governs how numbers become claims.

When to use

• A numeric result is entering a build-log entry, a decision's evidence, or a paper draft.
• Designing a campaign: choosing the replication unit and count before launch.
• Comparing methods, configurations, or arms.
• Reviewing a draft for naked point estimates.

When NOT to use

• Deterministic quantities (an exact count, a closed-form value, a proven bound) — report the value; intervals around determinism are theater.
• Exploratory probes not yet feeding a claim — but promote them through this skill before they do.
• Choosing seeds/realization mechanics — run-provenance.

Runbook

1. Declare the replication unit before running — what varies across replications (random instances, seeds, data splits, noise draws) and what stays fixed. The unit defines what population the claim generalizes to.
2. Set N in the run contract, not after seeing results. Working default: ~10 realizations per cell for expensive computations, more when cheap; the honest minimum is whatever supports the interval you intend to report.
3. Run paired where comparing. All arms see the same realization list (same instances, same seeds); report per-realization differences, not differences of aggregates.
4. Summarize with center + spread + N: median (or mean) with an interquartile band / bootstrap CI / standard error — and N visible in every table and caption. For small N or unknown distributions prefer rank-based tests (e.g. paired signed-rank) over normality assumptions.
5. Keep denominators visible. Failed, infeasible, and timed-out cells are part of the result: "median over the 6/10 realizations that completed; 4 infeasible (recorded)" — never silently condition on success (evidence-convention rule 5).
6. Report direction, magnitude, and uncertainty — not a bare p-value. A significant difference of negligible size and a non-significant trend are both reportable truths; say which you have.

Rules

1. One run, no claim. A single-realization number may appear only as a labeled illustration ("representative realization"), never as the result.
2. The comparison set is declared in advance (the run contract / expectation field — see evidence-convention). Comparisons invented after seeing the numbers are labeled post-hoc and carry no confirmatory weight.
3. No selective replication. Adding realizations until the test passes, or dropping "unlucky" seeds, is p-hacking with extra steps; N is fixed by the contract, deviations are recorded.
4. Match before you compare. Arms differing in more than the treatment (different instances, different budgets) produce differences with no interpretation; matching is a design property, not a statistical patch.
5. Uncertainty from the right variance. Spread across realizations ≠ spread within one realization's iterates; state which is shown.
6. Small-N humility. With N ≤ 5, report the raw per-realization table alongside any summary — readers can then judge for themselves.

Configuration

• {{evidence_dir}} — campaign outputs and per-realization tables.

Provenance & maintenance

Authored from the statistical-reporting canon for computational experiments (replication, pairing, visible denominators, rank-based small-N tests), grounded in working practice: a campaign design of ~10 realizations per cell with paired rank-based comparison across arms on identical realization lists, infeasible cells recorded verbatim as findings, and per-realization difference tables computed post-hoc from the evidence CSVs. See the pack's examples/ directory for the worked exemplar mapping.

Re-verify in your project:

• grep -h "seeds\|realizations" {{evidence_dir}}/*/meta.json | head -3 — N is declared in the contract, not improvised.
• Spot-check a recent table/caption for center + spread + N all present.
• grep -rn "INFEASIBLE\|TIMEOUT\|failed" {{evidence_dir}}/*/README.md | head — denominators are recorded, not conditioned away (adapt status names).