Ieeesp reproducibility

IEEE S&P Reproducibility

Use this while the evidence is being produced — months before ieeesp-artifact-evaluation packages it. Security results decay in ways ML results do not: targets get patched, infrastructure changes, and an attack that "worked in March" can be unreproducible by review time. Reproducibility at S&P is first about freezing the world you measured.

Freeze the target, not just the code

Evidence type	What must be pinned	Decay risk if not
Software exploit	Target version, build flags, distro, patch level	Silent fix ships mid-review
Microarchitectural attack	CPU model + stepping, microcode, OS mitigations state	Microcode update changes timing
Network measurement	Scan dates, vantage points, target-list snapshot	Internet moved; sample unrecoverable
Web/API study	Crawl date, client fingerprint, geographic origin	Server-side behavior shifts
ML attack/defense	Model weights hash, dataset version, threat-budget ε	"Same" model retrained differently

Record these in the paper, not only in lab notes — a reviewer asking "does this still work after <vendor>'s April update?" is a reproducibility question about the world, and the answer is a documented snapshot date.

Determinism ledger for probabilistic evidence

Attacks with success probabilities, fuzzing campaigns, and randomized defenses need trial discipline:

• Report trials, not anecdotes: success rate over n runs with n stated, plus dispersion — a "9/10 successful" exploit and a "measured once" exploit are different claims.
• Fuzzing comparisons follow the field's known pitfalls: equal CPU-time budgets, multiple campaigns, identical seed corpora — state all three.
• Randomized defenses (ASLR-like, moving-target): evaluate across the randomness, not one lucky layout.
• Timing measurements: report the noise floor and the machine's quiescence conditions (isolated cores, frequency pinning) or the numbers will not transfer.

Reproducibility ledger (one row per experiment in the paper):
  exp_id | claim it supports | target snapshot (ver/date/hw) |
  trials & seeds | dispersion reported? | rerun cost (time/hw/$) |
  rerunnable by outsider? (yes / gated / world-dependent)

The last column becomes the availability statement and the honest badge target later.

When ethics limits release, say exactly what and why

S&P reviewers accept withheld material when the reasoning is specific:

• "Exploit for CVE-pending issue withheld until the fix ships; released to reviewers via the chairs on request" beats a silent gap.
• User-derived datasets: describe schema and collection so others can re-collect ethically, even when raw data cannot ship.
• Never fabricate openness — a promised-but-empty repository found during review or after publication is a reputation event, and at this venue reviewers do check.

Cheap habits that pay at rebuttal time

• One env.lock per experiment directory: container digest, package list, kernel and microcode versions, dumped automatically by the run script.
• Raw outputs archived before aggregation; the rebuttal question is always about a number two steps upstream of the figure.
• A regenerate_figures.sh that goes from archived raw data to every figure — this is also the artifact-evaluation core later.
• Date-stamped disclosure and measurement logs, because ethics questions in review are answered with timelines (ieeesp-author-response).

What this venue does not require

Keep effort calibrated: S&P has no submission-time reproducibility checklist in the verified 2026/2027 materials (待核实 each cycle), artifact evaluation is post-acceptance and optional, and appendices are explicitly not guaranteed reader attention. The reproducibility work above is therefore aimed at three audiences in order: your own rebuttal, the shepherd, and the AE committee — not at a submission-form requirement.

Output format

[Ledger status] <n>/<total> experiments with snapshot + trials + dispersion
[World-dependence] <which results cannot be re-run by anyone, ever — flagged in text?>
[Release plan] open / gated (reason) / withheld (reason) — per component
[Rebuttal readiness] raw data archived ✓/✗ · env locks ✓/✗ · figure regen ✓/✗
[Gaps to close before registration week] <ordered list>