Nsdi reproducibility

NSDI Reproducibility

A networked-systems result is a function of topology, traffic, timing, and code — and three of those four are absent from the PDF unless deliberately recorded. NSDI rewards the discipline institutionally: artifact badges after acceptance and a Community Award for the best paper whose code and/or dataset is public by the final-papers deadline. But the work happens during the experiments, not after the decision email.

The four provenance ledgers

Keep each as a versioned file next to the results it explains:

Ledger	Contents	Loss mode it prevents
Topology	node specs, NIC/switch models, link speeds, RTT matrix, kernel + NIC settings	"worked on our cluster," unreproducible knee points
Traffic	trace source + collection context, scaling/anonymization transforms, synthetic-generator parameters + seeds	headline numbers tied to data nobody can regenerate
Configuration	every config diff from defaults, per system and per baseline	untuned-baseline accusations you cannot rebut
Run	per-experiment: commit hash, config snapshot, fault schedule, raw-log location, analysis-script hash	figures that cannot be regenerated at revision time

The run ledger has a second life NSDI makes unusually likely: a one-shot revision letter can demand new measurements on the same setup, 3-4 months after submission, at a subsequent deadline. Teams with ledgers rerun in days; teams without them rebuild the testbed from memory.

Determinism is different here

Unlike ML reproducibility, networked-systems results are legitimately non-deterministic — background load, timer jitter, and cross-traffic vary. The honest posture is not "identical numbers" but characterized variance:

• Fix what can be fixed: seeds for generators and fault schedules, pinned software versions, isolated or documented-shared testbeds.
• Measure what cannot: repeat headline experiments across days/trace slices; publish the spread with the artifact so an evaluator knows a 7% delta is noise, not failure.
• Timestamp everything; time-of-day and neighbor effects are real explanations, but only if recorded.

What can actually ship?

Decide per data item before the camera-ready crunch, because the answer shapes the paper's claims:

1. Public now — code, synthetic generators, testbed configs, analysis scripts. Default to shipping.
2. Public after transformation — production-derived traces that can be anonymized/aggregated. Document the transform and what properties it preserves; state in the paper that released traces differ from raw ones.
3. Never public — proprietary traces, customer data. Plan the fallback: a synthetic workload with matched characteristics, shipped alongside a fidelity note. Do not let a never-public trace be the only support for a headline claim if a shippable proxy can corroborate it.

Anonymize the artifact itself for review-time supplements (repo owners, hostnames, paths, company strings in configs); de-anonymize only at final-paper time.

Wiring it into the repo

# Layout that keeps paper and artifact from drifting
experiments/
  <exp-id>/run.sh          # topology + config + fault schedule, self-describing
  <exp-id>/provenance.json # commit, trace id, seeds, dates, operator
figures/
  Makefile                 # every paper figure regenerated from raw logs:
                           #   make fig6 -> pulls exp logs, runs analysis, emits PDF
paper/
  claims.md                # claim -> exp-id -> figure mapping, reviewed at freeze

The claims.md cross-map is the cheapest anti-drift device: at submission freeze, walk it once; any claim whose exp-id is stale gets rerun or reworded. The shared smoke-checker (../../resources/code/README.md) covers package hygiene but not topology/trace fidelity — those checks are yours.

Blind now, open later

Reproducibility material crosses the double-blind boundary twice, and each crossing has a checklist:

• At submission (if any package is uploaded as auxiliary material): scrub repository owners, commit-author emails, hostnames that embed the institution, and license files naming the organization; keep the scrubbed and true versions as separate branches so de-anonymization is a merge, not a rewrite.
• At final papers: flip to the public, attributed release — DOI'd archive, real repository, institutional acknowledgment — dated before the final-papers deadline if the Community Award matters (nsdi-camera-ready).

Cheap wins ordered by payoff

1. provenance.json per experiment, written by the runner script (an hour to automate).
2. Figure Makefile with raw-log inputs (removes an entire class of revision pain).
3. Baseline config diffs recorded (rebuts the most common reviewer attack).
4. Variance characterization for the two headline results.
5. Early legal check on trace release — licensing takes longer than deadline gaps.

Signs the discipline is slipping

• A figure in the draft that nobody can regenerate this week.
• "The good run" language — selecting runs by outcome rather than by protocol.
• Baseline numbers copied forward from an earlier paper draft instead of re-measured on the current topology.
• Testbed changes (kernel upgrade, NIC swap) mid-campaign without a ledger entry splitting before/after results.

Any one of these, found at submission freeze, costs days; found by an artifact evaluator or a revision reviewer, it costs the result's credibility.

Output format

[Ledger status] topology / traffic / configuration / run — each present/partial/absent
[Variance] headline results with characterized spread? which lack repeats?
[Shippability] data items -> public / transformable / never (fallback named)
[Drift check] claims.md walked? stale claim list
[Award posture] on track for public code+data by final-papers deadline? 
[Next actions] ordered by payoff-per-hour