Icse supplementary

ICSE Supplementary Material

ICSE's page model is stricter than most authors coming from ML venues expect, and it inverts their packing instincts. The 2027 research-track format (read 2026-07-08): main text ≤ 10 pages inclusive of all figures, tables, and appendices, plus up to 2 pages of references only. There is no unlimited in-PDF appendix. Everything else travels as supplementary material — an anonymized link in the paper or an upload via HotCRP — and must be assumed optional reading.

The placement rule

One question decides placement: must a reviewer see it to score the paper?

Content	Placement	Why
Anything a criterion score depends on: RQ answers, key tables, protocol essentials, the threat analysis	The 10 pages	Optional material cannot carry mandatory weight
Full prompt templates, extended proofs, extra result tables, per-subject breakdowns	Package, summarized in-body	Body states the finding + pointer; package holds the mass
Tool source, benchmark, scripts, raw outputs	Replication package	This is the verifiability substrate (icse-reproducibility)
Interview guides, codebooks, instruments	Package, named in the availability statement	Reviewers audit method through them when they choose to
Demo video, screenshots gallery	Package, only if genuinely clarifying	Never as a substitute for a precise in-body description

The failure mode is load-bearing supplement: "due to space, the evaluation methodology is described in the supplementary material" invites the reviewer to score what the body shows — an unevidenced claim. Compress into the body instead: a protocol table instead of two prose pages, a summary row instead of the full matrix.

Summarize-and-point discipline

Every relocation leaves a residue in the body — the conclusion plus a precise pointer:

Patches for all 214 bugs, with per-bug generation logs, are in the package (results/patches/); Table 5 reports the aggregate. The full 41-item codebook is in codebook.pdf; Table 2 shows the 6 categories with ≥10% prevalence.

Precise pointers (directory, file, section of the package README) also pay forward: they become the artifact-evaluation roadmap after acceptance.

Package layout reviewers can navigate in five minutes

replication-package/
├── README.md            # map: claim/table/figure -> command -> output
├── REQUIREMENTS.md      # env: versions, container digest, hardware, runtime
├── LICENSE              # open license even at review time (anonymized holder)
├── code/                # tool source, pinned dependencies
├── benchmark/           # subjects or fetch-scripts with SHAs + mining date
├── scripts/             # one entry point per paper table/figure:
│   ├── run_rq1.sh       #   names match the RQs, not internal jargon
│   └── make_table3.py
├── results/             # raw outputs the paper's numbers derive from
└── appendix.pdf         # extended tables/proofs that did not fit the body

The README's first section is a claims map: each headline claim, the command that regenerates its evidence, expected runtime, and expected output. A reviewer who checks one row and succeeds extends trust to the rest; one who hits a broken path on the first try generalizes that, too.

Anonymity of the archive

The supplement must hold the same double-anonymous line as the paper: git archive exports (no history), scrubbed notebook metadata, no /home/username/ paths in configs or logs, no institutional hostnames, no author names in LICENSE/setup.py/CITATION files, hosting through an anonymizing service rather than a personal account. Scrub, then re-run the entry-point scripts — anonymization that breaks paths converts your evidence into a liability.

What must stay out of the package

Screening for exclusions is as important as packing inclusions: third-party code or datasets whose licenses forbid redistribution (link with pinned fetch-scripts instead); human-subjects raw data beyond what consent covers — transcripts, recordings, un-aggregated survey rows; credentials, API keys, and internal hostnames hiding in config files and CI logs; and proprietary industrial code shown to you under NDA, even in fragments quoted in comments. A replication package is a publication: it gets the same legal and ethical review as the paper, and a leak in the supplement is harder to retract than a sentence.

Size and dependency realism

No supplement size cap was verifiable for 2027 (待核实 on the live HotCRP form), so engineer for reviewer patience instead: keep the download small (fetch-scripts for large corpora, with checksums), keep the smoke test under minutes on a laptop, and never require credentials, cluster schedulers, or paid API keys for at least a reduced-scale verification path. Provide cached outputs beside every expensive step so the analysis chain can be verified without the compute.

Reverify each cycle

The 10+2 model, the HotCRP supplementary option, and the availability- statement placement are 2027-cycle facts; page models have changed across ICSE editions before. Confirm the current call before deciding any placement that would be expensive to reverse in deadline week.

Output format

[Body/package split] each relocated item -> residue sentence + pointer present?
[Load-bearing check] any criterion-relevant content living only in the package? (must be none)
[Package audit] layout roles present; claims map covers every headline number
[Anonymity + runnability] scrub done; entry points re-run post-scrub
[Reviewer path] smoke test minutes, laptop-feasible, credential-free