Colm supplementary

COLM Supplementary Material

The 2026 format gives you a strict 9-page main text, unlimited citation pages, and appendix space whose guaranteed audience is nobody — reviewers may read it, the rebuttal may point into it, but no rule obliges anyone to open it. That asymmetry is the whole design principle: the main text carries every load-bearing element, and the supplement carries the audit trail that makes the main text checkable.

The placement rule

Ask of each artifact: is this needed to believe the claim, or to re-run it? Believe → main text. Re-run → appendix/supplement. Both → summarize in the main text, deposit fully in the appendix, and cross-reference in both directions.

Artifact	Placement	Note
Representative prompt (or skeleton)	Main text	Readers cannot judge an LM evaluation without seeing how the model was asked
Full prompt set, verbatim, incl. system prompts	Appendix	Paraphrased prompts are not reproducible; include exact strings
Decoding configs per experiment	Appendix table	One row per reported table cell group (colm-experiments)
Contamination analysis method + results	Summary in main text; full in appendix	The conclusion is load-bearing; the n-gram machinery is audit trail
Per-task / per-model full result grids	Appendix	Main text shows the aggregate + the interesting slice
Human-eval protocol: instructions, pay, demographics, agreement	Appendix	Screenshots of annotator UI earn trust
Failure cases and transcripts	Appendix, curated	A dozen representative transcripts beat 400 raw ones
Code + data package	Supplementary upload / anonymized repo	Must be anonymous at submission (see below)
Raw API responses archive	Release artifact (acceptance)	Often too large for review upload; describe it and commit to release

LM-specific supplement content reviewers actually use

Three appendix sections repeatedly decide borderline LM papers:

1. The prompt appendix. Exact strings, exemplar counts, formatting whitespace included. Prompt paraphrase is a known variance source, so approximation here undermines every number.
2. The "what we tried" ledger. Prompt variants and hyperparameters attempted per system — the fairness evidence for baseline comparisons.
3. The contamination appendix. Overlap statistics per evaluation set, or the documented reason none could be computed.

Anonymization of the code/data package

COLM's instructions ban identity-revealing links, and supplements are the usual leak. Before upload, scrub: repository remotes and commit author fields (git log travels inside .git/), Hugging Face org names in from_pretrained(...) calls, W&B entity names in configs, API org IDs in cached headers, absolute paths (/home/<username>/...), and internal cluster hostnames.

# Build the review package from a clean export — never zip a working directory
git archive --format=tar HEAD | (mkdir -p /tmp/colm-pkg && tar -x -C /tmp/colm-pkg)
grep -rniE 'wandb|hf_[A-Za-z0-9]{20,}|api[_-]?key|/home/[a-z]|<lab-name>' /tmp/colm-pkg | head
grep -rlE 'from_pretrained\("[^"]*/' /tmp/colm-pkg --include='*.py' | head   # org-scoped model IDs
(cd /tmp && zip -rq colm-supplement.zip colm-pkg -x '*.git*')

git archive (not cp -r) is the load-bearing choice: it cannot smuggle .git/, untracked scratch files, or environment files into the package.

Referencing the supplement from the main text

• Point at specific anchors ("App. C.2, Table 9"), never "see appendix".
• In the rebuttal window (May 22 - June 8 in 2026), precise anchors let you answer a reviewer in one sentence plus a pointer — the highest-leverage use of everything you deposited.
• Do not promise supplement content that is not there; reviewers who follow a dangling pointer stop trusting the intact ones.

Ordering the appendix for the reader you hope exists

Appendices get opened with a question in hand, not read linearly. Order sections by the probability of the question:

1. A. Experimental details — models, revisions, decoding configs, harness commit: the reproduction question.
2. B. Prompts — verbatim, grouped by experiment: the "how did you ask" question.
3. C. Contamination analysis — the trust question.
4. D. Full results — per-task, per-model grids: the "does it hold on X" question.
5. E. Baseline tuning ledger — the fairness question.
6. F. Human/judge evaluation protocol — instructions, agreement, biases.
7. G. Failure analysis — curated transcripts.
8. H. Compute — the cost question.

Within each, lead with a two-line summary of what the section shows — the reviewer deciding whether to trust an unread appendix trusts one with visible structure. And keep appendix numbering frozen after submission: the rebuttal will cite "App. C.2" and a renumbered revision breaks every pointer in the discussion thread.

What not to include

• Anything load-bearing that appears only in the supplement — if the 9 pages cannot make the case alone, restructure (colm-writing-style).
• Un-curated dumps (every transcript, every checkpoint) — volume without curation reads as evasion.
• Data whose license or ToS forbids redistribution — describe it and state the restriction instead (colm-artifact-evaluation).
• 待核实: no supplementary size cap was verifiable for 2026; check the OpenReview form's actual limits rather than assuming.

Timing note for the 2026-style calendar

In 2026 the supplement traveled with the paper at the March 31 deadline — no separate later upload was posted (待核实 whether future cycles add one). Plan accordingly: the anonymization sweep and package build belong in deadline week's T-3 slot (colm-submission), not in an imagined grace period. And because the May 22 - June 8 rebuttal leans on appendix anchors, whatever you deposit in March is the ammunition you will fight with in June — deposit for your future self, who will be tired and have eighteen days.

Output format

[Placement audit] load-bearing content in supplement? none / move up: <items>
[Prompt appendix] verbatim + system prompts ▢   [Tried-ledger] ▢   [Contamination appendix] ▢
[Anonymity scan] clean / hits: <files>
[Pointer integrity] all anchors resolve / dangling: <refs>
[Package] built via clean export ▢  size: <n> MB (cap 待核实)