Neurips experiments

NeurIPS Experiments

Use this skill before submission or rebuttal when the main question is whether the evidence supports the NeurIPS claim. It is not enough to win a leaderboard; reviewers need to know why the result is scientifically meaningful.

Experiment audit

• Baselines: include strong, current, tuned baselines and explain any missing comparison.
• Ablations: isolate the mechanism, not just remove components at random.
• Robustness: test across seeds, datasets, distribution shifts, scales, hyperparameters, or realistic deployment conditions when relevant.
• Compute: disclose hardware, training time, resource assumptions, and whether comparisons are fair.
• Data: document splits, contamination controls, license, demographic or domain coverage, and privacy/consent limits.
• Negative results: use them to calibrate claims; NeurIPS has a contribution type for negative results, but the bar remains high.
• Use-inspired work: connect results to the real task without turning the paper into an application report with no ML contribution.

Claim-to-evidence ladder

NeurIPS reviewers read experiments against the claim type. Put every headline claim on the ladder before deciding whether the evidence is strong enough.

Claim type	Minimum evidence	Strong evidence
Beats prior methods	tuned current baselines, same data splits, comparable compute	multiple suites, budget-matched tuning, significance or variance reporting
Mechanism explains gains	ablation tied to the proposed mechanism	intervention or diagnostic that rules out the obvious rival mechanism
Scales better	at least two meaningful scales and fixed protocol	trend across scales with compute, memory, and failure modes disclosed
Robust to distribution shift	one out-of-domain or stress split	several shifts with error analysis and narrowed claims where it fails
Useful in the real world	task-relevant metric and realistic constraint	deployment-like evaluation, safety/fairness/privacy caveats, and cost analysis
Negative result	faithful implementation and fair reproduction attempt	explains when the prior claim holds, fails, or needs qualification

If the evidence only clears the minimum column, write the claim in minimum-column language. Reserve strong-column language for results that survive the stronger checks.

Baseline fairness table

Before submission, make a table the reviewer could audit without trusting your narrative.

Baseline issue	Required disclosure	Reviewer failure mode
Tuning budget	search space, number of trials, early stopping, compute cap	new method gets more tuning than baselines
Implementation source	official code, reimplementation, or third-party fork	weak reproduced baseline blamed on prior work
Data protocol	splits, leakage checks, preprocessing, augmentation	accidental train/test contamination
Resource match	hardware, batch size, wall-clock, memory, total cost	speed/accuracy tradeoff hidden
Selection rule	validation metric and checkpoint choice	cherry-picked best seed or test-set tuning

Missing baselines are acceptable only when the omission is named and justified: unavailable code, incompatible task, prohibitive compute, licensing, safety, or a scope mismatch. Do not silently omit the strongest comparison.

Review-dimension stress test

Map the experimental section to the dimensions reviewers will naturally score.

Dimension	Experiment question
Quality	Does the design isolate the proposed contribution rather than a confound?
Clarity	Can a reader reproduce the protocol from the paper, appendix, and checklist?
Significance	Does the effect matter beyond a narrow benchmark increment?
Contribution fit	Do the experiments match General, Theory, Use-Inspired, Concept & Feasibility, or Negative Results framing?
Ethics/reproducibility	Are data, compute, privacy, bias, and artifact limits disclosed honestly?

Rebuttal triage gate

Not every missing experiment is rebuttal-feasible. Sort reviewer requests by value and risk.

Request	Rebuttal action	Camera-ready or future-work action
Missing variance / seed concern	run a small seed sweep or report existing variance	expand seed grid if accepted
Missing obvious baseline	add it only if implementation and tuning are fair in time	otherwise explain omission and add after review
Mechanism unclear	add a diagnostic ablation or error slice already supported by the code	rewrite mechanism framing if evidence stays indirect
Dataset contamination worry	add leakage check and describe split construction	archive scripts and checklist support
New benchmark family	usually too large for response unless already prepared	narrow claim and schedule full evaluation later

Rebuttal-ready evidence

Prepare small, high-signal clarifications that can fit in an author response: a missing baseline table, a sanity check, an error analysis, a variance estimate, or a concise proof sketch. Do not depend on a complete post-review paper rewrite.

Output format

[Evidence status] strong / adequate / weak
[Main unsupported claim] <claim>
[Critical missing experiment] <baseline/ablation/robustness/data/compute>
[Review dimension at risk] quality / clarity / significance / contribution fit / ethics-reproducibility
[Baseline fairness] tuned / comparable compute / same splits / missing justified
[Small rebuttal result] <result feasible during response>
[Claim rewrite] <narrower claim if evidence stays as is>