Cikm experiments

CIKM Experiments

The empirical program of a CIKM paper answers to three evaluation cultures. IR culture wants ranked-metric discipline on recognized collections with significance testing; mining culture wants mechanism isolation, scaling behavior, and honest baselines; KM/database culture wants evidence that the method survives real data — heterogeneous, noisy, incomplete. Design the experiment section as the union of what the claimed lanes require, not the maximum of one.

Claim-lane-evidence contract

For every claim sentence in the introduction, write down which lane it invokes and what that lane's standard evidence is:

Claim invokes...	Required evidence pattern	Frequent hole
Better retrieval/ranking	Standard collections, tuned baselines, metric@k with paired significance tests	Untuned baselines; "significant" without a test
A better mechanism	Ablation removing exactly that mechanism; sensitivity to its key parameter	Ablations that vary three things at once
Scalability	Time/memory vs. input-size curves on stated hardware	A single wall-clock number, no curve
Robust to real data	Noise/incompleteness injection or a genuinely messy dataset	Only clean benchmark data
KG/side-information helps	The identical model minus the KG, plus a KG-quality sweep	Confounding KG addition with capacity increase
Deployed impact (Applied track)	Launch evidence: online metrics, A/B or pre/post windows, traffic scale	Offline proxy metrics presented as deployment proof

The Applied Research row is a track requirement, not a style preference: CIKM 2026's call demands substantiation by a system launch, data release, or equivalent practical evidence.

Dataset strategy for a blended panel

• Include at least one dataset each lane recognizes as legitimate for the task — a public IR/recommendation benchmark, a graph/KG corpus, and (when claiming practice) an industrial or organizational dataset, released or described honestly.
• Report per-dataset licenses, sizes, and collection windows; the resource-minded part of the pool reads dataset tables the way IR readers read metric tables.
• For LLM-era work, state the contamination position: whether test queries, documents, or KG facts could sit in a pretrained model's training data, and what was done about it. The blended pool increasingly asks this across all lanes.

Mechanism isolation at the boundary

CIKM contributions typically fuse components (a ranker + a KG; a miner + an index). The decisive experiment is the one that holds the fusion fixed and removes only the claimed novelty — otherwise the panel cannot tell the contribution from the engineering. Budget for it first; it is the experiment reviewers ask for when it is missing, and with no confirmed rebuttal channel at CIKM (待核实), a missing ablation cannot be repaired mid-review.

Reporting floor

For every results table/figure:
  - datasets named with split protocol and seed policy
  - baseline tuning budget stated (same search space as the proposed method?)
  - variance across runs, and the test behind any "significant"
  - hardware + software versions for anything timed
  - pointer to the artifact path that regenerates it

The floor exists because the appendix cannot absorb it: 2026 budgets count appendices inside the page limit, so this information lives in captions, in the protocol paragraph, or in the cited artifact — never nowhere.

Variance and significance protocol

The IR lane brings the family's strictest statistics culture, and it grades the whole paper by these habits:

• Multiple runs with distinct seeds for anything stochastic; report the spread (std or CI), not just the best run — "best of five" is a red flag, "mean ± std of five" is the norm.
• Paired tests for system comparisons on shared queries/instances; say which test and at what level, and correct for multiple comparisons when sweeping many configurations.
• Effect sizes over asterisks alone: a 0.2-point gain can be "significant" on a large query set and still not matter; state both the margin and the test.
• Per-slice reporting when the claim is slice-shaped (the gains "concentrate on relation-dependent queries" claim needs the slice table that shows it).

LLM-era pitfalls the panel now checks

• Contamination: pretrained components may have seen the test collections; state the exposure analysis or choose post-cutoff data.
• LLM-as-judge: if an LLM scores outputs, validate it against human labels on a sample and report agreement; unvalidated judge scores are decoration.
• Prompt variance: prompted baselines need the same tuning-budget parity as trained ones — report the prompt-search effort on both sides.
• API drift: hosted-model results must pin model version and date, or the experiment is unrepeatable by construction (cikm-reproducibility).

Design vignette

A paper claims: entity linking over enterprise wikis improves internal search. The minimum honest program: (1) linking quality vs. TAGME-line baselines on a public corpus, with significance; (2) end-to-end search quality with linking on/off — same index, same ranker; (3) an ablation degrading link precision synthetically to show the dependence; (4) one messy-data run (stale pages, duplicate entities) with the failure modes described. Four experiments, each answering a different lane.

Budgeting the program against the May gate

Experiment programs at this venue fail by sequencing more than by design. Order of execution when time is short: (1) the decisive ablation — it defines whether there is a paper; (2) the headline comparison with proper baselines and variance — it defines how strong; (3) one lane-coverage run for whichever community the claim still leaves unserved; (4) robustness/messy-data sweeps; (5) everything else. Cut from the bottom, never from the top. And freeze the protocol (splits, metrics, tuning budgets) in writing before results exist — protocol decisions made after seeing numbers are the reproducibility crisis's origin story, and the blended panel includes people who ask when the protocol was fixed.

Output format

[Contract table] <claim → lane → evidence → status>
[Decisive ablation] <what is removed, what stays fixed>
[Panel coverage] <which lane still has no dataset/evidence speaking to it>
[Reporting floor] <captions/protocol/artifact items missing>
[Next run] <the single experiment to schedule first>