Ppopp experiments

PPoPP Experiments

Design the evaluation to clear PPoPP's twin bar: the contribution must be correct under concurrency and measurably scalable. A speedup with no correctness argument, or a correctness proof with no scaling data, each fails half the venue. Reviewers are parallel-systems experts who will interrogate the baseline, the machine, and the variance before they believe a number.

Match evidence to the claim

Claim shape	Evidence PPoPP expects	Common failure it catches
A lock-free/wait-free structure	Throughput vs. thread count under varied contention; a linearizability/progress argument; memory-reclamation overhead	Single contention level; "no race seen" instead of an argument
A parallel runtime/scheduler	Overhead vs. sequential; strong+weak scaling on real workloads; load-balance behavior	Microbenchmarks only; no real application
A GPU/accelerator technique	Speedup over a strong GPU baseline; occupancy/divergence analysis; transfer costs counted	Ignoring host-device transfer; a weak baseline kernel
A parallel algorithm	Scaling on real inputs; NUMA/locality effects; comparison to the best known implementation	One input; a naive baseline
A memory-model / race tool	Soundness/coverage claims; runtime overhead; false-positive/negative characterization	Overhead unmeasured; no ground truth

The scalability story

• Show the curve. Report performance as a function of thread/core count, not one configuration. The interesting information is the shape: linear region, saturation point, collapse.
• Distinguish strong vs. weak scaling and label which you show. Strong scaling (fixed problem, more cores) and weak scaling (problem grows with cores) answer different questions; conflating them is a classic PPoPP tell.
• Sweep the topology. Cross-socket and NUMA effects, thread pinning, and (for GPUs) occupancy and divergence often dominate; a single-socket-only result invites "what about NUMA?"
• Count the hidden costs. Memory reclamation, host-device transfer, allocation, and scheduling overhead belong inside the reported numbers, not in a footnote.

Correctness under concurrency

• Provide an argument, not just testing: linearizability (with linearization points), lock-freedom/wait-freedom (progress), or a checked property. "Passed a stress test" bounds confidence but does not establish correctness.
• Name the memory model you assume (C/C++11 atomics, the GPU model, hardware TSO) and show your synchronization is correct under it, not just under sequential consistency.
• If you use a model checker or race detector to support the claim, report its configuration and what it covers.

Baselines that survive scrutiny

• Compare to the strongest real competitor, at the competitor's best settings, on the same machine — not to your own unoptimized code and not to a strawman.
• Rebuild and re-tune baselines yourself where feasible; citing a competitor's paper number measured on different hardware is not a fair comparison.
• If you are the first at something, construct the most credible reasonable baseline and justify it.

Measurement hygiene

[Repeats]     multiple runs; report median/mean with variance (error bars / percentiles)
[Warm-up]     discard JIT/cache/allocator warm-up; state the steady-state protocol
[Pinning]     pin threads to cores; state the topology and the pinning policy
[Isolation]   quiescent machine; no co-tenants; disable turbo/frequency scaling or report it
[Inputs]      real workloads plus targeted microbenchmarks; state sizes and sources
[Provenance]  exact CPU/GPU, socket/NUMA layout, memory, compiler and flags, OS

A single-run bar chart with no error bars, on an unstated machine, is the evaluation a PPoPP reviewer trusts least.

Anticipate the rebuttal questions at design time

The two questions PPoPP reviewers ask most — "does it still scale at higher core counts / on another GPU?" and "how does it compare to baseline X?" — cannot be answered in the short rebuttal window if the runs were never made. Pre-run the larger core sweep and the obvious alternative baseline before submission so the numbers are already in hand (see ppopp-author-response).

Output format

[Twin bar] correctness argument present? scalability curve present? both required
[Scaling] strong/weak labeled, core sweep, NUMA/GPU topology, hidden costs counted? yes/no
[Correctness] hazard + argument (linearizability/progress) under a named memory model? yes/no
[Baselines] strongest real competitor, same machine, tuned? yes/no
[Hygiene] repeats+variance, warm-up, pinning, isolation, provenance? list gaps
[Rebuttal pre-runs] higher core count + alternative baseline already measured? yes/no