Edbt artifact evaluation

EDBT Artifact Evaluation

Use this for the artifact and reproducibility package. EDBT's community values re-runnable database-systems work, and the published record is open access on OpenProceedings, so a clean, archived package strengthens the paper and its permanent citation. Two things to internalize: an artifact is judged by an evaluator actually running it, and the review-time package (possibly anonymized) is not the same deliverable as the permanent, DOI-archived package cited at camera-ready.

待核实: whether the current EDBT cycle runs a formal artifact-evaluation track, offers a reproducibility badge, and makes it optional or required is decided per edition — confirm on the current host-site call. The engineering below applies regardless of whether a badge is on offer.

What evaluators open first (database-systems flavor)

Claim type	First thing inspected	Common failure caught
A system / operator	The build + one run command	Undocumented deps; only-builds-on-authors'-cluster
A scalability result	The harness that sweeps sizes/nodes	Numbers with no script; hard-coded to one cluster
An Experiments & Analysis study	The comparison harness that regenerates every table	Only the authors' system tuned; missing configs
A data/workload contribution	The derivation scripts + the derived data	Description shipped, data missing; provenance unpinned

Assume an evaluator gives your package a bounded time budget on a machine that is not yours. Design for the first ten minutes — a clean build and a small demo run — to succeed.

Packaging plan

[Container]   ship a Dockerfile or a pinned environment (build recipe / lockfile); avoid
              "install these 40 things and configure the cluster by hand"
[README]      one-screen orientation: what it is, how to build, how to run a demo, how to reproduce
              each result, expected runtime and outputs, and what needs a big cluster
[Mapping]     an explicit table: paper claim -> script -> expected result (table/figure)
[Workloads]   the derived workload/query-log itself (or a documented access path), with pinned
              dataset versions, not just a query
[Environment] the hardware/cluster spec assumed, and a reduced-scale path a reviewer can actually run
[License]     an OSI-approved code license and clear data terms, compatible with the CC-BY-NC-ND record
[Archive]     deposit in a DOI-issuing repository (Zenodo, figshare, Software Heritage) for the
              permanent, citable record

Review-time package vs. permanent package

• At submission: if the cycle is double-blind (待核实 per cycle), the package must be anonymized — no owner strings, cluster names, lab paths, or identity-revealing links; if single-blind, still scrub stale paths and credentials.
• After acceptance: deposit the de-anonymized, licensed package in a DOI-issuing archive; this is the version the camera-ready cites and the permanent open-access record points to.

Worked vignette: packaging an operator + evaluation

A paper contributes a query-processing operator and a cluster evaluation. To make it re-runnable: ship a container with the engine and operator pre-built; a run_demo.sh that runs the operator on a small bundled workload in under a minute; a reproduce/ directory whose scripts regenerate each table from logged results and, where feasible, from a fresh reduced-scale run; a claim-to-script mapping in the README; the derived workloads with pinned dataset versions; and an Apache/MIT license. State honestly which results are turnkey at small scale and which need the full cluster.

Calibration

• The full-cluster experiments may not be turnkey on an evaluator's machine — provide a reduced-scale path and say clearly what it does and does not reproduce.
• For an Experiments & Analysis paper the artifact stakes are highest: the whole comparison should regenerate, with every system's configuration documented, because the study is the contribution.
• Confirm the current cycle's artifact/reproducibility process, any badge, and its anonymity rules on the host-site call (待核实).

Output format

[Package role] review-time (anonymized?) / permanent DOI-archived
[Contents] <system/build / workloads / harness / provenance / license>
[Ten-minute test] does build + demo succeed on a clean machine? yes/no
[Claim mapping] <claim -> script -> expected result present? yes/no>
[Scale honesty] <what is turnkey small-scale vs. needs a cluster>
[Archive] DOI-issuing repository + compatible license? yes/no
[Fixes before upload] <ordered list>