Ecai experiments

ECAI Experiments

The ECAI question is always the same: is the evidence proportional to the claim? But ECAI spans symbolic and applied AI, so "evidence" ranges from a proof to a fair empirical comparison. Choosing the right kind of evidence for your claim shape is the first and most important decision.

Choose the evidence type by claim shape

Claim	Primary evidence	Common ECAI failure
"Complete / sound / optimal / (1+ε)-bounded"	A proof, all assumptions explicit	Asserting it empirically only
"More efficient / fewer expansions / faster"	A controlled comparison on standard instances, with spread	One lucky run; unfair baseline tuning
"Learns/generalizes/calibrates better"	Fair comparison + a reason why, seeds, significance	A single benchmark delta with no mechanism
"Handles a broader class / new setting"	A construction/encoding + worked cases	Toy examples only
"Works in the real world"	A credible deployment demonstration (PAIS)	Benchmark abstraction standing in for deployment

A provable claim needs a proof; an empirical claim needs a fair, seeded comparison; a claim about understanding needs an explanation, not just a number.

For theory / KR / planning / argumentation

• Prove it, completely. The body sketches; the supplement carries full proofs (ecai-reproducibility). State every assumption (finiteness, admissibility, language fragment).
• Standard instances for empirical planning/search. Use recognized domains/benchmarks (e.g. the community's standard planning domains) so node/quality numbers are comparable; report per-domain results, not just an aggregate.
• Complexity claims get the reduction or the algorithm, not a hand-wave.

For ML / learning-based contributions

• Fair baselines, fairly tuned. Give the baseline the same tuning budget as your method; a hobbled baseline is the fastest way to lose a reviewer.
• Seeds and spread. Report mean and variance/CI across multiple seeds; a single run is not evidence. State the number of runs.
• Explain the win. ECAI rewards why a method works (an ablation isolating the responsible component, a theoretical reason) over a leaderboard delta.
• Contamination and leakage. For LLM/pretrained components, check train/test overlap and document model identifiers with dates; cache outputs so results reproduce.

For multi-agent contributions

• Specify the environment, agents, episodes, and metrics exactly; multi-agent results are notoriously protocol-sensitive.
• Compare against the right baselines for the setting (cooperative/competitive), and report across seeds and environment variations, not one map.
• If the contribution is fundamentally about agent interaction, sanity-check whether AAMAS is the better-matched pool (ecai-topic-selection).

For applied AI (PAIS)

• Lead with the real-world claim and constraints (data availability, latency, cost, safety), not a benchmark score.
• Show the method survives real conditions; a deployment story that only reports offline accuracy under-delivers on the PAIS bar.

Ablations and honesty

• Ablate the mechanism you credit. If you attribute the gain to component C, remove C and show the drop.
• Report negative and null results where they bound the claim — in a single-round process, self-reported limits cost less than reviewer-discovered ones (ecai-review-process).
• No cherry-picking domains, seeds, or metrics; report the protocol that generated every number.

Fit the 7-page body

Evidence a reviewer needs to judge the claim (the proof idea, the key comparison, the main table) stays in the body; full proofs, extra domains, and ablation grids go to the supplement (ecai-supplementary). Do not exile the decision-critical comparison to save space.

Output format

[Claim -> evidence] each claim mapped to proof / controlled comparison / deployment demo
[Proof completeness] provable claims proved with explicit assumptions? yes/no
[Baseline fairness] baselines tuned comparably? seeds + spread reported?
[Why it works] mechanism explained (ablation/theory), not just a number? yes/no
[Provenance] datasets/models/seeds pinned; outputs cached? gaps: <list>
[Body/supplement] decision-critical evidence inside 7 pages? yes/no