Iros experiments

IROS Experiments

Use this before submission when the empirical story is not yet locked. At IROS, experiments exist to prove a robot did something reliably, under stated conditions — not to top a leaderboard.

Experiment audit

• Map each claim to the evidence that supports it: a real-robot trial set, a simulation study, an ablation, or a field deployment. A claim with no matched evidence is an overclaim.
• Define the success criterion before running, in one sentence a skeptic would accept, and hold to it; a criterion invented after seeing results is a reviewer red flag.
• Report trial counts and resets: how many attempts, how start conditions were randomized, and what reset happened between trials. Hidden resets inflate reliability.
• Publish a failure taxonomy with counts, not just a success rate; the failures are where reviewers calibrate trust.
• Make baselines fair: run them on the same platform, sensors, and route, with comparable tuning effort, or state precisely why not.
• Separate simulation from real and report the sim-to-real gap as a number; an implied zero gap is the fastest way to lose a reviewer.

The evidence ladder

Claim altitude	Evidence IROS expects	Reject pattern avoided
"The system works"	Trials with n, success interval, and resets stated	"One hero run shown as if typical"
"It is reliable"	Failure taxonomy with counts across conditions	"Success rate with no failures reported"
"It transfers"	Real-robot numbers plus the measured sim-to-real gap	"Sim results implying real performance"
"It beats prior work"	Same-hardware baseline on the same task	"Comparison against a weaker or re-tuned baseline"
"It runs onboard"	Measured rate and power under the real compute budget	"Real-time asserted, never measured"

Small-n statistics for real robots

Real trials are expensive, so n is small — but small n does not excuse a bare mean. Report a success count as a proportion with a confidence interval (a Wilson interval behaves better than normal approximation at small n), and for paired system-vs-baseline comparisons on the same trials, prefer a paired test over independent means. State n every time; "usually succeeds" is not a measurement.

Vignette: a manipulation reliability study

Suppose the system claims reliable grasping of unseen objects. The matching plan: fix an object set and a success criterion (lifted and held 3 seconds), run a stated number of trials per object with randomized poses, log every failure by cause (slip, mis-localization, collision), and report the per-object and pooled success rates with intervals. A simulation sweep over object mass then maps where the grasp model degrades, and the real-vs-sim gap is stated — every panel tied to a specific claim.

Trial-logging template (one row per attempt):
  trial_id, object/scenario, start_pose_seed, outcome{success|fail},
  failure_cause, reset_type, wall_time, notes
Aggregate to: success rate + interval per condition, failure histogram, sim-to-real delta.

Reporting floor

• Every reliability figure carries n and a criterion; every timing claim carries a measured rate and the compute/power budget it ran under.
• Report the compute actually consumed on the robot, not a desktop proxy.

Output format

[Experiment readiness] strong / adequate / weak
[Claim -> evidence map] <claim: trials/sim/ablation/deployment>
[Missing evidence] <trials/resets/failures/baseline/transfer>
[Statistics] criterion set? interval reported? paired where paired?
[Decision-critical next run] <one experiment on the robot>