Experiment planner

Experiment Planner

Overview

Use this skill before paper writing when the user needs to turn a research idea into a testable story and experiment plan. It is an adapter over existing research-agent ideas, not a replacement for the user's writing, review, rebuttal, figure, evidence, or GitHub release skills.

Core Boundaries

• Default domain: general deep learning and computer science research. Adapt to collaborative perception, 3D perception, or autonomous driving only when the task context calls for it.
• For discussion-only planning, keep output in chat unless the user asks for a saved artifact. When the user asks to implement or run experiments in a repository, persist the pre-run result contract: update the paper's final LaTeX tables when a manuscript is in scope; otherwise update the existing experiment-planning document or create experiment-plan.md at the repository root.
• Do not launch long experiments, deploy GPU jobs, modify code, or retry failed runs unless the user explicitly asks for execution.
• Do not replace paper-section-playbook, paper-refinement-skills, paper-review-panel, rebuttal-response-skills, paper-visual-craft, or github-project-release; hand off to them only after the research plan or results are ready.
• Treat external projects as references, not installed dependencies. Read references/source-map.md before discussing provenance or upgrading this skill from upstream sources.

Default Workflow

1. Grill consensus: use $grill-me style interaction to clarify problem, motivation, proposed claim, baseline/control, compute budget, success criteria, and unacceptable shortcuts. Ask one high-impact question at a time when the answer changes the experiment plan.
2. Literature inspiration: after a preliminary consensus, use $research-evidence for related papers, novelty risk, prior experiment patterns, and unsupported claims. Use $search-first when the task may need existing code, datasets, tools, or implementations.
3. Story viability check: decide whether the idea can support a clean paper story: important problem, credible gap, specific method difference, feasible validation, and claims that will not outrun the evidence.
4. Claim freeze: freeze the smallest verifiable claim before planning runs. Avoid changing the story repeatedly while experiments are running.
5. Paper/table contract freeze: before scheduling runs, define the final main-result, ablation, and necessary diagnostic tables. For every metric, record its plain-language definition, unit, direction, aggregation, and any delta reference. Use -- for unavailable values and do not write claims from placeholder cells.
6. Idea validation first: design the smallest pilot/smoke/sanity experiment that can falsify or support the core hypothesis. If multiple GPUs are idle, parallelize only independent exploration runs with clear ownership.
7. Minimum sufficient matrix: only after the pilot passes, add the main result and claim-critical ablations. Add robustness, diagnostics, efficiency, qualitative results, or failure analysis only when they support a paper claim or answer a credible reviewer question; do not add them for symmetry.
8. Subagent coordination: keep the main session responsible for planning, task decomposition, and final result acceptance. Use explorer for read-only repo/config/protocol investigation. Use worker for implementation with explicit file or module ownership. Do not manually override subagent model or reasoning settings unless the user explicitly requests it.
9. Run discipline: test that the command starts and produces plausible small outputs; remove test data after smoke checks; launch the full run only after sanity passes; inspect the first few samples/logs/artifacts; stop continuous monitoring once the run is confirmed healthy unless the user asks otherwise.
10. Conditional seed policy: record an existing seed and keep compared runs under the same evaluation and checkpoint-selection policy. For expensive training such as autonomous driving, accept a single training run by default. Require repeated seeds only when variance could change the central claim, the margin is small, the runs are inexpensive, or the venue requires them.
11. Default decisions: ask only questions whose answers materially change the plan. If a non-critical choice goes unanswered, use the recommended default and record it as an assumption.

Output Contract

Default to a concise in-chat experiment matrix. Before producing a matrix, read references/experiment-matrix.md.

The matrix must include:

• research question
• core hypothesis
• paper claim
• storyline
• literature inspiration
• baseline/control
• table contract
• metric definitions
• idea validation experiment
• expected signal
• failure modes
• diagnostic checks
• follow-up experiments
• subagent/task ownership
• compute/resource assumptions
• seed policy
• success gate
• claim gate
• next action

Use unknown or needs user input for unresolved fields instead of inventing project facts. Keep recommendations executable, but do not perform execution inside this skill unless the user asks for implementation or running commands.

Handoff Rules

• Use $research-evidence before making novelty, citation, or literature coverage claims.
• Use $search-first before proposing new implementation utilities, pipelines, tool integrations, or dataset-processing code.
• Use writing skills only after the experiment story is stable enough to draft a paper section, rebuttal, review, table, or figure.
• For code work, assign worker tasks with disjoint write scopes and remind the worker not to revert others' changes.
• For investigation, assign explorer tasks that are specific, read-only, and non-overlapping with the main session's current work.

Failure Modes To Catch

• The idea is interesting but not falsifiable with available data or compute.
• The proposed contribution is only a presentation change, not a testable method or analysis difference.
• The baseline/control is missing, unfair, or weaker than the claim requires.
• The pilot experiment cannot distinguish mechanism from implementation noise.
• The plan jumps to full benchmark runs before smoke and sanity checks pass.
• The story changes after seeing results without recording a clear reason.
• Subagents receive vague tasks, overlapping write scopes, or authority to run long jobs without main-session acceptance.