Idea spark agentic

Idea Spark - Agentic Skill

Convert an under-specified research direction into ONE reviewer-defensible Oral-level research proposal via a 5-phase workflow. This is the agentic variant: Phase 0 literature grounding and Phase 3.1 collision checking are performed by isolated Claude Code sub-agents using WebSearch/WebFetch, so no .env, OpenReview, Semantic Scholar, OpenAlex, or arXiv API credentials are needed.

Everything else (ideation patterns, sub-patterns, Phase 1-4 prompts, validators, card renderers) is contract-compatible with the original idea-spark skill.

When to use

• Same as idea-spark: "Give me a research idea in {area}", "What's the bottleneck?", "Help me sharpen this direction into an Oral-level proposal."
• Prefer this variant when you do not have connector API keys or want the retrieval agent to reason about relevance, year windows, and cross-community aliases rather than relying on BM25 query matching.

When NOT to use

• Same as idea-spark: code review, pure summarization, cross-decade surveys, unconstrained brainstorming, engineering-integration tasks.
• Do not use if you require deterministic, reproducible connector-backed retrieval (use legacy idea-spark instead).

Setup (first use only)

1. Copy this directory into your Claude Code skills folder — <project>/.claude/skills/idea-spark-agentic for one project, or ~/.claude/skills/idea-spark-agentic for all projects — and make sure the WebSearch, WebFetch, and Agent tools are permitted.
2. No .env is required.
3. Optional: install pymupdf and beautifulsoup4 for full-text fetch quality (otherwise fulltext falls back to abstracts):

   pip install pymupdf beautifulsoup4
   

4. Optional: install xelatex or tectonic if you want PDF idea cards in addition to Markdown.

How to run: the next loop

The canonical driver is the run-state navigator:

python3 "$SKILL_DIR/scripts/run.py" next --dir "$RUN_DIR" --query "<user's research question>"

Run-dir convention (same as idea-spark): $PWD/ideaspark_run/<topic-slug>. One run = one directory. Never reuse a directory that already contains a phase0/.

The host loop is:

1. Run next.
2. Do what it says (bash → run the command; llm_subagent → spawn an isolated sub-agent with ONLY the listed file paths).
3. Run next again. Repeat until terminal state (DONE, do_not_generate, or phase_3_failed).

Use this checklist:

• Phase 0: deterministic setup → agentic retrieval sub-agent → lit_results.json + lit_table.md + .lit_grounding_mode
• Phase 0+: full-text fetch (phase0_fulltext) - MANDATORY before Phase 1
• Phase 1: bottleneck identification → proceed or do_not_generate
• Phase 2: gap×pattern selection + candidate generation (ONE sub-agent, TWO outputs) → citation gate → coherence gate
• Phase 3: collision retrieval setup → agentic collision sub-agent → assemble → audit (5 checks) → revise/merge/re-audit or abandon→retry
• Phase 4: skeleton → fill → assemble → implementability audit → validate → render → return 3 cards inline

Context discipline

A full run accumulates ~180-250k tokens of intermediate state. Run every LLM-driven phase in an isolated context (sub-agent or subprocess). Write every phase artifact directly to disk; never paraphrase large JSON into chat. Compact the parent context between phases. See idea-spark design-notes for the rationale.

Phase reference

Orchestrator entry points

Phase	Entry point
navigator	next --dir "$RUN_DIR" [--query "..."]
Phase 0 setup	phase0 --query "<user text>" --out $RUN_DIR/phase0/
Phase 0+ full-text (mandatory)	phase0_fulltext --out $RUN_DIR/phase0/
Phase 1 anchor top-up	phase1_fulltext_topup --out $RUN_DIR/phase0/ --paper-id <anchor paper_id>
Phase 3.1 setup	phase3_collision --idea-json <canonical candidate> --out $RUN_DIR/phase3_collision/
Phase 3.1 assembly	phase3_collision_assemble --sig .sig_raw.json --alias .alias_raw.json --out $RUN_DIR/phase3_collision/
Phase 3.3 merger	phase3_merge_revisions --phase2 <canonical> --revisions <p3.3-patch> --critique <p3.2-report> --out $RUN_DIR/phase3_revise/
Phase 2.3 merger	phase3_merge_revisions --phase2 <p2.2-output> --revisions <p2.3-output> --out $RUN_DIR/phase2_coherence/ --out-name refined_candidate.json
Phase 4 skeleton	phase4_skeleton --candidate <final_candidate-or-p2.2> --phase1 ... --phase2-select ... --phase3-critique ... [--phase3-revise ...] --phase0-dir $RUN_DIR/phase0/ [--collision ...] --out $RUN_DIR/phase4/
Phase 4 assemble	phase4_assemble --skeleton $RUN_DIR/phase4/phase4_skeleton.json --fill-map $RUN_DIR/phase4/fill_map.json --out $RUN_DIR/phase4/
Phase 4 render	phase4_render --expansion $RUN_DIR/phase4/phase4_expansion.json --out $RUN_DIR/phase4/
Validators	validate --phase2 ... [--phase3 ...] [--phase4 ...] [--phase4-impl ...]

The LLM-driven phases (1 / 2.1 / 2.2 / 2.3 / 3.2 / 3.3 / 4.fill / 4.1.5 / falsification re-audit) have no orchestrator subcommand: read the prompt at references/system-prompts/<phase>.txt, gather the inputs listed at its top, and Write the JSON described under Output: to $RUN_DIR/<phase>/<phase>_output.json.

Phase 0 - Agentic literature grounding

Phase 0 is split into a deterministic setup step and an LLM sub-agent step.

Step 0a - setup (Bash):

python3 "$SKILL_DIR/scripts/run.py" phase0 --query "<user's research question>" --out "$RUN_DIR/phase0/"

This writes:

• $RUN_DIR/phase0/query.txt - the raw user query.
• $RUN_DIR/phase0/user_refs.json - URL/ID references extracted from the query.

Step 0b - agentic retrieval (sub-agent):

Spawn an isolated sub-agent with references/agentic-retrieval-prompt.md as its system prompt. Pass ONLY these file paths:

• $RUN_DIR/phase0/query.txt
• $RUN_DIR/phase0/user_refs.json
• references/intent-recognition.md
• references/pattern-summary-rubric.md
• references/schemas.md
• references/intake-routing.md

The sub-agent:

1. Checks intake-routing.md OOD triggers. If OOD, writes $RUN_DIR/do_not_generate.md and stops.
2. Derives 4-6 search queries per intent-recognition.md Map mode (including one ESCAPE-MECHANISM query in solution vocabulary).
3. Runs WebSearch for each query with year filters, then WebFetch on promising URLs.
4. Deduplicates to ~30-50 unique papers.
5. Tags each paper with 1-3 ideation patterns per pattern-summary-rubric.md.
6. Writes:
   • $RUN_DIR/phase0/lit_results.json - deduplicated paper records (retrieved_via: websearch, source: websearch).
   • $RUN_DIR/phase0/lit_table.md - exact 9-column table required by Phase 1.
   • $RUN_DIR/phase0/.lit_grounding_mode - sentinel containing agentic.

Step 0c - full-text fetch (Bash, mandatory):

python3 "$SKILL_DIR/scripts/run.py" phase0_fulltext --out "$RUN_DIR/phase0/"

Reuses scripts/fetch_sections.py (arXiv HTML first, then PDF fallbacks). No API keys. Phase 1 hard-gates on fulltext_cache.json.

Phase 1 - Bottleneck identification

One isolated LLM call. Prompt: references/system-prompts/bottleneck_identify.txt (which now accepts .lit_grounding_mode == agentic). Inputs: user query + intake, phase0/lit_table.md, phase0/fulltext_cache.json, phase0/lit_results.json. Output phase1/phase1_output.json with state ∈ {proceed, do_not_generate}.

Phase 2 - Selection + generation

Identical to idea-spark. Run 2.1 and 2.2 back-to-back in ONE isolated sub-agent, writing BOTH phase2_select_output.json and phase2_generate_output.json. Then run the deterministic citation gate (validate --phase2 ...), then the coherence gate (2.3, fresh context).

Phase 3 - Quality gauntlet

3.1 collision - agentic (setup + sub-agent + assembly):

Setup:

python3 "$SKILL_DIR/scripts/run.py" phase3_collision --idea-json "$RUN_DIR/phase2_coherence/refined_candidate.json" --out "$RUN_DIR/phase3_collision/"

This writes candidate.json and .agentic_collision_pending.

Sub-agent: spawn with references/agentic-collision-prompt.md. Inputs:

• $RUN_DIR/phase3_collision/candidate.json
• references/intent-recognition.md
• references/schemas.md

The sub-agent runs WebSearch for signature_terms[] (last ~10 months) and alias_terms[] (last ~48 months), WebFetch promising URLs, deduplicates per channel, and writes .sig_raw.json + .alias_raw.json.

Assembly:

python3 "$SKILL_DIR/scripts/run.py" phase3_collision_assemble --sig "$RUN_DIR/phase3_collision/.sig_raw.json" --alias "$RUN_DIR/phase3_collision/.alias_raw.json" --out "$RUN_DIR/phase3_collision/"

This scores relevance, truncates per channel to ≤120 hits, and writes collision_hits.json, collision_hits.full.json, and .lit_grounding_mode = agentic.

3.2 audit, 3.3 revise, retry, falsification re-audit: identical to idea-spark.

Phase 4 - Expansion + packaging

Identical to idea-spark: skeleton → fill → assemble → implementability audit → validate → render.

Validators

python3 "$SKILL_DIR/scripts/run.py" validate \
  --phase2 <canonical candidate file> \
  --phase3 <phase3_critique_output.json or phase3_revise_output.json> \
  --phase4 $RUN_DIR/phase4/phase4_expansion.json \
  --phase4-impl $RUN_DIR/phase4/phase4_implementability.json

Same validators as idea-spark. Retry budget cap = 2.

Configuration

• IDEASPARK_DEFAULT_COMPUTE - optional standing compute profile, read from the shell environment (not .env). Precedence: user query > this env var > factory default.

Output surface policy

Same as idea-spark: no calendar projections, no experiment matrix / ablation plan / baseline table. The skill produces IDEA + falsifiability + feasibility judgment.