Source discovery

source-discovery skill

What this skill does

Upstream entry of the knowledge pipeline:

/source-discovery → /corpus-ingest → /wiki-build → /wiki-lint → /cleanup

Input: a topic (and optionally scope/source classes). Output: a self-contained raw corpus on disk + a manifest describing every source + a handoff note for /corpus-ingest.

Hard boundary: the skill ends when raw is saved and manifest is written. It does NOT transcribe video, does NOT run OCR on PDFs, does NOT write Karpathy wiki, does NOT produce a final report for the user. Those are downstream skills.

When to invoke

Triggers (user-facing):

• "find materials on X"
• "collect raw sources for the wiki"
• "what's out there about topic Y: YouTube/GitHub/papers/docs"
• "prepare a corpus for /corpus-ingest"
• "deep research on a topic"
• "/source-discovery <topic>"

Do NOT invoke when:

• User asks a direct factual question (just answer)
• User wants a short summary or recommendation (use web_search directly)
• User wants existing corpus analyzed (that's /corpus-ingest or /wiki-build)

Inputs

Required:

• topic — what to research, e.g. "healthy sleep", "n8n best practices"

Optional (ask if unclear, otherwise use defaults):

• scope: quick | standard | thorough (default: standard)
• target artifact: wiki | report | course_map | product_research (default: wiki)
• source classes: subset of {web, github, youtube, papers, docs, social, local} (default: {web, github, youtube})
• language: ru | en | mixed (default: mixed)
• freshness: any | recent_2y | recent_6m (default: any)
• exclude_domains: list of domains to skip

Output contract

Write to the current project root (use cwd). inventory/ and handoff_to_corpus_ingest.md are at the PROJECT ROOT, NOT inside raw/source_discovery/<slug>/. Only notes/, sources/, snapshots/, downloads/ live inside the slug folder.

<project_root>/
  raw/
    source_discovery/
      <YYYY-MM-DD>_<slug>/
        sources/
          web/
          github/
          youtube/
          papers/
          social/
          local/
        snapshots/            — markdown via Jina Reader (one .md per URL)
        downloads/            — PDFs, videos, archives
        notes/                — agent's scratch notes per perspective
  inventory/                  — PROJECT ROOT, not inside raw/<slug>/
    source_discovery_manifest.json
    source_discovery_report.md
    failed_angles.md
  handoff_to_corpus_ingest.md — PROJECT ROOT, not inside raw/<slug>/

<slug> = lowercased topic, spaces→underscores, non-ascii→transliterated, max 40 chars.

Workflow

Execute these phases in order. Each phase has explicit completion criteria. Do not skip phases.

Phase 1 — Clarify scope (skip if obvious)

If topic is ambiguous (e.g. "sleep" could be biology vs tech), ask ONE clarifying question. If clear, proceed silently.

Set <slug> and create the directory tree above.

Phase 2 — Perspective discovery (STORM-inspired)

Goal: find 3-7 distinct viewpoints from which the topic is studied / talked about. This is what separates this skill from naive web search.

Method:

1. web_search for "<topic> overview" and "<topic> wikipedia"
2. From results, extract adjacent subtopics and stakeholder roles
3. Synthesize 3-7 named perspectives, each with a one-line description

Example for "healthy sleep":

• medical — neurology, sleep disorders, insomnia treatment
• circadian — circadian biology, light exposure, chronotypes
• biohacker — sleep tracking, supplements, optimization
• parent_child — child sleep, infant sleep regression
• athlete — sleep for performance, recovery

Write perspectives to notes/perspectives.md before moving on.

Phase 3 — Query plan

For each perspective, draft 3-6 specific search queries. Mix:

• Authority queries: "<topic> <perspective> guidelines", "<perspective> meta-analysis"
• Source-type queries: "<topic> <perspective> github", "<topic> <perspective> youtube interview"
• Contradiction queries: "<topic> <perspective> myths", "<topic> <perspective> critique"

Save to notes/query_plan.md. Estimate total searches: ~3-6 per perspective × N perspectives.

Phase 4 — Breadth search (PARALLEL via subagents)

This is the heavy phase. Spawn one subagent per perspective via the Agent tool, all in a SINGLE assistant message with multiple Task tool blocks — that is what makes them run in parallel. If you dispatch them in separate messages you lose the parallelism and the multi-agent benefit (+90% on breadth-first queries) is gone.

Concrete pattern (illustrative):

[single assistant message containing N Task tool calls]
  Task(perspective=medical_circadian, queries=[...], output_dir=...)
  Task(perspective=biohacker_tracking, queries=[...], output_dir=...)
  Task(perspective=practical_hygiene, queries=[...], output_dir=...)

Each subagent gets a self-contained prompt:

• Its perspective name + description
• Its query list from Phase 3
• The output paths it owns: raw/source_discovery/<...>/sources/<class>/<perspective>__<n>__<slug>.json for metadata and snapshots/<perspective>__<n>__<slug>.md for content
• Strict instruction: "save each found source as JSON metadata + fetch markdown via Jina Reader (curl -sS https://r.jina.ai/<URL>), write directly to disk, do not return source contents to the orchestrator — only return a list of saved file paths + 1-line each"

Subagent template:

You are a research subagent for perspective: <perspective_name>.
Description: <perspective_desc>.

Your queries:
1. <query>
2. <query>
...

For each query:
1. Run web_search
2. Pick 3-8 best results (skip SEO farms, listicles, AI-generated content)
3. For each picked result:
   a. Save metadata as <output_path>/sources/<class>/<perspective>__<idx>__<slug>.json
      Fields: url, title, author_or_org, source_type, tier (1-4), language, why_included, query_that_found_it, fetch_status, snapshot_path
   b. Fetch markdown via: curl -sS --max-time 30 "https://r.jina.ai/<URL>" -o <output_path>/snapshots/<perspective>__<idx>__<slug>.md
      Rate-limit handling: on HTTP 429 sleep 5s and retry, then 10s, then 20s (exponential backoff, max 3 retries).
      Only after 3 failed retries OR if response < 500 chars, fall back to web_fetch.
   c. **SAVE RAW. DO NOT SUMMARIZE.** The snapshot file must contain the actual fetched markdown (Jina output) or actual web_fetch raw text. Do NOT rewrite, condense, reformat, or extract "key points". Do NOT add your own headers, commentary, or "Retrieved via X" preamble. The corpus normalization happens in `/corpus-ingest`. Your job is fetch-and-persist, not synthesis.
   d. Track failures in <output_path>/notes/<perspective>__failed.md (per-perspective fetch failures only — aggregate failed search angles go to `inventory/failed_angles.md` in Phase 8)

Source quality heuristics:
- Tier 1 (primary): official docs, peer-reviewed papers, standards, author/org repos
- Tier 2 (curated): established publications, textbooks, expert blogs with citations
- Tier 3 (community): forums, Reddit, HN, tutorials, video walkthroughs
- Tier 4 (unverified): SEO content, AI-generated, anonymous posts, content mirrors

Skip rules:
- URL contains "best-of-2024", "top-10", "complete-guide-to" without author attribution → likely SEO
- AI-generated detectors not required, but use judgement: if structure is bullet-list-only with no specifics, downgrade to tier 4 or skip
- Duplicates: if title+author already saved, skip

Token budget: stop after <budget> tool calls or <max_sources> sources, whichever first.

Return to orchestrator: ONLY a JSON array of {file_path, title, tier, source_type}. No source contents.

Per-scope budgets:

• quick: 2-3 perspectives, 4-6 queries each, 10-20 sources total, max ~30 tool calls per subagent
• standard: 4-6 perspectives, 5-8 queries each, 30-80 sources total, max ~50 tool calls per subagent
• thorough: 6-10 perspectives, 6-10 queries each, 100+ sources total, max ~80 tool calls per subagent

Phase 5 — Classify and dedupe

After all subagents finish:

1. Read every metadata JSON in sources/
2. Detect duplicates: same canonical URL (strip query params except essential), or same title+author
3. Mark duplicates by adding duplicate_of: <other_id> to JSON metadata
4. Verify tier assignments — spot check 10% randomly

Phase 6 — Knowledge gap reflection (LangChain-inspired)

Read all collected metadata + perspective notes. Ask yourself:

• Are there perspectives barely covered? (< 5 sources)
• Are there contradictions worth investigating? (e.g. one source says X, another says not-X)
• Are there obvious authoritative sources missing? (e.g. for medical topics — no WHO/NIH; for code topics — no official docs)

Write notes/gaps.md with 2-5 explicit gaps. If scope=quick, stop here. If standard or thorough, proceed to Phase 7.

Phase 7 — Depth search (only for standard / thorough)

For each gap from Phase 6, run 1-3 targeted queries. Add results to existing perspective folders, not new ones. Update metadata.

Phase 8 — Persist manifest

Write inventory/source_discovery_manifest.json:

{
  "topic": "<topic>",
  "slug": "<slug>",
  "created_at": "<ISO-date>",
  "scope": "<scope>",
  "perspectives": [
    {"name": "...", "description": "...", "source_count": N}
  ],
  "sources": [
    {
      "id": "<perspective>__<idx>",
      "perspective": "...",
      "title": "...",
      "url": "...",
      "source_type": "official_docs|paper|repo|video|blog|forum|social|other",
      "tier": 1,
      "language": "ru|en|...",
      "local_metadata": "sources/<class>/<file>.json",
      "snapshot_path": "snapshots/<file>.md",
      "download_path": null,
      "status": "saved|metadata_only|failed|deferred",
      "why_included": "...",
      "feeds_perspectives": ["..."],
      "duplicate_of": null,
      "copyright_caveat": null,
      "notes": "..."
    }
  ],
  "failed_angles": [
    {"query": "...", "reason": "..."}
  ],
  "stats": {
    "total_sources": N,
    "by_tier": {"1": ..., "2": ..., "3": ..., "4": ...},
    "by_type": {...},
    "duplicates_removed": N
  }
}

Phase 9 — Handoff note

Write handoff_to_corpus_ingest.md:

# Handoff: source-discovery → corpus-ingest

Topic: <topic>
Date: <date>
Scope: <scope>
Total sources: N (tier1: N, tier2: N, tier3: N, tier4: N)

## What is here

raw/source_discovery/<slug>/
  sources/        — metadata JSONs grouped by class
  snapshots/      — Jina Reader markdown (ready for ingest)
  downloads/      — N PDFs, M videos (need transcription/OCR in ingest)
  notes/          — perspectives, query plan, gaps, per-perspective failures

## Recommended next steps for /corpus-ingest

1. Snapshots: already markdown — copy to text_corpus/ with light cleanup
2. PDFs in downloads/: run through pdf-to-markdown
3. Videos: download + Deepgram transcription
4. Apply text_corpus normalization rules

## Caveats

<list copyright/licensing issues, paywalled content, etc.>

## Known gaps

<reference to notes/gaps.md>

Phase 10 — User report (concise)

Output to user (chat) only:

• Topic + scope + slug
• Total sources + breakdown by tier
• Top 5 most valuable sources (tier 1, one line each)
• 2-3 main gaps
• Path to manifest + handoff
• Next command: /corpus-ingest <slug>

DO NOT dump source contents to chat. DO NOT recap full perspective lists.

Quality gates (check before reporting done)

• Every source in manifest has: url, title, source_type, tier, status, why_included
• Failed search angles disclosed in failed_angles.md and manifest
• Duplicates marked (not deleted — kept for audit)
• Social/community sources marked tier 3 or 4
• Paid/copyrighted content has copyright_caveat set
• No fabricated URLs — every URL in manifest was actually visited or attempted
• handoff_to_corpus_ingest.md exists and is non-empty
• No wiki, no topic nodes, no synthesis report has been written (those belong downstream)

Anti-stuck rules (jina-inspired)

• Hard cap: stop subagent after its tool-call budget regardless of progress
• If 3 consecutive searches return zero relevant results, disable that perspective and note in failed_angles
• Beast Mode fallback: if entire phase 4 produces < 5 sources, expand scope by one tier (quick → standard) and retry once, with explicit note in report

What this skill must NOT do

• Build Karpathy wiki (/wiki-build does that)
• Transcribe audio/video (/corpus-ingest does that)
• OCR scanned PDFs (/corpus-ingest does that)
• Write a polished research report (out of scope — use /longread if needed)
• Make purchasing decisions (e.g. don't recommend paid courses)
• Fabricate sources to fill the manifest
• Return source contents to the orchestrator's context (always write to disk)
• Pre-summarize, rewrite, or reformat snapshots. Snapshots are raw fetch output. No "key points" extraction, no own headers, no "Retrieved via X" preamble. That's downstream work.
• Duplicate files across inventory/ and notes/. Each artifact has exactly ONE canonical location:
  • inventory/source_discovery_manifest.json — the manifest (only here)
  • inventory/source_discovery_report.md — final report (only here)
  • inventory/failed_angles.md — aggregated failed search angles (only here)
  • notes/perspectives.md, notes/query_plan.md, notes/gaps.md — scratch (only here)
  • notes/<perspective>__failed.md — per-perspective fetch failures (only here) Do not copy any of these between inventory/ and notes/.

Tool usage cheatsheet

• web_search — primary discovery
• web_fetch — fallback when Jina fails
• Agent (Task tool) — parallel subagents per perspective
• Bash with curl -sS https://r.jina.ai/<URL> — clean markdown of any URL, free, no key
• Write — manifest, handoff, notes
• Bash with yt-dlp --skip-download --write-info-json — YouTube metadata only. If yt-dlp is missing, do NOT install it automatically. Ask for confirmation with the exact command, or skip YouTube downloads and save a metadata-only stub.

Notes for the orchestrator

You are the orchestrator. Your job is to plan, dispatch subagents, then aggregate metadata they wrote to disk. Do not run web_search yourself in Phase 4 — that's subagent work. After subagents return path-lists, read only the JSON metadata you need to build the manifest. Never load snapshot contents into your context.