Web scraper

Web Scraper

./scrape <command> [options] — Run ./scrape help for full option reference.

Content Precision Levels

The fetch command outputs filtered content by default. Three precision levels:

Level	Flag	Behavior	When to use
Default	(none)	PruningContentFilter removes boilerplate	Most scenarios — good enough
Precision	--selector ".css"	Extract only the matched container	Production-quality docs, zero noise
Raw	--raw	Full page, no filtering	Debugging, page structure analysis

Precision workflow: fetch a sample page → inspect remaining noise → identify content container via CSS selector → re-fetch all pages with --selector.

Engine Architecture

L0: Pure HTTP (httpx + selectolax + markdownify)
    fetch → strip boilerplate → markdownify → clean content

L1: Browser (crawl4ai + Playwright, networkidle wait)
    fetch → [JS interaction] → PruningContentFilter → fit_markdown → clean content
    Handles: SPAs, anti-bot, JS-rendered content, folded/lazy content

Auto mode: L0 probe → detect SPA/empty → fallback to L1

Smart Discovery (3-tier):
    1. sitemap.xml → 2. Nav DOM extraction → 3. Browser deep crawl

Workflow Patterns

Single page fetch

# Auto-detects if browser needed
./scrape fetch https://docs.example.com/api/create

# Force browser for known SPA sites
./scrape fetch https://open.feishu.cn/document/... --engine cdp

# Precision extraction with CSS selector
./scrape fetch https://developer.work.weixin.qq.com/document/path/90196 --engine cdp --selector ".ep-doc-area"

Interactive content extraction (SPA/folded content)

# Auto-expand all collapsed/folded sections (universal preset)
./scrape fetch https://open.feishu.cn/document/... --expand-all --wait 5000

# Scroll through entire page to trigger lazy loading
./scrape fetch https://example.com/infinite-scroll --scroll-full

# Custom JavaScript before extraction
./scrape fetch URL --js "document.querySelectorAll('.expand-btn').forEach(e => e.click())"

# JS from file (for complex interactions)
./scrape fetch URL --js-file /path/to/interact.js --wait 5000

# Wait for specific element before extraction
./scrape fetch URL --wait-for ".api-response-table"

# Combine: expand + custom JS + wait
./scrape fetch URL --expand-all --js "extraAction()" --wait-for ".loaded" -o /tmp/docs/

Execute JavaScript on a page

# Execute JS and return metadata (no markdown conversion)
./scrape exec URL --js "return document.title"

# Extract data from page's JavaScript state
./scrape exec URL --js "return JSON.stringify(window.__NEXT_DATA__)"

# Execute JS then fetch page as markdown
./scrape exec URL --js "expandAll()" --then-fetch -o /tmp/result.md

Batch-download documentation

./scrape fetch https://docs.example.com --auto -o /tmp/docs/
./scrape fetch --from-file urls.txt -o /tmp/docs/
./scrape fetch --from-file urls.txt --merge -o /tmp/docs/all.md

Files are automatically named by page title (e.g., 读取成员.md, Getting_Started.md).

Build organized local documentation

1. Discover site structure: ./scrape discover <url> --json → get URLs + titles
2. Filter relevant URLs (Agent selects subset based on user's needs)
3. Write filtered URLs to a file
4. Fetch to output directory: ./scrape fetch --from-file urls.txt -o /path/to/docs/
5. Organize (Agent moves/renames files into logical folder structure)

Analyze site structure

./scrape discover https://docs.example.com
./scrape discover https://docs.example.com --engine cdp --deep --max-pages 100

Extract OpenAPI specs

./scrape openapi https://api.example.com/v3/api-docs -o /tmp/api.md
./scrape openapi https://api.example.com/swagger-ui/ -o /tmp/api.md

Key Options

Option	Commands	Description
--engine MODE	discover, fetch	auto (default), http (L0 only), cdp (L1 only)
--selector CSS	fetch, exec	CSS selector for content area (precision mode)
--raw	fetch, exec	Output full page (skip content filtering)
--js CODE	fetch, exec	JavaScript to execute before extraction
--js-file PATH	fetch, exec	JavaScript file to execute before extraction
--wait-for CSS	fetch	Wait for CSS selector to appear before extraction
--expand-all	fetch	Auto-expand collapsed/folded sections (universal preset)
--scroll-full	fetch	Scroll entire page to trigger lazy loading
--then-fetch	exec	After JS execution, also fetch page as markdown
--session ID	exec	Reuse browser session for multi-step interactions
--js-only	exec	Skip navigation, execute JS on existing session
--auto	fetch	Auto-discover pages then fetch all
--from-file FILE	fetch	Read URLs from file (one per line)
-o PATH	fetch, exec, openapi	Output directory or file path
--merge	fetch	Merge all pages into single markdown
--json	discover, fetch	Machine-readable JSON output
--summary-only	fetch	Only show URL, title, char count
--max-pages N	discover, fetch	Limit pages (discover: 200, fetch: 50)
--deep	discover	Follow links for browser-based BFS crawl

Requirements

• uv: curl -LsSf https://astral.sh/uv/install.sh | sh
• Dependencies and Playwright Chromium auto-installed on first run