Use firecrawl

Use Firecrawl (self-hosted, OpenRouter-backed, any harness)

Stand up a local Firecrawl instance in Docker, wire its LLM + embeddings to OpenRouter (not OpenAI), verify Docker can mount everything before committing, and connect the result to whatever harness the agent is running in. Zero recurring cost to Firecrawl cloud; you pay only OpenRouter per-token for LLM features.

Defaults (do not change unless the user asks):

Component	Model	Cost
LLM (JSON extraction, summary)	deepseek/deepseek-v4-pro	~$0.14/M tokens
Embeddings (semantic search, changeTracking)	qwen/qwen3-embedding-8b	$0.01/M tokens
Provider endpoint	https://openrouter.ai/api/v1	OpenAI-compatible

The user only provides their OpenRouter API key. Everything else is baked.

When to use

• User says "spin a local Firecrawl", "self-host Firecrawl", "give me web crawling", "set up web_search locally", "replace Firecrawl cloud"
• User wants web crawling/scraping in any harness without paying Firecrawl SaaS
• User has Docker + an OpenRouter key and wants it wired

Do NOT use for:

• Firecrawl cloud (managed) — that's just an API key, no Docker
• General self-hosting of other tools — this is Firecrawl-specific
• A throwaway POC — use spike instead

What the user needs before starting

1. Docker running (OrbStack, Docker Desktop, or colima). The skill verifies this in the preflight step — do not assume.
2. An OpenRouter API key — get one at https://openrouter.ai/keys (sign in with Google/GitHub, add credits, create key). Starts with sk-or-v1-.
3. ~5GB free RAM for the containers (Mac 16GB min, 24GB comfortable).
4. ~10 min for the first build (Rust + Playwright Chromium download).

If any are missing, tell the user honestly and stop — don't improvise.

The plan (9 steps, in order)

1. ask-key → 2. detect-harness → 3. preflight → 4. clone → 5. env
                                                          ↓
9. wire-harness ← 8. verify ← 7. smoke ← 6. up ←─────────

Step 1 — Ask for the OpenRouter key

Use clarify with a single open-ended question:

"Pégame tu OpenRouter API key (la consigues en https://openrouter.ai/keys — empieza con sk-or-v1-). La voy a escribir en el .env del repo local, no la voy a commitear."

Do NOT proceed without a real key. If the user asks "donde consigo la key", point them to https://openrouter.ai/keys and stop. Do not fabricate a key.

Step 2 — Detect the harness

The skill must identify which harness it's running in to know where to write the wiring (step 9). Detect in this order — first hit wins:

Signal	Harness	Wiring target
~/.hermes/.env exists OR HERMES_HOME env var set	Desktop agent (Hermes)	~/.hermes/.env + restart desktop
pi command available OR ~/.pi/ dir exists	Coding agent extension (Pi)	Shell env + extension install
~/.config/opencode/ dir exists OR OPENCODE_CONFIG_DIR set	MCP-based agent	MCP config JSON + restart
None of the above	Generic shell	Shell env vars in ~/.bashrc / ~/.zshrc

Detection commands (run silently, don't spam the user):

test -f ~/.hermes/.env && echo "desktop-agent" || \
  (command -v pi >/dev/null 2>&1 || test -d ~/.pi) && echo "extension-agent" || \
  (test -d ~/.config/opencode || test -n "$OPENCODE_CONFIG_DIR") && echo "mcp-agent" || \
  echo "generic"

If multiple harnesses are detected (e.g. both Hermes and Pi installed), ask the user which one to wire via clarify. Do not guess.

Step 3 — Preflight: verify Docker can mount everything

Run these checks before cloning. A failure here saves a 10-min build.

# (a) Docker engine alive
docker ps >/dev/null 2>&1 && echo "docker: up" || echo "docker: DOWN — start OrbStack/Docker Desktop/colima"

# (b) Available RAM (need ~5GB free for 6 containers)
sysctl hw.memsize 2>/dev/null | awk '{print "RAM:", int($2/1024/1024/1024)"GB"}' || \
  free -h 2>/dev/null | awk '/^Mem:/{print "RAM:", $2}'

# (c) Port 3002 free
lsof -nP -iTCP:3002 -sTCP:LISTEN 2>/dev/null | head -2 || echo "port 3002: free"

# (d) Disk space (need ~2GB for images + build)
df -h . | awk 'NR==2{print "disk free:", $4}'

# (e) OpenRouter key reachable (validates the key format + provider up)
curl -sS --max-time 10 -o /dev/null -w "openrouter: HTTP %{http_code}\n" \
  "https://openrouter.ai/api/v1/models" 2>/dev/null || echo "openrouter: unreachable"

Abort conditions (tell the user honestly, don't try to fix):

Failure	What to tell the user
Docker not running	"Start OrbStack / Docker Desktop / colima and run docker ps to confirm, then try again."
RAM < 8GB	"This needs ~5GB for containers. Your machine has <8GB. Close other apps or use a beefier host."
Port 3002 in use	"Port 3002 is taken by another process. Stop it or edit PORT in the .env."
OpenRouter unreachable	"Can't reach OpenRouter. Check your internet connection or the key validity."

If all preflight checks pass, proceed to step 4.

Step 4 — Clone the Firecrawl repo

INSTALL_DIR="${INSTALL_DIR:-$HOME/Documents/Projects/firecrawl-local}"
git clone --depth 1 https://github.com/firecrawl/firecrawl.git "$INSTALL_DIR"

• Use --depth 1 (81MB, no history). Do NOT use sparse-checkout — the monorepo's apps/api/Dockerfile needs the full build context.
• If the directory already exists, ask the user: reuse or reclone? Don't rm -rf without confirmation.

Step 5 — Write the .env

Copy templates/dot-env.template from this skill to $INSTALL_DIR/.env. Replace __OPENROUTER_KEY__ with the real key from step 1.

Key fields (do not change unless the user asks):

OPENROUTER_API_KEY=<user's real key>
OPENAI_API_KEY=${OPENROUTER_API_KEY}
OPENAI_BASE_URL=https://openrouter.ai/api/v1
MODEL_NAME=deepseek/deepseek-v4-pro
MODEL_EMBEDDING_NAME=qwen/qwen3-embedding-8b
USE_DB_AUTHENTICATION=false
TEST_API_KEY=fc-local-dev-key

Security check after writing:

grep "^\.env$" "$INSTALL_DIR/.gitignore"   # must return ".env"
git -C "$INSTALL_DIR" status --short        # must NOT show .env

If .env is NOT in .gitignore, add it manually before proceeding.

Step 6 — Write the Mac-friendly override

Copy templates/docker-compose.override.yaml from this skill to $INSTALL_DIR/. This reduces the default resource requests (8GB api + 4GB playwright) to MacBook-friendly values (4GB api + 2GB playwright).

Pre-flight validate (must pass before up):

cd "$INSTALL_DIR"
docker compose --env-file .env config --services
# expect 7 services: api, playwright-service, redis, rabbitmq, nuq-postgres, foundationdb, foundationdb-init
docker compose --env-file .env config | grep "MODEL_EMBEDDING_NAME"
# expect: MODEL_EMBEDDING_NAME: qwen/qwen3-embedding-8b

Step 7 — docker compose up -d --build + smoke test

First build: 5–10 min (Rust toolchain + Playwright Chromium + Go shared libs). Use background execution and poll manually — do NOT block the session with a 600s foreground timeout.

cd "$INSTALL_DIR"
docker compose --env-file .env up -d --build

After build, docker ps should show 6 containers Up (foundationdb-init exits after setup — correct, it's a one-shot). Wait 30s after Up before probing — the api container builds its DI graph and connects workers.

Smoke test — 3 tests, in order:

Test 1 — No-LLM scrape (regression baseline, <1s):

curl -sS --max-time 15 -X POST "http://localhost:3002/v2/scrape" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer fc-local-dev-key" \
  -d '{"url":"https://example.com","formats":["markdown"]}'

Test 2 — LLM JSON extraction (validates OpenRouter + Deepseek, 5–15s):

curl -sS --max-time 60 -X POST "http://localhost:3002/v2/scrape" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer fc-local-dev-key" \
  -d '{"url":"https://example.com","formats":["markdown",{"type":"json","prompt":"Return a JSON object with: title (string), purpose (one sentence), isIana (boolean)"}]}'

Test 3 — Embedding search (validates Qwen3 embeddings, 2–5s):

curl -sS --max-time 60 -X POST "http://localhost:3002/v2/search" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer fc-local-dev-key" \
  -d '{"query":"python web scraping library","limit":3}'

Step 8 — Verify (formal evidence)

Copy scripts/verify.sh from this skill to a mktemp -t hermes-verify-firecrawl-XXXXXX.sh path, execute, and delete after. The script asserts 25 checks across 7 sections. A 25/0 pass is the evidence.

SCRIPT=$(mktemp -t hermes-verify-firecrawl-XXXXXX.sh)
cp <skill-dir>/scripts/verify.sh "$SCRIPT"
bash "$SCRIPT" "$INSTALL_DIR"
rm -f "$SCRIPT"

Report the pass/fail numbers to the user. If any fail, check the pitfalls section below — most failures are env-var mismatches, not real bugs.

Step 9 — Wire the harness (harness-specific)

This is the step that changes per harness. Use the detection from step 2.

The universal contract is two env vars:

Env var	Value
FIRECRAWL_API_URL	http://localhost:3002
FIRECRAWL_API_KEY	fc-local-dev-key

Every Firecrawl client (Hermes plugin, pi-firecrawl, firecrawl-mcp, firecrawl-py) reads these two vars. What changes is where they go and how the harness picks them up.

Desktop agent (Hermes)

# Append to ~/.hermes/.env
echo '' >> ~/.hermes/.env
echo '# Self-hosted Firecrawl' >> ~/.hermes/.env
echo 'FIRECRAWL_API_URL=http://localhost:3002' >> ~/.hermes/.env
echo 'FIRECRAWL_API_KEY=fc-local-dev-key' >> ~/.hermes/.env

Restart: Cmd-Q the desktop app → reopen. The plugin re-initializes on boot. You CANNOT restart the desktop from inside it without killing the session — tell the user honestly to do it manually.

Verify post-restart: trigger the web extract or web search tool in a fresh chat. If it returns markdown / results, it's wired. If it says "Web tools not configured", the env wasn't read — check the .env syntax.

Coding agent extension (Pi)

Uses the Firecrawl extension which reads the same env vars.

# 1. Install the extension (if not already installed)
pi install npm:pi-firecrawl

# 2. Set the env vars in the shell the agent launches from
# Add to ~/.zshrc or ~/.bashrc:
echo '' >> ~/.zshrc
echo '# Self-hosted Firecrawl' >> ~/.zshrc
echo 'export FIRECRAWL_API_URL=http://localhost:3002' >> ~/.zshrc
echo 'export FIRECRAWL_API_KEY=fc-local-dev-key' >> ~/.zshrc

# OR set the key inside the agent (persists across sessions):
# /firecrawl key fc-local-dev-key

Restart: restart the agent session (exit + relaunch). The extension reads FIRECRAWL_API_URL from the environment on startup.

Verify post-restart: use the scrape tool inside the agent. If it returns markdown, it's wired.

MCP-based agent

Uses an MCP server config. The firecrawl-mcp npm package reads FIRECRAWL_API_KEY and FIRECRAWL_API_URL from its environment block.

# Read existing config (or start fresh)
CONFIG_FILE="$HOME/.config/opencode/config.json"
mkdir -p "$(dirname "$CONFIG_FILE")"

# Write the MCP block (merge if config already exists)
cat > "$CONFIG_FILE" << 'JSON'
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "firecrawl": {
      "type": "local",
      "command": ["npx", "-y", "firecrawl-mcp"],
      "environment": {
        "FIRECRAWL_API_KEY": "fc-local-dev-key",
        "FIRECRAWL_API_URL": "http://localhost:3002"
      }
    }
  }
}
JSON

Restart: restart the agent. Verify with /mcp — Firecrawl should be listed as attached. Then ask the agent to scrape a URL.

Generic (shell env vars)

For any other harness that reads standard env vars:

SHELL_RC="$([ -f ~/.zshrc ] && echo ~/.zshrc || echo ~/.bashrc)"
echo '' >> "$SHELL_RC"
echo '# Self-hosted Firecrawl' >> "$SHELL_RC"
echo 'export FIRECRAWL_API_URL=http://localhost:3002' >> "$SHELL_RC"
echo 'export FIRECRAWL_API_KEY=fc-local-dev-key' >> "$SHELL_RC"
source "$SHELL_RC"

Tell the user to restart their agent/IDE so the new env vars are picked up.

Operating the instance

# Start (after first build, ~10s)
cd ~/Documents/Projects/firecrawl-local && docker compose --env-file .env up -d

# Stop (keeps data)
docker compose down

# Stop + delete all data (nuclear — don't do this casually)
docker compose down -v

# View logs
docker compose logs -f api

# Restart only the api (after .env change, ~15s)
docker compose --env-file .env up -d --force-recreate --no-deps api

What each feature costs (OpenRouter per-token)

Feature	LLM call?	Embedding call?	~Cost per call
scrape markdown only	No	No	$0
scrape with jsonOptions	Yes	No	~$0.001–0.01
search	Maybe	Yes	~$0.001
changeTracking	No	Yes	~$0.001
agent (beta, may be gated)	Yes	Yes	~$0.01

For 50 pages scraped with JSON extraction: ~$0.05–$0.50. Embeddings add ~$0.001. LLM dominates cost; embeddings are 1–2 orders of magnitude cheaper.

Common pitfalls

1. docker ps says Up but API doesn't respond. The container is still warming up. Wait 30s and probe with curl --max-time 5 http://localhost:3002/.

2. LLM call returns 401 Missing Authentication header. The .env was not re-read by the running container. Fix: docker compose --env-file .env up -d --force-recreate --no-deps api (not just restart).

3. /v2/agent returns Agent beta is not enabled. Product gate in self-hosted builds. Not fixable without upstream changes.

4. changeTracking returns warning: Comparing failed. First scrape has no baseline. Scrape the same URL twice — the second call will diff. Not an error.

5. RabbitMQ logs show Cannot get a message from queue ... noproc. nproc is a Unix System V tool missing on macOS. Firecrawl falls back to Postgres. Not blocking — ignore.

6. OpenRouter /v1/models doesn't list embedding models. OpenRouter has 14 embedding models under a different filter. qwen/qwen3-embedding-8b is addressable even if not listed. Test the call, don't trust the listing.

7. Sparse git clone fails on the monorepo. Use full --depth 1 clone (81MB). Don't be clever with sparse-checkout.

8. .env not in .gitignore. The upstream .gitignore should exclude it. If not, add it manually before any git add. Never commit the key.

9. Hermes web_search still says "not configured" after wiring. The desktop process has env vars baked in at boot. A chat reload is not enough — the app must be fully restarted (Cmd-Q + reopen).

10. Extension agent doesn't pick up the Firecrawl URL. The firecrawl extension reads FIRECRAWL_API_URL from the process environment, not from a config file. Make sure the env var is exported in the shell that launches the agent, not just in a subshell.

11. MCP agent server not attaching. Run the agent's doctor command to inspect MCP load errors. The firecrawl-mcp package needs Node.js 18+ and npx on the PATH.

Verification checklist

• .env has real OPENROUTER_API_KEY (not placeholder)
• .env is in .gitignore, not tracked by git
• docker compose --env-file .env config renders MODEL_EMBEDDING_NAME and OPENAI_BASE_URL
• docker-compose.override.yaml applied (api 4G, playwright 2G)
• 6/6 containers Up (api, postgres, rabbitmq, redis, playwright, foundationdb)
• POST /v2/scrape markdown returns 200 + "Example Domain" in <1s
• POST /v2/scrape with jsonOptions returns non-empty JSON (LLM works)
• POST /v2/search returns ranked results in >1s (embeddings work)
• Verify script (ad-hoc) ran with 0 failures
• Harness detected and wiring written to the correct location
• User told to restart their harness and given a smoke-test command