agentsclimarketplace
Sign in

Web scraper

Skill Yesterday-AI/skills/plugins/office/skills/web-scraper

Extract content from any web page. Provides a tiered approach from lightweight fetch to full browser rendering. Use when web_fetch fails or pages require JavaScript.From its SKILL.md

Install
npx -y skills add Yesterday-AI/skills --skill web-scraper

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

3 things to look at

  • 1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
  • runs commandsInstructs the agent to run 7 commands, including `web_fetch url="https://example.com" extractMode="markdown"` and 6 more.
  • fetches URLsInstructs the agent to fetch 2 URLs, including https://example.com and 1 more.

SKILL.md

2.2 KB, 484 tokens by cl100k_base, as published. Nobody here has run it

Web Scraper

Extract readable content from any URL using the best available method.

Tiered Approach (use in order)

Tier 1: web_fetch (Built-in, fast, no deps)

Best for: Static pages, APIs, markdown docs, RSS feeds.

web_fetch url="https://example.com" extractMode="markdown"

Strengths: Zero setup, fast (~200ms), low cost. Weakness: No JavaScript rendering. Fails on SPAs, login walls, Cloudflare.

Tier 2: web_search (Built-in, Brave API)

Best for: Finding URLs, getting snippets, quick facts.

web_search query="topic keyword"

Strengths: Fast, structured results with snippets. Weakness: Needs Brave API key. Rate limited (2000/month free).

Tier 3: Playwright Headless (Full rendering)

Best for: JavaScript-heavy pages, SPAs, login walls, social media.

node skills/web-scraper/scripts/scrape.js "https://example.com"

Strengths: Renders everything a real browser would. Handles Cloudflare, JS, dynamic content. Weakness: Slow (~5-8s), needs Chromium installed, heavier on resources.

Tier 4: DuckDuckGo Lite (Fallback search)

Best for: When Brave API is unavailable.

web_fetch url="https://lite.duckduckgo.com/lite?q=search+terms"

Strengths: No API key needed. Weakness: Gets blocked after ~10 queries. Captcha.

Decision Tree

Need content from URL?
├── Static page / API / docs → Tier 1 (web_fetch)
├── Need to find URLs first → Tier 2 (web_search)
├── Page needs JS / is SPA / social media → Tier 3 (Playwright)
└── No Brave API key → Tier 4 (DuckDuckGo)

Setup (Tier 3 only)

npm install playwright
npx playwright install chromium
npx playwright install-deps chromium  # System libs, may need sudo

See Also

  • x-reader -- Specialized for Twitter/X content

What ships with it: 1 file

4.4 KB alongside SKILL.md, 1 of them executable

scripts/

Keep looking

Skills are one crate of 325,949. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.