agentsclimarketplace

Web scraper

Skill Yesterday-AI/skills/plugins/office/skills/web-scraper

Yesterday's PUBLIC plugin catalog for Claude Code and Cursor

Install
npx -y skills add Yesterday-AI/skills --skill web-scraper

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Extract content from any web page. Provides a tiered approach from lightweight fetch to full browser rendering. Use when web_fetch fails or pages require JavaScript.

SKILL.md

2.2 KB, as published. Nobody here has run it

Web Scraper

Extract readable content from any URL using the best available method.

Tiered Approach (use in order)

Tier 1: web_fetch (Built-in, fast, no deps)

Best for: Static pages, APIs, markdown docs, RSS feeds.

web_fetch url="https://example.com" extractMode="markdown"

Strengths: Zero setup, fast (~200ms), low cost. Weakness: No JavaScript rendering. Fails on SPAs, login walls, Cloudflare.

Tier 2: web_search (Built-in, Brave API)

Best for: Finding URLs, getting snippets, quick facts.

web_search query="topic keyword"

Strengths: Fast, structured results with snippets. Weakness: Needs Brave API key. Rate limited (2000/month free).

Tier 3: Playwright Headless (Full rendering)

Best for: JavaScript-heavy pages, SPAs, login walls, social media.

node skills/web-scraper/scripts/scrape.js "https://example.com"

Strengths: Renders everything a real browser would. Handles Cloudflare, JS, dynamic content. Weakness: Slow (~5-8s), needs Chromium installed, heavier on resources.

Tier 4: DuckDuckGo Lite (Fallback search)

Best for: When Brave API is unavailable.

web_fetch url="https://lite.duckduckgo.com/lite?q=search+terms"

Strengths: No API key needed. Weakness: Gets blocked after ~10 queries. Captcha.

Decision Tree

Need content from URL?
├── Static page / API / docs → Tier 1 (web_fetch)
├── Need to find URLs first → Tier 2 (web_search)
├── Page needs JS / is SPA / social media → Tier 3 (Playwright)
└── No Brave API key → Tier 4 (DuckDuckGo)

Setup (Tier 3 only)

npm install playwright
npx playwright install chromium
npx playwright install-deps chromium  # System libs, may need sudo

See Also

  • x-reader -- Specialized for Twitter/X content

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.