Scraping & extraction
918 rows from 221 repositories
Getting structured data out of things that were not meant to give it.
What Scraping & extraction skills agree on
107 skill files read, by 109 of the 148 authors on this shelf whose files we hold, 2026-09-06
Counted by distinct author, so one author publishing three of these counts once. Where a claim sits in fewer files than authors, that is said: a claim held by forty authors across three files is one file people copied, not forty people who agreed. Near-identical wordings are grouped and the other wordings are shown, so the grouping is yours to check.
What they tell the agent to do
- Add delays between requests7 of 109 in 6 filesalso worded as Add configurable random delays between requests; Add delays between requests for rate limits
- Close the browser when finished6 of 109 in 4 filesalso worded as Close the browser in a finally block; Close the browser when done
- Authenticate every request with the x-api-key header5 of 109 in 2 filesalso worded as Authenticate requests with the API key header
- Default to Scraper APIs when one exists5 of 109 in 2 filesalso worded as use official APIs instead of scraping when possible
- Map site structure before a full crawl5 of 109 in 3 filesalso worded as map a site before crawling it; run map before a full crawl
- Use the Web Scraping API for arbitrary URLs4 of 109 in 1 file
- Treat data as valid only if requestMetadata.status is ok4 of 109 in 1 file
- Set client timeout to at least 300 seconds4 of 109 in 1 file
- Retry only 429 and 5xx with exponential backoff4 of 109 in 1 file
- Cap concurrency at your plan limit4 of 109 in 1 file
- Persist the async job id immediately4 of 109 in 1 file
- Poll job status every 10 to 30 seconds4 of 109 in 1 file
- Download job results immediately once finished4 of 109 in 1 file
- Disable jsRendering first; enable only if needed4 of 109 in 1 file
- Check SERP rich snippets before direct page access4 of 109 in 1 file
What they tell it not to do
- Never hardcode or log the API key6 of 109 in 3 filesalso worded as Never hardcode the API key; Never hardcode API keys
- Do not scrape non-public or unauthorized data5 of 109 in 2 filesalso worded as never collect unauthorized or private data
- Never retry 4xx errors4 of 109 in 1 file
- Do not treat HTTP 200 alone as success4 of 109 in 1 file
- Do not use a cookies parameter4 of 109 in 1 file
- Do not use Scraper Jobs when an API equivalent exists4 of 109 in 1 file
- Do not skip the tvly install step3 of 109 in 1 file
- Do not fall back to other tools3 of 109 in 1 file
- Never treat examples as a substitute for security review3 of 109 in 1 file
- Never take destructive or costly actions without user approval3 of 109 in 1 file
What they expect to be installed
- curl22 of 109 in 17 files
- Playwright19 of 109 in 17 files
- Puppeteer15 of 109 in 11 files
- firecrawl_scrape9 of 109 in 8 files
- firecrawl_search9 of 109 in 8 files
- python39 of 109
- WebFetch8 of 109
- npm8 of 109
- jq7 of 109
- requests7 of 109
What they ask it to produce
- Structured JSON extraction results14 of 109 in 9 filesalso worded as JSON file of extracted structured data; Structured JSON from schema-based extraction
- Screenshots8 of 109 in 6 filesalso worded as screenshot files; Page screenshots
- Scraped page content as markdown or JSON6 of 109 in 5 filesalso worded as Markdown of scraped pages; Markdown page content from scrape
- Markdown file for each crawled page5 of 109 in 3 filesalso worded as one markdown file per crawled page; .crw/pages/page-* one markdown file per page
- Markdown corpus from crawler jobs4 of 109 in 1 file
- Downloaded csv, json, or xlsx job results4 of 109 in 1 file
- PDF documents4 of 109 in 2 files
- Puppeteer automation scripts4 of 109 in 2 filesalso worded as Working Puppeteer scripts
When Scraping & extraction authors say to reach for one
The situations these authors wrote into their own files, counted out of the same 109 authors, with the skills that name each one
- User asks to crawl or scrape a site14 of 109 in 12 files
- Seo firecrawl
- Tavily ai skills tavily crawl 1.0.1
- and 2 more on this shelf
- User needs web scraping12 of 109 in 8 files
- User wants structured data from websites6 of 109 in 5 files
- Firecrawl agent
- Ghost scraper
- and 1 more on this shelf
- User needs search engine results5 of 109 in 2 files
- Hasdata
- and 10 more on this shelf
- When crawling an entire site5 of 109 in 4 files
- Firecrawl automation
- and 7 more on this shelf
- User asks to extract data from a URL5 of 109
- User needs structured data extraction4 of 109 in 1 file
- Hasdata
- and 10 more on this shelf
- User needs ecommerce, travel, jobs, or local business data4 of 109 in 1 file
- Hasdata
- and 10 more on this shelf
How Scraping & extraction skills are built
833 skill directories by 178 authors, read from their repositories’ own file trees 2026-08-05
The middle bundle among those shipping files is 3 files, 21.1 KB beside SKILL.md
Counted by distinct author, same as above, so one author publishing forty template copies counts once. SKILL.md itself is not counted as a file, so a single-file skill is one where that file is the whole skill.
The shape
- SKILL.md is the whole skill71 of 178 authors, 534 of 833 skills
- files ship beside it107 of 178 authors, 299 of 833 skills
- executable scripts ship inside63 of 178 authors, 168 of 833 skills
The folders they converge on
- references/55 of 178 authors, 117 of 833 skills
- scripts/50 of 178 authors, 118 of 833 skills
- evals/12 of 178 authors, 14 of 833 skills
- reference/10 of 178 authors, 15 of 833 skills
- examples/9 of 178 authors, 11 of 833 skills
- agents/8 of 178 authors, 14 of 833 skills
gosom/google-maps-scraper/skills/google-maps-scraper Skill
5,506★ reposcrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place
us/crw/io.github.us/crw MCP server
552★ repoOpen-source web scraper for AI agents with scrape, crawl, and map tools
alex-on-ai/WebReaper/io.github.alex-on-ai/webreaper MCP server
142★ repoAI-native web scraper: scrape, crawl and map any site to clean markdown over stdio. MIT-licensed.
yfe404/web-scraper Skill
82★Intelligent web scraping Claude Code skill with automatic strategy selection and TypeScript-first Apify Actor development
brettdavies/crawl4ai-skill Skill
no license42★Scrape JavaScript-heavy sites and extract structured data via reusable CSS schemas. Portable agent skill wrapping the Crawl4AI CLI and Python SDK.
ea-toolkit/architecture-catalog/.claude/skills/crawl-apis Skill
38★ repoGit-native architecture catalog — Markdown registry, schema-driven UI, AI-ready
goesByhc/cn-scraper-mcp/io.github.goesByhc/cn-scraper-mcp MCP server
22★ repoMCP server for AI agents to search major Chinese internet platforms with local-first login support.
houtini-ai/seo-crawler-mcp/io.github.houtini-ai/seo-crawler-mcp MCP server
16★ repoCrawl and analyse websites for SEO errors using Crawlee with SQLite storage
lisniuse/taoguba-crawler-skill Skill
15★淘股吧帖子爬取skill，给 AI agent（AI智能体）使用。
ogarciarevett/stacker/skills/stacker Skill
15★ repoAgent skill: stack overlapping skills — run the best candidates in parallel, score outputs, merge the result
mgicoda/icoda-skills/aeo-audit Skill
no license6★ repoFree Claude skills and agent skills for AI marketers. aeo-audit checks AI bot access, schema markup, and generates a scored HTML report.
creativault/skills/creator-scraper-cv Skill
no license6★ repo通用AI助手，为creativault打造的技能包
ofershap/mcp-server-scraper/io.github.ofershap/scraper MCP server
5★ repoWeb scraping MCP — extract clean markdown, links, and metadata from any URL.
pedro-rivas/android-puppeteer-mcp/io.github.pedro-rivas/android-puppeteer-mcp MCP server
no license5★ repoMCP server for Android automation with UI interaction, screenshots, and device control
thangnguyenworkspace/industry-pulse/.claude/skills/research-crawl-email Skill
4★ repoWatchlist-driven intelligence pipeline crawling LinkedIn, X, RSS, and email into one neutral signals brief.
scrapeless-ai/llm-chat-scraper-skill Skill
no license3★OpenClaw skill for scraping ChatGPT, Gemini, Perplexity and Grok responses. Built for AI agents and GEO monitoring.
chrisdothtml/ai-job-finder/.claude/skills/create-scraper Skill
3★ repoLet AI read the careers pages for you, so you can spend your time doing literally anything else
thirdwatch-dev/scraping-skills/skills/anti-bot-scraping Skill
3★ repoWeb scraping skills for Claude & coding agents — anti-bot bypass, build-vs-buy, and ready-made scrapers for jobs, e-commerce, reviews, social, leads, real estate, travel, food & SEO. npx skills add thirdwatch-dev/scraping-skills
Aidress skill find verified data extraction agent
Aidress-ai/aidress-skill-find-verified-data-extraction-agent Skill
2★Find a Verified Data Extraction Agent. Agent discovery, trust verification, and capability routing powered by Aidress — the coordination registry for autonomous AI agents. Use for data agent discovery, trust verification, and capability routing — via Aidress (https://api.aidress.ai).
sam6dvpte34/lead-generation-skill/skills/accounting-firm-lead-scraper Skill
no license2★ repoAgent-ready lead generation and contact discovery Skills, best for LinkedIn, Google Maps, website contact scraping, prospect enrichment, and intent signal research
robot-resources/skills Skill
2★Agent skills for Robot Resources.
sam6dvpte34/social-media-skill/skills/bilibili-comment-scraper Skill
no license2★ repoAgent-ready social media crawling and task automation Skills, best for Instagram, TikTok, YouTube, X/Twitter, LinkedIn, Facebook, Reddit, and Xiaohongshu
09ashishkapoor/Roxy_SEO/skills/roxy-seo Skill
2★ repoRoxy SEO: an open-source agent skill and PI extension for SEO, AEO, GEO, AI search visibility, technical audits, structured data, and content strategy.
ChocoData-com/amazon-scraper-api-mcp/io.github.ChocoData-com/amazon-scraper-api-mcp MCP server
2★ repoScrape Amazon products, search, and async batch ASIN lookups across 20 marketplaces
Perufitlife/multi-scraper-mcp/io.github.Perufitlife/multi-scraper-mcp MCP server
no license2★ repo14 web scrapers as MCP tools: Reddit, Amazon, Google Maps, Yelp, YouTube, Indeed & more.
yubinkim444/ai-first-scraper-mcp/io.github.yubinkim444/ai-first-scraper-mcp MCP server
2★ repoThree MCP tools: fetch_page, fetch_pages_batch, search_web. Ad-free Markdown for AI agents.
scrapeless-ai/scrapeless-agent-browser/skills/scraping-browser-skill Skill
1★ repoThe ultimate cloud browser automation CLI purpose-built for AI agents.
Universal data acquisition pipeline skill
Pranjay-kumar/universal-data-acquisition-pipeline-skill Skill
1★Codex/Claude skill for designing robust scraping, API, Playwright, and authorized data acquisition pipelines
wha7ev9r/rsshub-route-skill/rsshub-route Skill
1★ repoRSSHub route builder skill — generates complete, PR-ready RSSHub route code (namespace.ts + handler + Radar rules + PR checklist) from a target website
GMapsExtractor/Google-Maps-Scraper-MCP-server/com.gmapsextractor/google-maps-scraper MCP server
1★ repoLive Google Maps business search, review, and photo data for AI agents over MCP.
Mcp gtm tech stack signal scraper
mambalabsdev/mcp-gtm-tech-stack-signal-scraper/com.mambabuilt/mcp-gtm-tech-stack-signal-scraper MCP server
1★ repoDetects a company CRM, sequencer, and marketing automation from its public website. Clay-ready.
mambalabsdev/mcp-gtm-hiring-signal-scraper/com.mambabuilt/mcp-gtm-hiring-signal-scraper MCP server
1★ repoDetects GTM hiring activity from company career pages via Greenhouse, Lever, Ashby. Clay-ready.
Br0ski777/twitter-scraper-x402/io.github.Br0ski777/twitter-scraper MCP server
1★ repoScrape Twitter/X profiles, tweets, search. No API key. Structured JSON. x402 pay-per-call.
Threads Research Scraper Skill
KarlYu130/Threads-Research-Scraper-Skill Skill
no license0★An agent-ready AI Skill and Playwright scraper for extracting live posts, search queries, and child reply comment threads on Meta Threads (threads.net).
lisniuse/cctv-news-scraper Skill
0★Scrape today's CCTV Xinwen Lianbo (新闻联播) daily news summary from EastMoney (东方财富网). Use when the user asks to get today's 新闻联播 content, CCTV news summary, or wants to fetch the latest 央视新闻联播要闻集锦 article. Triggers on requests mentioning 新闻联播, xinwen lianbo, CCTV news digest, or EastMoney news scraping.
WebCrawlerAPI/skills/webcrawlerapi Skill
no license0★ repoSkill to get website or webpage content in markdown for your AI agent
Thanane15M/scrapling-skill Skill
0★Claude Skill for Scrapling — adaptive web scraping with self-healing selectors.
johnisanerd/claude-skill-google-jobs-scraper/apify-google-jobs-scraper Skill
no license0★ repoClaude/agent skill: scrape Google Jobs listings billed per delivered result, with employer and board exclusion filters. Installs via npx skills add.
Apify app store reviews scraper
johnisanerd/claude-skill-app-store-reviews-scraper/apify-app-store-reviews-scraper Skill
no license0★ repoClaude/agent skill: scrape Apple App Store reviews into structured JSON. Installs via npx skills add.
johnisanerd/claude-skill-duckduckgo-scraper/apify-duckduckgo-scraper Skill
no license0★ repoClaude/agent skill: scrape DuckDuckGo search results into structured JSON. Installs via npx skills add.
Apify yandex scraper bulk serp
johnisanerd/claude-skill-yandex-scraper-bulk-serp/apify-yandex-scraper-bulk-serp Skill
no license0★ repoClaude/agent skill: bulk Yandex SERP collection with predictable per-result billing. Installs via npx skills add.
johnisanerd/claude-skill-scrape-google-jobs/apify-scrape-google-jobs Skill
no license0★ repoClaude/agent skill: scrape Google Jobs listings to structured JSON with the Apify Google Jobs Scraper. Installs via npx skills add.
johnisanerd/claude-skill-linkedin-posts-scraper/apify-linkedin-posts-scraper Skill
no license0★ repoClaude/agent skill: scrape public LinkedIn posts into structured JSON. Installs via npx skills add.
brightstone111/web-scraping-skills/plugins/web-scraping/skills/cdp-graphql-scraper Skill
0★ repoClaude Code skill marketplace for authorized web scraping — attach to your logged-in Chrome over CDP and harvest the JSON/GraphQL the page already fetches, with anti-block hygiene and resumable state.
meturley/trail-marker-geo/skills/ai-crawler-check Skill
0★ repoA collection of AI agent skills for optimizing political campaign websites for AI search (GEO). Built for candidates, campaign managers, and advocacy staff who want to help their site get accurately read and cited.
johnisanerd/claude-skill-google-news-api/apify-google-news-api Skill
no license0★ repoClaude/agent skill: Google News articles as structured JSON. Installs via npx skills add.
johnisanerd/claude-skill-scrape-google-shopping/apify-scrape-google-shopping Skill
no license0★ repoClaude/agent skill: scrape Google Shopping products, prices, and sellers. Installs via npx skills add.
gdm257/cc-plugins/plugins/agent-skills/skills/crawlee-skill Skill
no license0★ repoElegant Claude Code Plugins
huoy348/wechatcrawler Skill
no license0★Crawls articles from WeChat Official Accounts. Searching the name of any accounts（using personal cookie）gets the content.
heart-wcl/agent-browser-guide Skill
0★Claude Code skill for browser automation via Vercel Agent Browser CLI. Replaces Playwright MCP with CLI-first approach.
cbrunnkvist/agent-skills/skills/katana-web-crawl Skill
0★ repoarufian/crawlit-skill Skill
0★Agent skill pack for Crawlit web scraping, crawling, and site mapping
MrBridgeHQ/apify-scraper-builder-claude/skills/apify-scraper-builder Skill
0★ repoapify-scraper-builder - Claude Code Agent Skill (target LLM: Claude). Part of the mr-bridge.com toolkit.
MrBridgeHQ/scraping-expert-claude/skills/scraping-expert Skill
0★ reposcraping-expert - Claude Code Agent Skill (target LLM: Claude). Part of the mr-bridge.com toolkit.
MrBridgeHQ/scraper-audit-expert-claude/skills/scraper-audit-expert Skill
0★ reposcraper-audit-expert - Claude Code Agent Skill (target LLM: Claude). Part of the mr-bridge.com toolkit.
Extraction automation architect
satishTheLegend/extraction-automation-architect Skill
0★Design a resilient web-scraping/automation system end-to-end: legality + strategy, anti-bot tactics, schema extraction, self-healing selectors, scheduling, and monitoring.
kev-hu/ai-toolkit/skills/apify-scraper Skill
0★ repoAI tools, hooks, skills, and prompts I actually use day to day — each with a what/why/how write-up
Albert-Tam/scrappycoco-integrations/plugins/scrappycoco/skills/scrappycoco Skill
no license0★ repoOfficial Scrappycoco MCP plugin and portable agent skill
ActableSite AI Crawler Monitor
unitedideas/actablesite-mcp/com.actablesite/ai-crawler-robots-monitor MCP server
0★ repoCheck AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
Br0ski777/web-scraper-x402/io.github.Br0ski777/web-scraper MCP server
0★ repoExtract clean markdown from any URL. Removes boilerplate. For RAG pipelines. x402.