Watercrawl crawl

Skill fernandoleyra/watercrawl-skill/skills/watercrawl-crawl

Open Source Agentic scrapping skill with 8 hyperfocused skills, to give research subagents web-search on steroids.

Install

npx -y skills add fernandoleyra/watercrawl-skill --skill watercrawl-crawl

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Crawl an entire website by following links and returning all pages as Markdown. Use when asked to crawl a site, scrape all pages, or get all content from a domain.

SKILL.md

1.8 KB, as published. Nobody here has run it

watercrawl-crawl

Crawl a website by following links from a starting URL.

Instructions

Parse arguments from $ARGUMENTS:
- url — required. Starting URL.
- max_pages — optional. Max pages to crawl. Default: 10. Maximum: 50.
- same_domain — optional. true (default) to stay on the same domain.
- depth — optional. Max link depth. Default: 3.
Validate inputs:
- If max_pages > 50, cap at 50 and inform the user.
- Extract the domain from the starting URL for domain filtering.
Initialize crawl state:
- visited = [] — already-fetched URLs
- queue = [starting_url] — URLs to fetch next
- results = [] — collected page data
Crawl loop (repeat until queue empty OR visited.length >= max_pages): a. Take the next URL from the queue b. Skip if already visited c. Fetch using WebFetch d. Add to visited and results: { url, title (first H1/H2), word_count, content } e. Extract links: find all [text](url) patterns in Markdown f. Filter links: same domain only (if same_domain=true), skip anchors, mailto:, tel: g. Add new links to queue
Report progress: Crawling page N/max_pages: <url>

Return summary:

## Crawl Results: <domain>
**Pages crawled:** N
**Total words:** ~N

### Pages found:
| # | URL | Title | Words |
|---|-----|-------|-------|

---
## Page Content
### Page 1: <url>
<content>

Limitations