agentsclimarketplace

Doc scraper

Skill sfc-gh-dflippo/snowflake-dbt-demo/.claude/skills/doc-scraper

This project demonstrates many of dbt's features when used with the Snowflake Data CloudFrom the repository description

Install
npx -y skills add sfc-gh-dflippo/snowflake-dbt-demo --skill doc-scraper

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

SKILL.md

2.1 KB, 463 tokens by cl100k_base, as published. Nobody here has run it

Snowflake Documentation Scraper

Scrapes docs.snowflake.com sections to Markdown with SQLite caching (7-day expiration).

Usage

First time setup (auto-installs uv and doc-scraper):

python3 .claude/skills/doc-scraper/scripts/doc_scraper.py

Subsequent runs:

doc-scraper --output-dir=./snowflake-docs
doc-scraper --output-dir=./snowflake-docs --base-path="/en/sql-reference/"
doc-scraper --output-dir=./snowflake-docs --spider-depth=2

Command Options

OptionDefaultDescription
--output-dirRequiredOutput directory for scraped docs
--base-path/en/migrations/URL section to scrape
--spider-depth1Link depth: 0=seeds, 1=+links, 2=+2nd
--limitNoneCap URLs (for testing)
--dry-run-Preview without writing

Output

output-dir/
├── SKILL.md              # Auto-generated index
├── scraper_config.yaml   # Editable config (auto-created)
├── .cache/               # SQLite cache (auto-managed)
└── en/migrations/*.md    # Scraped pages with frontmatter

Configuration

Auto-created at {output-dir}/scraper_config.yaml:

rate_limiting:
  max_concurrent_threads: 4
spider:
  max_pages: 1000
  allowed_paths: ["/en/"]
scraped_pages:
  expiration_days: 7

Troubleshooting

IssueSolution
Too many pagesLower --spider-depth or edit config
Missing pagesIncrease --spider-depth
Cache corruptionDelete {output-dir}/.cache/ (rare)

Keep looking

Skills are one crate of 614,194. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.