Dl crawl
20 crawlers avec TokenBucket, CircuitBreaker, cursor persistenceFrom its SKILL.md
npx -y skills add TheWatcher01/skills --skill dl-crawlAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
SKILL.md
1.0 KB, 229 tokens by cl100k_base, as published. Nobody here has run it
Crawlers — Datalake Souverain
20 crawlers with shared patterns:
- AsyncGenerator for memory-efficient streaming
- TokenBucket rate limiting per source
- withRetry exponential backoff
- CircuitBreaker prevents cascading failures
- Cursor persistence resume on interrupt (dl_crawl_cursors)
Commands
pnpm crawl -- --source sirene --department 31 --max 5 --dry-run
pnpm crawl -- --source all --persist
pnpm enrich -- --siren 123456789
pnpm enrich -- --batch --limit 10 --type association
pnpm process -- --cross-ref --score --limit 1000
Key patterns
- SIREN = universal key across all sources
- Dry run:
--dry-runflag on all CLIs - Audit trail: dl_crawl_runs + dl_crawl_cursors
- Source config: src/config/sources.ts (20 sources + CircuitBreaker config)
- OSINT tools: src/config/tools.ts (12 tools)
What ships with it
Read from the repository
Just SKILL.md. No reference files, no scripts.