agentsclimarketplace

Dl crawl

Skill TheWatcher01/skills/.claude/skills/dl-crawl

Agent Skills collection — Reusable capabilities for AI coding agents. Install: npx skills add TheWatcher01/skills

Install
npx -y skills add TheWatcher01/skills --skill dl-crawl

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

20 crawlers avec TokenBucket, CircuitBreaker, cursor persistence

SKILL.md

1.0 KB, as published. Nobody here has run it

Crawlers — Datalake Souverain

20 crawlers with shared patterns:

  • AsyncGenerator for memory-efficient streaming
  • TokenBucket rate limiting per source
  • withRetry exponential backoff
  • CircuitBreaker prevents cascading failures
  • Cursor persistence resume on interrupt (dl_crawl_cursors)

Commands

pnpm crawl -- --source sirene --department 31 --max 5 --dry-run
pnpm crawl -- --source all --persist
pnpm enrich -- --siren 123456789
pnpm enrich -- --batch --limit 10 --type association
pnpm process -- --cross-ref --score --limit 1000

Key patterns

  • SIREN = universal key across all sources
  • Dry run: --dry-run flag on all CLIs
  • Audit trail: dl_crawl_runs + dl_crawl_cursors
  • Source config: src/config/sources.ts (20 sources + CircuitBreaker config)
  • OSINT tools: src/config/tools.ts (12 tools)

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.