Llms txt crawler

Skill agykit/agykit/.agent/skills/llms-txt-crawler

"The missing link between a chat-bot and a work-bot: a collection of elite skills for real-world application."

Install

npx -y skills add agykit/agykit --skill llms-txt-crawler

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Fetch and crawl llms.txt files from websites. Parses the llms.txt format to extract page URLs and downloads all listed content. Use when you need to gather documentation or content from a website that provides an llms.txt file.

SKILL.md

2.8 KB, as published. Nobody here has run it

llms.txt Crawler Skill

This skill enables you to fetch llms.txt files from websites and crawl all pages listed within them. The llms.txt format is a standard way for websites to provide LLM-friendly content listings.

Overview

The llms.txt file typically follows this format:

# Site Name

## Section Name

- [Page Title](https://example.com/page.md): Description of the page
- [Another Page](https://example.com/another.md): Another description

This skill parses these files and downloads all linked content.

Usage

Basic Usage

Run the crawl script with a target URL:

cd /path/to/skills/llms-txt-crawler/scripts
npm install  # First time only
node crawl.js --url https://example.com

Command Line Options

Option	Short	Description	Default
`--url`	`-u`	Base URL of the site with llms.txt	Required
`--output`	`-o`	Output directory for crawled files	`./output`
`--format`	`-f`	Output format: `md`, `json`, or `txt`	`md`
`--delay`	`-d`	Delay between requests in milliseconds	`500`
`--concurrent`	`-c`	Maximum concurrent requests	`3`

Examples

Crawl agentskills.io documentation:

node crawl.js --url https://agentskills.io --output ./agentskills-docs

Crawl with custom rate limiting:

node crawl.js --url https://example.com --delay 1000 --concurrent 2

Output as JSON:

node crawl.js --url https://example.com --format json

Output Structure

The script creates the following output structure:

output/
├── llms.txt              # Original llms.txt file
├── index.json            # Metadata about all crawled pages
└── pages/
    ├── page-1.md
    ├── page-2.md
    └── ...

Error Handling

Network errors: Retries up to 3 times with exponential backoff
Rate limiting: Respects delay settings between requests
Missing pages: Logs warnings but continues crawling other pages
Invalid URLs: Skips and logs invalid URLs

Integration Tips

When using this skill in an agent workflow:

First run the crawler to download content
The index.json file contains metadata about all pages
Use the downloaded markdown files for context or analysis