Llms txt and crawler access
Skill wakqasahmed/agentic-commerce-skills/skills/agentic-commerce/llms-txt-and-crawler-access
Canonical agentic commerce skills, aggregated by wakqasahmed/skills
npx -y skills add wakqasahmed/agentic-commerce-skills --skill llms-txt-and-crawler-accessAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
What its author says it does
Copied from the file, not written here
Review crawler and AI-agent access files for ecommerce sites, including robots.txt, sitemap, llms.txt, and AI bot rules.
SKILL.md
1.1 KB, as published. Nobody here has run it
LLMs.txt And Crawler Access
Workflow
- Check
robots.txtfor blocked product/category/policy paths and AI bot rules. - Check sitemap availability, freshness, and product/category coverage.
- Check whether
llms.txtexists and points to useful public resources. - Check canonical URLs and redirects that may confuse crawlers.
- Recommend minimal changes that improve agent discoverability without exposing private areas.
Delegation
This skill is the ecommerce-scoped access review. For depth, delegate to the ai-visibility pack instead of duplicating its work:
- Full robots/meta-robots/header analysis →
robots-ai-crawler-audit. - Sitemap coverage, canonicals, and redirect chains →
sitemap-discovery-audit. - Drafting a new
llms.txt→llms-txt-generator.
Guardrails
- Never recommend opening admin, cart, checkout session, account, or private order paths to crawlers.
- Keep access recommendations scoped to public storefront evidence.