Fetch url as markdown

Skill CodeAlive-AI/ai-driven-development/skills/fetch-url-as-markdown

Fetch a web page (URL) and return clean Markdown via local trafilatura, with Exa MCP as a fallback for JS-rendered or anti-bot pages. Use when the user asks to read, fetch, scrape, summarize, or quote a URL — prefer this over the built-in WebFetch tool. Don't use for binary files (PDFs, images, archives) or for fetching API/JSON endpoints.From its SKILL.md

Install

npx -y skills add CodeAlive-AI/ai-driven-development --skill fetch-url-as-markdown

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

runs commandsInstructs the agent to run 3 commands, including `python3 ~/.claude/skills/fetch-url-as-markdown/scripts/fetch_url.py "<URL>"` and 2 more.

SKILL.md

3.6 KB, 855 tokens by cl100k_base, as published. Nobody here has run it

URL to Markdown

Fetch any web URL and get clean, readable Markdown — main content only, no navigation/footer/ads. Local + free by default; smart fallback to Exa MCP when the page can't be extracted locally.

Workflow (the only thing the agent needs to remember)

Try trafilatura first:

python3 ~/.claude/skills/fetch-url-as-markdown/scripts/fetch_url.py "<URL>"

If exit code is 1 or 2 → fall back to Exa MCP with the same URL:
```
mcp__exa__web_search_advanced_exa(
    query="<URL>",
    includeDomains=["<host of URL>"],
    numResults=1,
    textMaxCharacters=50000,
    type="auto"
)
```
(mcp__exa__crawling works too if the server exposes it; the web_search_advanced_exa call above is the always-available variant — pin the host with includeDomains and use the URL itself as the query.)
Exit code 3 means trafilatura is not installed — install once:
```
python3 -m pip install --break-system-packages trafilatura
```

Exit codes (what they mean for the fallback decision)

Code	Meaning	Action
0	Markdown printed to stdout	done
1	DownloadError — network/HTTP/timeout/anti-bot block at fetch	fall back to Exa
2	ExtractionError — empty extract, JS/Cloudflare wall, or stub body (<200 chars)	fall back to Exa
3	trafilatura missing	install (see above), then retry
4	UnsupportedContentTypeError — URL is binary (PDF, image, archive)	don't fall back to Exa; use the right specialized skill (e.g. `pdf` for PDFs)

Defaults baked into the script

output_format="markdown", include_formatting=True — keeps headings/lists/code structure where the source HTML uses real <h1..h6> etc.
include_links=True, include_tables=True
with_metadata=True → emits a YAML frontmatter (title, author, date, url, hostname)
favor_recall=True, deduplicate=True — readable but trims duplicates
Real-browser User-Agent + 30s timeout configured in scripts/settings.cfg
Anti-stub guards (built into the script):
- rejects Content-Type other than text/html|application/xhtml+xml|text/plain|application/xml|text/xml → exit 4
- sniffs raw HTML for Cloudflare / "Please enable JavaScript" / Imperva / DataDome wall markers → exit 2
- rejects extracted bodies under 50 chars (configurable via --min-body N, 0 to disable) → exit 2

Useful flags

... fetch_url.py "<URL>" --no-links     # strip hyperlinks
... fetch_url.py "<URL>" --no-tables    # strip tables
... fetch_url.py "<URL>" --no-metadata  # omit YAML header
... fetch_url.py "<URL>" --comments     # include user comments (off by default — usually noise)
... fetch_url.py "<URL>" --images       # include image refs (experimental)
... fetch_url.py "<URL>" --precision    # terser output, drops borderline content

When to choose what

Situation	Tool
Article, blog post, docs, README, wiki	trafilatura (default) — local, free
JS-heavy SPA, login-walled, Cloudflare	Exa fallback (the script will signal exit 2)
Bulk / many URLs	trafilatura — no quota, no API key
Already failed twice on a domain	Exa directly

What ships with it: 3 files

15.1 KB alongside SKILL.md, 1 of them executable

scripts/

fetch_url.pyruns8.3 KB
settings.cfg893 B

README.md6.0 KB

Gives 0 of the 12 instructions most docs writing skills give in 855 tokens

Counted across 1,951 of the 3,904 authors here whose files we hold, read 2026-09-06

Use third-person for skill descriptionsin 54 of 1951, across 35 files
Start descriptions with Use whenin 43 of 1951, across 29 files
Run baseline scenarios before writing any skillin 40 of 1951, across 26 files
Use active voicein 40 of 1951, across 36 files
Map file responsibilities before defining tasksin 36 of 1951, across 29 files
Use checkbox syntax for tracking stepsin 35 of 1951, across 27 files
Ask one question at a timein 35 of 1951
Offer execution options after saving the planin 33 of 1951, across 24 files
Include complete code in every stepin 33 of 1951, across 27 files
Design units with clear boundaries and interfacesin 31 of 1951, across 23 files
Announce the skill usage at the startin 30 of 1951
Verify agent compliance after adding the skillin 29 of 1951, across 17 files

Said here and by no other author read

Try trafilatura first for URL fetching
Fall back to Exa MCP if trafilatura fails
Install trafilatura if exit code is 3
Use Exa MCP for JS-rendered or anti-bot pages
Use the URL as the query for Exa MCP
Pin the host domain when using Exa MCP

Grouped from the skills themselves: near-identical wordings counted once, and counted by distinct author, so one author publishing three of these counts once. Length counted with cl100k_base; the agent that loads this file may tokenize it differently.