Efficient web research

Efficient Web Research Skill

A protocol for accessing web content in the most token-efficient, accurate, and structured way — using the right tool at the right depth, and stopping as soon as the question is answerable.

---

When to Use

• Use this skill when the task matches this description: Protocol for token-efficient web research. Use when accessing URLs, GitHub repos, or running search queries. Prevents full-page fetching waste.

Core Principle

Fetch the minimum needed to answer. Skim before you dive. Stop when you can answer.

Every unnecessary fetch wastes tokens and adds noise. This skill enforces a layered approach where you escalate fetch depth only when shallower layers fail.

---

Step 1 — Classify the Input

Before fetching anything, identify what kind of input you received:

Input Type	Example	Go To
GitHub repo URL	github.com/user/repo	GitHub Protocol
Specific page URL	docs.python.org/3/library/os	URL Protocol
Topic / query (no URL)	"how does RAFT consensus work"	Search Protocol
Multiple URLs	List of links	Multi-URL Protocol
PDF / file link	.pdf, .txt, .md URL	File Protocol

---

GitHub Protocol

Use when input is a GitHub URL (repo, file, PR, issue, etc.)

Step 1 — Parse the URL

github.com/{owner}/{repo}                → Repo root
github.com/{owner}/{repo}/tree/{branch}  → Directory
github.com/{owner}/{repo}/blob/{branch}/{path} → Single file
github.com/{owner}/{repo}/issues/{n}     → Issue
github.com/{owner}/{repo}/pull/{n}       → Pull request

Step 2 — Use GitHub API (preferred over scraping)

Always prefer the GitHub API. It returns clean JSON — no HTML parsing needed.

# Repo metadata (name, description, language, stars, topics)
GET https://api.github.com/repos/{owner}/{repo}

# File tree (see what files exist — very cheap)
GET https://api.github.com/repos/{owner}/{repo}/git/trees/{ref}?recursive=1

# Single file content (base64 encoded)
GET https://api.github.com/repos/{owner}/{repo}/contents/{path}?ref={ref}

# README only (usually enough to understand the repo)
GET https://api.github.com/repos/{owner}/{repo}/readme

Step 3 — Layered Fetch for Repos

Layer 1 (always do first):
  → Fetch repo metadata + README only
  → Can you answer the user's question now? YES → STOP. NO → continue.

Layer 2 (only if needed):
  → Fetch file tree to understand structure
  → Identify the 1-3 most relevant files based on the question
  → Can you answer now? YES → STOP. NO → continue.

Layer 3 (last resort):
  → Fetch specific relevant files only (never fetch all files)
  → Prioritize: main entry point, config files, key modules

Token Rules for GitHub

• README alone answers ~70% of "what does this repo do" questions — always try it first
• Never fetch more than 3 files in a single research turn
• If a file exceeds ~300 lines, read only the top (imports + class/function signatures)
• Decode base64 content from API before passing to context

---

URL Protocol

Use when the user gives a specific non-GitHub URL (docs, articles, blogs, etc.)

Step 1 — Assess the URL type

Site type	Likely works with	Notes
Static docs / MDN / ReadTheDocs	read_url_content	Fast, clean, cheap
News articles / blogs	read_url_content	Usually fine
SPAs / React/Next.js apps	browser_subagent	JS-rendered
Auth-gated pages	browser_subagent	Needs login
Raw GitHub files (raw.githubusercontent)	read_url_content	Direct text

Step 2 — Layered Fetch

Layer 1 — Skim
  → Fetch the URL with read_url_content
  → Read only headings (H1, H2, H3) and first paragraph
  → Does this page contain what the user needs? NO → try a different URL or search. YES → continue.

Layer 2 — Targeted Extract
  → If the page has anchor links (e.g. /docs/page#section), fetch with the anchor
  → Extract only the relevant section (200–500 tokens max)
  → Can you answer? YES → STOP.

Layer 3 — Full Fetch
  → Fetch full page, strip boilerplate (nav, footer, ads, cookie banners, sidebars)
  → Cap at 2000 tokens. Summarize before passing to answer.

Layer 4 — Browser Subagent (last resort only)
  → Use ONLY if read_url_content returns empty, garbled, or JS-placeholder content
  → Instruct subagent: "Navigate to [URL], wait for content to load, extract [specific section]"
  → Do NOT use browser_subagent for static pages — it's expensive

What to Strip from Fetched Pages

Always remove before using fetched content:

• Navigation menus and breadcrumbs
• Cookie banners and GDPR notices
• "Related articles" / "You might also like" blocks
• Footer content (copyright, links)
• Social share buttons
• Ads and sponsored content

Extract and keep:

• Main article / documentation body
• Code blocks
• Tables with data
• Numbered steps or procedures

---

Search Pr