Web fetch

Web Fetch

Fetch one URL, prove it was allowed, extract the part the caller asked for, and return that slice by digest with the provenance needed to trust it later.

What this skill does

web-fetch resolves a single URL against a host allowlist, retrieves it, extracts text, metadata, or links, and seals the result so a downstream step can cite the fetch without re-fetching. It checks the final host against the allowlist before and after redirects, retrieves up to max_bytes, and returns the final URL, the HTTP status, a content_digest over the retrieved body, the extracted slice, and a provenance block recording when it ran, every redirect hop, and how many bytes it read. The body is referenced by digest; only the extracted slice is inlined.

This is the primitive an agent reaches for when it has already decided which page to read. The decision it makes easier is "can I read this page, and what did it actually say", with the answer backed by a digest instead of a remembered paraphrase. The calling agent decides which sources matter and synthesizes across them; web-fetch retrieves exactly one source and refuses anything off the allowlist.

The request path is native Runx code. Skills do not supply a JavaScript HTTP client: the runtime owns public-network enforcement, redirect admission, timeouts, retries, byte limits, response digests, and receipt-safe evidence.

When to use this skill

• An agent has chosen a specific page and needs its content bound to a content_digest so a later step can cite it without re-fetching.
• A research pass needs each source retrieved through a single bounded net:allowlist fetch with a complete redirect chain and byte count.
• A review must later prove what a page said at fetch time.
• A follow-on governed operation needs one source extracted as text, metadata, or links.

When not to use this skill

• To judge, rank, or synthesize across sources. That remains caller reasoning; web-fetch retrieves exactly one source and refuses to reason over many.
• To reach a host the caller did not declare in allowlist, including a host reached only through a redirect.
• To write anything. The only scope is net:allowlist; there is no repo, file, wallet, or send authority here.
• To inline a large raw body. The extracted slice is the payload; the full body lives behind content_digest.
• To carry secrets. Request headers may reference a credential by ${secret} handle, but no header value, cookie, token, or auth string appears in the output or the receipt.

Procedure

1. Require url and allowlist. Either missing returns needs_agent; the fetch cannot run without a target and a declared scope.
2. Match the URL host against allowlist. On a miss, return policy_denied before any network call, recording the attempted host and the allowlist it was checked against.
3. Fetch, following redirects, re-checking each redirect target's host against the same allowlist. A redirect that lands off-allowlist halts the fetch, returns policy_denied with the hop that failed, and discards partial bodies. Cap the read at max_bytes when set.
4. Compute content_digest over the retrieved body.
5. Extract per extract: text (readable body text, default), metadata (title, description, canonical, declared language, content type), or links (absolute hrefs found in the document).
6. Return fetch_result with the final URL, status, digest, extracted slice, and provenance. Flag truncated reads in provenance; never return a clipped read as if whole.

Edge cases and stop conditions

• Missing url or allowlist: return needs_agent; the fetch has no target or no scope to check against.
• Host off the allowlist: stop with policy_denied before any network call; record the attempted host, not a response body (there is none).
• Redirect off the allowlist: halt the fetch, return policy_denied naming the hop that failed, and discard the partial body.
• Read clipped by max_bytes: flag truncated: true in provenance; the digest is over the bytes actually retrieved.
• Large raw body: never inline beyond the extracted slice; anything bigger than the requested view is reachable only through content_digest.
• Credential in a header: reference it by ${secret} handle only; no header value, cookie, or token reaches the output or the receipt.

Output schema

fetch_result:
  decision: ready | needs_agent | policy_denied
  final_url: string            # URL after redirects, the one the digest is over
  status: number               # HTTP status of the final response
  content_digest: string       # digest of the retrieved body, algorithm prefix included
  extract_mode: text | metadata | links
  extracted: string | object | array   # string for text, object for metadata, array of hrefs for links
  provenance:
    fetched_at: string         # timestamp of the fetch
    redirects: array           # ordered host hops, each re-checked against the allowlist
    bytes: number              # bytes read
    truncated: boolean         # true when max_bytes clipped the read
  policy:
    allowlist_decision: allowed | denied
    attempted_host: string     # set on policy_denied
    allowlist_checked: array   # the hosts the request was checked against

The sealed runx.receipt.v1 carries the final URL, status, content_digest, byte count, the redirect chain, and the allowlist decision. It carries no header values, no cookies, and no raw body beyond the digest.

Worked example

Input: url of the HTTP Semantics RFC, an allowlist of www.rfc-editor.org and rfc-editor.org, extract: text, and max_bytes: 200000.

Output: decision: ready; the host matched the allowlist before the request left; no redirects; status 200; content_digest is taken over the retrieved body; extracted holds the readable text slice; provenance records fetched_at, an empty redirect chain, 184302 bytes, and truncated: false. The receipt seals with the final URL, status, digest, byte count, and the allowlist decision; no header value reaches it.

Inputs

• url (required): the single URL to fetch; its host must match the allowlist.
• allowlist (required): permitted hosts or host patterns; the URL and every redirect target must match an entry.
• extract (optional): text, metadata, or links. Defaults to text.
• max_bytes (optional): cap on bytes read; a clipped read is flagged truncated in provenance.