Extractability check

Extractability Check

This skill inspects a single web page the way an AI retrieval crawler sees it and reports whether the content is extractable and citable. Most AI retrieval bots (the fetchers behind AI answer engines) do not execute JavaScript — they read the raw server-rendered HTML of the first response. If the substantive content only appears after client-side JS hydration, those crawlers see an empty shell, and nothing on the page can be cited no matter how good it is. That single check is the most important thing this skill does.

Hard rules (read first)

• Diagnostic only. Never modify, submit to, or write anything to the user's site. Only fetch and report.
• Report only what's in the fetched HTML. Never assume content exists, never infer that "it probably renders fine." If it's not in the bytes you fetched, report it as not visible to the crawler.
• Be honest about scope. This checks static-HTML visibility and structure — what a non-JS crawler can see and how well-organized it is for extraction. It does not measure live citation performance (whether a given engine actually cites the page). That requires ongoing engine/citation monitoring, which is out of scope. Say so plainly; don't imply this predicts real-world citation.
• One page per run. This is a page-level diagnostic, not a site crawl.

Workflow

1. Get the URL

If the user hasn't given a full URL, ask for one. You need a single concrete page (not a domain).

2. Fetch the page as a non-JS crawler

Run the bundled fetch script. It requests the raw HTML with a plain crawler-style user agent, follows redirects, does not execute JavaScript, and reports what came back:

python3 scripts/fetch_page.py "<URL>"

The script writes the fetched HTML to a temp file and prints a JSON block of pre-computed signals (byte size, whether a JS-framework root element looks empty, JSON-LD blocks found, heading list, visible text length, etc.). The JSON includes raw_html_path — the location of the saved HTML. Read both the JSON summary and, if you need to judge borderline cases, the file at raw_html_path itself.

If the fetch fails, report that honestly and explain why it matters — don't just say "the script failed." Do not fall back to a JS-rendering or browser fetch and pretend it's what the crawler sees.

• 403 / 401 / Cloudflare or Captcha challenge page: this is a finding, not an error. Explain that the site's firewall/bot-mitigation is actively blocking simple crawler-style user agents, and that AI search crawlers making similarly plain requests are likely blocked too — so the page may be effectively invisible to AI retrieval regardless of how good its content is. Recommend the user check their bot-protection allowlist; note this gates everything else, like hydration does.
• Timeout / DNS / connection error: report it plainly and suggest re-checking the URL; distinguish it from an active block.
• SSL / certificate error (CERTIFICATE_VERIFY_FAILED): this is an environment problem, not a site block and not a content problem. It usually means the machine running the script can't verify HTTPS certificates (common on python.org macOS builds that never ran Install Certificates.command; the script already tries the certifi CA bundle as a fallback). Say plainly that the check couldn't run because of a local certificate-verification issue — do not report it as the site being down, blocking crawlers, or having bad content. Suggest the user install/repair their certificate store (or pip install certifi) and re-run.

3. Run the five checks

For every check, base the verdict strictly on raw.html / the script's signals. See references/checks.md for detailed detection heuristics and edge cases. In brief:

1. JS-hydration visibility (most important). Is the substantive body content present in the raw HTML, or does the page ship a near-empty root (e.g. <div id="root"></div>, <div id="__next"></div>) that fills in only after JS runs? Compare visible text length and the framework-root emptiness signal. If the real content isn't in the bytes, flag this loudly — every other check is moot until it's fixed.
   • Watch for the "fake hydration" mirage. A page can render content server-side but wrap it in hidden styling (display:none, visibility:hidden, aria-hidden="true", or a bare hidden attribute) that JS only toggles open later — common in tabs and accordions. Check the hidden_content_blocks signal. If substantive text is physically in the bytes but hidden, note that: the text is technically present, but crawlers may devalue hidden content and users of a non-JS view never see the toggle. Report it as a partial/at-risk case, not a clean pass.
2. Front-loading. Do the key facts (who/what/claims a citation would need) appear in roughly the top third of the extracted text, or are they buried below navigation, intros, and filler? Crawlers weight early content.
3. Dates & sources. Do factual claims carry dates and attributable sources, or are they undated and unattributed? Un-sourced, undated claims are hard to cite with confidence. Hold a high bar and resist grading generously: to count as dated and sourced, a claim must pair a specific year or month with a named external publication, organization, person-with-affiliation, or a link. Vague attributions like "studies show," "according to reports," "experts say," or "recently" do not count as sourced, and a bare year with no source does not count as sourced. When in doubt, mark it weak rather than crediting it.
4. Structured data. Is there Person or FAQPage JSON-LD in the HTML? Report which schema types are present (or that none are).
5. Headings. Are headings phrased as questions or clear topics (extractable, matchable to queries), or are they vague slogans/marketing lines?

4. Report and route to fixes

Produce the report in the exact structure below. For each finding that needs work, name the specific fix skill in this suite so the user knows where to go next:

• Missing/weak Person or FAQPage schema → candidate-schema
• Buried or undated/unsourced facts → answer-block-writer
• Vague, unsupported claims → claim-strengthener
• Prose that needs reshaping (slogan headings, wall-of-text structure) → page-restructurer

CRITICAL — no false diagnostics. Do not recommend a downstream fix skill unless that specific check came out as a definitive "Needs Work" grounded in the fetched HTML. The urge to fill out a longer "What to fix next" list is not a reason to route — a page with fine headings, fine claims, and fine schema should get no fix recommendations for those dimensions, and you should say they're fine. If the only problem is hydration (or a fatal block from Step 2), recommend only that and stop; the other checks are moot until the content is actually reachable, so don't pile on speculative fixes. The gatekeeper problems have no fix skill here — for JS-only/partial rendering advise the user's web developer to add server-side rendering (SSR), pre-rendering, or a static-HTML fallback; for a bot block advise checking the firewall's bot-mitigation allowlist.

Report structure

ALWAYS use this exact template:

# Extractability Check — <URL>

**Fetched:** <status code, final URL if redirected, byte size> · non-JS crawler view

## 1. Is your content actually visible? (most important)
<Verdict: VISIBLE / PARTIAL / JS-ONLY / BLOCKED> — <one-line plain explanation of what the raw HTML contained>
<If JS-only or partial: state that AI crawlers likely see little/none of this, and that this must be fixed before anything else matters.>
<If BLOCKED (403/401/Cloudflare/Captcha): state that the site's bot mitigation refused a plain crawler, so AI retrieval bots are likely blocked too — the page may be effectively invisible regardless of content.>

## 2. Are the key facts front-loaded?
<Verdict + what sits in the top third vs buried>

## 3. Do claims have dates and sources?
<Verdict + examples of dated/sourced vs undated/unsourced claims found in the HTML>

## 4. Structured data (JSON-LD)
<Which schema types were found, or "none". Note specifically whether Person / FAQPage are present.>

## 5. Headings — questions/topics or slogans?
<Verdict + a few example headings pulled from the HTML>

## What to fix next
<Bulleted list. Each item: the problem found + the fix skill to use. Order by impact — hydration first if it's an issue. Only list dimensions that scored "Needs Work"; if a dimension is fine, do not invent a fix for it. If only hydration (or a Step 2 block) is broken, list only that.>
<IMPORTANT: the gatekeeper problems have no fix skill in this suite — they need developer work on the actual codebase, not content edits. For JS-ONLY / PARTIAL, do NOT name a fix skill; instead advise the user to consult their web developer about server-side rendering (SSR), pre-rendering, or a static-HTML fallback so the content ships in the first response. For BLOCKED, advise checking the firewall/bot-mitigation allowlist so legitimate crawlers aren't refused. Only the four content checks (schema, front-loading/dates, claims, headings) route to the fix skills.>

## Scope & limits
This is a static-HTML visibility and structure check based only on the raw HTML fetched above. It does not measure whether any AI engine actually cites this page live — that requires ongoing citation monitoring and is out of scope. Findings reflect what was in the fetched bytes, nothing assumed.

Keep the tone direct and candidate-facing: plain language, no jargon dumps. Lead with the hydration verdict because it gates everything else.

Worked example

A synthetic run against a React campaign site that ships an empty #root div and relies entirely on client-side rendering:

# Extractability Check — https://example-campaign.org/issues

**Fetched:** 200 OK, no redirect, 4,812 bytes · non-JS crawler view

## 1. Is your content actually visible? (most important)
JS-ONLY — the raw HTML shipped in the first response is almost entirely a bare `<div id="root"></div>` with no substantive text; the visible content only appears after client-side JavaScript renders the React app.
This must be fixed before anything else matters — AI retrieval crawlers that don't execute JavaScript see an empty page, so nothing here can currently be cited no matter how good the content is.

## 2. Are the key facts front-loaded?
Not applicable — there's no content in the fetched bytes to evaluate.

## 3. Do claims have dates and sources?
Not applicable — there's no content in the fetched bytes to evaluate.

## 4. Structured data (JSON-LD)
None found.

## 5. Headings — questions/topics or slogans?
None found in the raw HTML (headings are rendered client-side).

## What to fix next
- **JS-only rendering (blocks everything):** the page ships an empty root element and relies entirely on client-side JS. This needs developer work, not a content fix — talk to your web developer about server-side rendering (SSR), pre-rendering, or a static-HTML fallback so real content ships in the first response. Once that's fixed, re-run this check to evaluate the actual content.

## Scope & limits
This is a static-HTML visibility and structure check based only on the raw HTML fetched above. It does not measure whether any AI engine actually cites this page live — that requires ongoing citation monitoring and is out of scope. Findings reflect what was in the fetched bytes, nothing assumed.

Runtime & scope

• Posture: Diagnostic only. It fetches a single page as a non-JS crawler would and reports what's extractable; it never modifies, submits to, or writes anything to the site.
• Neutrality: Politically neutral — it evaluates structure and visibility, never content or positions.
• Runtime: Python 3.6+, standard library only. Optionally uses certifi as an HTTPS-certificate fallback if it happens to be installed; not required.