Web scraping

Web Scraping Skill

1. Trigger Phrases

Activate this skill when the user says any of:

• "scrape", "extract data from", "crawl", "pull data from [website]"
• "monitor [site] for changes", "parse this page"
• "get all the [items] from [url]", "automated data extraction"
• "reverse engineer this API", "intercept mobile app traffic"

---

2. Pre-Scrape Analysis Gate

Mandatory before writing ANY scraping code. Do not skip steps.

Step 1: Check for a Public API

Google "[site name] API" or "[site name] developer docs"
IF public API exists with the data you need → use it. Done. No scraping needed.
Common examples with APIs: Twitter/X, Reddit, GitHub, Yelp, Google Maps, LinkedIn (unofficial)

Step 2: Inspect Network Traffic

Open target URL in browser → DevTools (F12) → Network tab → filter XHR/Fetch
Navigate the page, click buttons, scroll — watch for JSON responses
IF clean JSON endpoint found:
  → Copy the request as cURL (right-click → Copy as cURL)
  → Reverse-engineer it with HTTPX (see reference/api_discovery.md)
  → This is almost always faster and more reliable than HTML scraping

Step 2.5: Check for Embedded JSON State

Before classifying as SPA, fetch the raw HTML with HTTPX (plain GET with browser headers).
Search the HTML source for embedded data:
  - <script type="application/ld+json">     → JSON-LD structured data (products, org info, store details)
  - <script id="__NEXT_DATA__">              → Next.js hydration payload (entire page data)
  - window.__INITIAL_STATE__                 → Redux/Vuex pre-loaded state
  - window.__PRELOADED_STATE__               → Same pattern, different variable name
  - Any <script> tag containing a large JSON blob (>1KB)

IF found:
  → Verify the JSON contains the specific fields you need (not just minimal SEO metadata)
  → Extract with regex + json.loads — no browser needed
  → See reference/api_discovery.md (Embedded JSON State section)

Step 3: Check robots.txt

Fetch https://[domain]/robots.txt
Note:
  - Disallowed paths (don't scrape these unless user explicitly overrides)
  - Crawl-delay directive (respect it — sets minimum seconds between requests)
  - Sitemap URL (useful for discovering all pages)

Step 4: Classify the Target

Determine which category fits. The answer feeds directly into the Tool Selection Router.

Signal	Classification
view-source: shows the data in raw HTML	Static HTML
view-source: is mostly empty <div id="root">	JS-rendered SPA
Page requires login before showing data	Behind auth
Cloudflare challenge page, CAPTCHA, or instant 403	Protected / anti-bot
Data only available in mobile app	Mobile API
Content is unstructured text, not in tables/lists	Unstructured

---

3. Target Classification → Tool Selection Router

Read top to bottom. Use the first match.

IF public API exists
  → requests/HTTPX + API key
  → DONE — no scraping needed

IF hidden JSON endpoint found (Step 2)
  → HTTPX + direct endpoint
  → See reference/api_discovery.md

IF embedded JSON state found in HTML source (Step 2.5)
  → HTTPX + regex + json.loads
  → See reference/api_discovery.md (Embedded JSON State section)

IF static HTML + small scale (<100 pages)
  → BeautifulSoup + HTTPX
  → See reference/static_scraping.md

IF static HTML + large scale (>100 pages)
  → Scrapy
  → See reference/static_scraping.md

IF JS-rendered SPA
  → Playwright
  → See reference/dynamic_scraping.md

IF behind auth (login required)
  → Playwright with session cookies
  → See reference/dynamic_scraping.md

IF unstructured OR layout changes frequently
  → LLM extraction (markdownify + Claude/OpenAI)
  → See reference/llm_extraction.md

IF behind Cloudflare / DataDome / CAPTCHA
  → Evasion techniques first, then appropriate tool
  → See reference/evasion_mobile.md

IF mobile app only (no web version)
  → Mobile API interception
  → See reference/evasion_mobile.md

---

4. Extraction Workflow

Follow these steps in order. Each step has a checkpoint.

1. RUN PRE-SCRAPE ANALYSIS (Section 2)
   checkpoint: You know the target classification and selected tool

2. WRITE EXTRACTION SCRIPT
   → Use the reference doc for your selected tool
   → Apply operational basics to every script (reference/operational_basics.md):
     - Rate limiting (minimum 1s between requests unless API allows more)
     - Retry logic with backoff
     - User-Agent header (never use default python-requests UA)
   → Use starter script from templates/starter_scripts.md as base

3. TEST ON 5 ITEMS FIRST
   → Run against 5 URLs/items only
   → Print results to console for user review
   checkpoint: Output shape matches expectations, no blocks

4. IF BLOCKED → diagnose and escalate (Section 6)
   → Do NOT retry the same request blindly
   → Identify block type, apply fix, re-test on 5 items

5. SCALE TO FULL DATASET
   → Add progress logging (items scraped / total, elapsed time)
   → Save incrementally (don't lose data on crash)

6. VALIDATE OUTPUT (Section 5)

7. EXPORT to target format (JSON lines, CSV, or user-specified)

---

5. Output Validation Checklist

Run after extraction, before delivering to user.

[ ] SCHEMA CHECK
    Every row has expected fields? Correct types (string, int, URL)?
    → If field missing in >5% of rows, flag it

[ ] NULL/EMPTY CHECK
    Any field with >20% null rate?
    → Flag to user: "field X is null in 25% of rows — site may not have this data for all items"

[ ] DEDUP
    Duplicate rows by URL or unique identifier?
    → Remove dupes, report count: "Removed 12 duplicate entries"

[ ] ENCODING CHECK
    UTF-8 throughout? No mojibake (é, ’, etc.)?
    → If found, re-decode with correct charset

[ ] SAMPLE INSPECTION
    Print first 5 rows formatted for user review before delivering full dataset

---

6. Feedback Loop — Blocked Request Escalation

When a request returns 403, CAPTCHA, empty response, or redirect loop:

DIAGNOSE THE BLOCK:

├── 403 Forbidden
│   ├── Missing headers? → Add User-Agent, Accept, Accept-Language, Referer
│   ├── Still 403? → TLS fingerprint blocked
│   │   → Switch to tls-client (reference/evasion_mobile.md)
│   └── Still 403? → IP blocked → Add proxy rotation (reference/operational_basics.md)
│
├── CAPTCHA / Cloudflare challenge page
│   ├── Simple JS challenge? → Playwright with stealth plugin (reference/dynamic_scraping.md)
│   ├── Turnstile / hCaptcha? → Managed browser service (reference/evasion_mobile.md)
│   └── reCAPTCHA v3? → Likely need scraping API service (ScrapingBee, Bright Data)
│
├── Empty response / blank page
│   ├── JS-rendered? → Switch to Playwright (reference/dynamic_scraping.md)
│   ├── Geo-blocked? → Try proxy in target country
│   └── Rate limited silently? → Add delays, reduce concurrency
│
├── 429 Too Many Requests
│   ├── Increase delay between requests
│   ├── Add jitter (random 1-3s extra)
│   └── Rotate proxies if available (reference/operational_basics.md)
│
└── Redirect loop / soft block
    ├── Check if cookies are required → Use session with cookie persistence
    └── Check if JavaScript sets a cookie → Switch to Playwright

When to give up on DIY scraping:

• Cloudflare Enterprise with Turnstile + behavioral analysis → use a scraping API service
• Site actively patches evasion techniques weekly → use a scraping API service
• CAPTCHA on every request with no bypass → use a CAPTCHA-solving service or scraping API

Scraping API services (last resort): ScrapingBee, Bright Data, Oxylabs, ScraperAPI