Cdp graphql scraper

Logged-in CDP + response-interception scraper

Most scraping breaks because it fights the site: it re-logs-in with brittle form automation, parses HTML that changes weekly, and fires requests fast enough to trip bot defenses. This skill teaches a calmer approach that survives far longer:

Attach to a browser you already control, and read what it's already saying. You launch your normal Chrome (with your real, human-established session), Playwright connects to it over CDP without opening a new browser, and you simply listen to the JSON the site's own frontend fetches as you navigate. No login automation, no HTML parsing, no synthetic API calls the server can fingerprint.

Authorized use only

This technique is for data you have a legitimate right to collect: your own accounts, sites whose Terms of Service permit it, APIs you're licensed for, or data you've been explicitly authorized to gather. Before scraping a third-party service, check its ToS and robots.txt, and keep request volume low enough that you never degrade the service for others. Automated collection can also violate a platform's ToS and get the account banned even when the IP is fine — so use a throwaway/dedicated account for anything against a service you don't own, never a personal or important one. When in doubt, prefer an official/paid API.

Why response interception beats DOM scraping

A modern web app renders from JSON it fetches over fetch/XHR — often a GraphQL endpoint. That JSON is richer, more stable, and cheaper to parse than the DOM:

• Stable: field names in an API payload change far less often than CSS/DOM.
• Complete: the payload usually carries more than the page shows (counts, ids, flags, pagination cursors).
• Cheap: no waiting on layout, no fragile selectors, no screenshot OCR.

So the game is: navigate like a human, and catch the responses in a listener.

Setup: attach to your own Chrome over CDP

Launch Chrome with remote debugging before running the script, log in normally (as a human — solve any captcha yourself, once), and leave it open:

# Windows
chrome.exe --remote-debugging-port=9222 --user-data-dir="C:\chrome-debug"
# macOS
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222

Verify it's reachable — this JSON means you're good:

curl http://127.0.0.1:9222/json/version

Then connect over CDP instead of launching a fresh browser — this reuses your real session, cookies, and fingerprint:

browser = await p.chromium.connect_over_cdp("http://127.0.0.1:9222")
context = browser.contexts[0]          # your existing window
page = context.pages[0]                # reuse an open tab
page.on("response", handle_response)   # <- the whole trick

A dedicated --user-data-dir keeps this debug profile separate from your day-to-day browsing.

The interception pattern

Register one response handler, filter to the endpoints you care about, and pull the fields out defensively (sites ship multiple response shapes simultaneously — old and new schema, logged-in vs. logged-out variants):

CAPTURED = {}   # keyed by whatever id you're collecting

async def handle_response(response):
    url = response.url
    if "/api/graphql" not in url and "/graphql" not in url:
        return
    try:
        text = await response.text()
        if not text.strip():
            return
        if looks_blocked(text, url, response.status):
            print("🚨 block signal — stopping")
            return
        data = json.loads(text)
    except Exception:
        return                     # partial/binary/non-JSON — ignore, don't crash
    node = dig(data, ...)          # tolerate several key paths, not just one
    if node:
        CAPTURED[node["id"]] = normalize(node)

Key habits: never let a single bad response throw; try old and new key paths before giving up; and treat an empty 200 as "gated," not "done."

scripts/cdp_scraper_template.py is a complete, runnable skeleton of this — queue, listener, delays, resume, and output. Start from it and fill in the two site-specific hooks (URL_MATCH and extract).

Anti-block hygiene (the part naive scrapers skip)

A scraper that fires as fast as it can gets soft-blocked within minutes. Pace it like a person and it can run for hours:

• Human-paced delays between navigations — randomized, not a fixed interval (e.g. random.uniform(8, 20) seconds). Fixed cadence is itself a bot tell.
• Batch cooldowns — after every N targets, sleep 60s+. Bursts look robotic.
• Soft-block auto-stop — if the expected response fails to appear several times in a row (e.g. 3 empty/absent captures), the site is likely throttling you. Stop on your own and resume later; pushing through deepens the block.
• Hard-block detection — treat HTTP 429, checkpoint_required, challenge URLs, or "action blocked" text as a full stop and alert the user; these precede an account/IP suspension.
• One crawler at a time — never run two scrapers against the same account concurrently. They share the session cookie, so the site sees double the rate and blocks you faster. If multiple agents/people share the repo, confirm who owns the run before starting.
• Protect the IP if needed — for larger jobs, a rotating residential/mobile proxy spreads requests across IPs. (Rotation protects the IP, not the account — a burner account is still what protects the account.)

Resumability (so a block or crash costs you nothing)

Record each successfully-collected id to a small state file and skip it on the next run. Then "resume" is just "run the same command again":

done = set(json.load(open(DONE, encoding="utf-8"))) if os.path.exists(DONE) else set()
queue = [h for h in targets if h not in done]
# ... after each success:
done.add(h); save(done)

Save after every success, and save again in a finally: block so a mid-run crash still persists progress.

⚠ One subtle trap: don't corrupt the state file with a BOM. On Windows, writing this JSON with PowerShell Out-File/Set-Content prepends a UTF-8 BOM, and a naive json.load then reads zero completed items — so the next run re-scrapes everything and hammers the site. Always write state with Python (json.dump(..., open(path, "w", encoding="utf-8"))), never a shell redirect.

Common runtime pitfalls

Symptom	Cause → fix
Script dies instantly, no logs	The debug Chrome was closed → relaunch it, rerun (resume handles the rest)
No output when run in background	Python buffered stdout → run with python -u
Emoji/Unicode crash on Windows	cp949 console → run with python -X utf8 (or sys.stdout.reconfigure(encoding="utf-8"))
Whole queue re-runs from scratch	BOM in the state file (see above) → rewrite it with Python, not a shell
Expected response never captured	Wrong URL filter, or the site gates that endpoint when logged-out → verify you're logged in and check the real URL in DevTools → Network
Empty 200 responses	Soft-block or gating, not "no data" → back off and retry later

Reference files

• scripts/cdp_scraper_template.py — a complete, generic scraper skeleton (CDP attach, response listener, human delays, batch cooldown, block detection, resumable state, JSONL output). Copy it and implement URL_MATCH + extract.