Data acquisition browser

Data Acquisition Browser

Act as the browser acquisition specialist. Use Patchright for warm-session capture when a normal browser must mint cookies or storage state before API endpoints are visible. Use Playwright for ordinary rendered-DOM fallback when structured HTTP probes are insufficient and no warm browser context is needed.

When the user asks for page loads only or "no API", use Patchright as a renderer and extract DOM/JSON-LD/meta/visible rows only. Do not harvest, replay, or recommend structured endpoints in that mode.

Shared Core

Read from ../data-acquisition-core/references/:

• source-access.md
• playwright-rendered-dom.md
• probing.md
• compliance-boundaries.md
• output-contracts.md

Helpers

Use scripts/patchright_cookie_endpoint_probe.mjs for warm-session cookie/storage generation and endpoint discovery. Use scripts/patchright_page_dom_probe.mjs for Patchright page-load-only DOM extraction. Use scripts/playwright_probe.mjs only for ordinary public rendered-DOM fallback.

From the repo root:

npm install
npx patchright install chrome
npm run probe:patchright -- "https://example.com/public-category" "outputs/example-patchright-endpoints.json"

The Patchright helper opens a persistent Chrome context, lets the page create ordinary browser-issued cookies/storage state, records JSON/API/XHR-looking requests and responses, saves local storage state under auth/, and writes a redacted endpoint report.

Page-load-only mode:

$env:PATCHRIGHT_HEADLESS = "0"
npm run probe:patchright-page -- "https://example.com/category" "outputs/example-page-dom.json"

This mode records rendered page metadata, JSON-LD, canonical URL, visible text, candidate links, product/listing-like DOM nodes, and a screenshot. It sets api_endpoint_discovery: false and replay_attempted: false.

Useful options:

$env:PATCHRIGHT_STORAGE_STATE = "auth\target-storage-state.json"
$env:PATCHRIGHT_USER_DATA_DIR = "auth\target-profile"
$env:PATCHRIGHT_HEADLESS = "0"
npm run probe:patchright -- "https://example.com/category" "outputs/target-endpoints.json"

Playwright fallback:

npx playwright install chromium
npm run probe:playwright -- "https://example.com/public-page" "outputs/example-playwright-probe.json"

Never print or commit storage state, cookies, account pages, or local browser profiles.

Autonomous Browser Fallback Rules

Use this ladder for tiny feasibility probes:

1. Try headless only when it is likely to behave like an ordinary public browser.
2. If headless returns a block page but the user allows browser probing, retry once with Patchright non-headless and a persistent local profile.
3. If non-headless loads the public page, extract a sample of at most 20 visible rows and classify the route as browser-context dependent.
4. If non-headless also returns access denied, CAPTCHA, login, or verification, stop. Do not add stealth, CAPTCHA solving, identity rotation, or bypass logic.
5. If the user requested no API, use probe:patchright-page; do not run endpoint discovery or replay.

Warm Session Capture

Use this when a normal public page, such as a retail category page, issues browser-context XHR/fetch requests that cold HTTP probes cannot reproduce.

Capture locally:

• user-owned browser cookies/storage state generated by Patchright
• observed endpoint URL and query params
• safe request headers such as Accept, Accept-Language, Origin, and Referer
• response field shape, pagination params, and sample rows

Do not capture or publish cookies, auth tokens, CAPTCHA tokens, account state, cart state, or fingerprint material.

Required sequence:

1. Run npm run probe:patchright against the public page or explicitly authorized owned-session page.
2. Review endpoint_candidates and network.structured_* in the report.
3. Convert promising URLs into EndpointPlan templates with path params, query params, pagination, and safe headers.
4. Replay only tiny probes in the same user-owned context when cookies/storage are required.
5. Mark the source access as owned_session and output label as non_public_authorized_result whenever replay depends on local state.

See ../data-acquisition-core/references/execution-adapters.md for optional runtime adapters, including Jacob Padilla's Stealth-Requests and Google-Colab-Selenium projects.