Data acquisition browser

Skill Pranjay-kumar/universal-data-acquisition-pipeline-skill/skills/data-acquisition-browser

Codex/Claude skill for designing robust scraping, API, Playwright, and authorized data acquisition pipelines

Install

npx -y skills add Pranjay-kumar/universal-data-acquisition-pipeline-skill --skill data-acquisition-browser

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Use for Patchright/Playwright-based public or authorized browser probing: warm-session cookie/storage generation, browser network capture, JSON/API route discovery from page loads, rendered DOM fallback, screenshots, tiny DOM samples, and user-owned storage-state workflows. Do not use for CAPTCHA solving, credential extraction, auth bypass, or rate-limit bypass.

SKILL.md

4.7 KB, as published. Nobody here has run it

Data Acquisition Browser

Act as the browser acquisition specialist. Use Patchright for warm-session capture when a normal browser must mint cookies or storage state before API endpoints are visible. Use Playwright for ordinary rendered-DOM fallback when structured HTTP probes are insufficient and no warm browser context is needed.

When the user asks for page loads only or "no API", use Patchright as a renderer and extract DOM/JSON-LD/meta/visible rows only. Do not harvest, replay, or recommend structured endpoints in that mode.

Shared Core

Read from ../data-acquisition-core/references/:

source-access.md
playwright-rendered-dom.md
probing.md
compliance-boundaries.md
output-contracts.md

Helpers

Use scripts/patchright_cookie_endpoint_probe.mjs for warm-session cookie/storage generation and endpoint discovery. Use scripts/patchright_page_dom_probe.mjs for Patchright page-load-only DOM extraction. Use scripts/playwright_probe.mjs only for ordinary public rendered-DOM fallback.

From the repo root:

npm install
npx patchright install chrome
npm run probe:patchright -- "https://example.com/public-category" "outputs/example-patchright-endpoints.json"

The Patchright helper opens a persistent Chrome context, lets the page create ordinary browser-issued cookies/storage state, records JSON/API/XHR-looking requests and responses, saves local storage state under auth/, and writes a redacted endpoint report.

Page-load-only mode:

$env:PATCHRIGHT_HEADLESS = "0"
npm run probe:patchright-page -- "https://example.com/category" "outputs/example-page-dom.json"

This mode records rendered page metadata, JSON-LD, canonical URL, visible text, candidate links, product/listing-like DOM nodes, and a screenshot. It sets api_endpoint_discovery: false and replay_attempted: false.

Useful options:

$env:PATCHRIGHT_STORAGE_STATE = "auth\target-storage-state.json"
$env:PATCHRIGHT_USER_DATA_DIR = "auth\target-profile"
$env:PATCHRIGHT_HEADLESS = "0"
npm run probe:patchright -- "https://example.com/category" "outputs/target-endpoints.json"

Playwright fallback:

npx playwright install chromium
npm run probe:playwright -- "https://example.com/public-page" "outputs/example-playwright-probe.json"

Never print or commit storage state, cookies, account pages, or local browser profiles.

Autonomous Browser Fallback Rules

Use this ladder for tiny feasibility probes:

Try headless only when it is likely to behave like an ordinary public browser.
If headless returns a block page but the user allows browser probing, retry once with Patchright non-headless and a persistent local profile.
If non-headless loads the public page, extract a sample of at most 20 visible rows and classify the route as browser-context dependent.
If non-headless also returns access denied, CAPTCHA, login, or verification, stop. Do not add stealth, CAPTCHA solving, identity rotation, or bypass logic.
If the user requested no API, use probe:patchright-page; do not run endpoint discovery or replay.

Warm Session Capture

Use this when a normal public page, such as a retail category page, issues browser-context XHR/fetch requests that cold HTTP probes cannot reproduce.

Capture locally:

user-owned browser cookies/storage state generated by Patchright
observed endpoint URL and query params
safe request headers such as Accept, Accept-Language, Origin, and Referer
response field shape, pagination params, and sample rows

Do not capture or publish cookies, auth tokens, CAPTCHA tokens, account state, cart state, or fingerprint material.

Required sequence:

Run npm run probe:patchright against the public page or explicitly authorized owned-session page.
Review endpoint_candidates and network.structured_* in the report.
Convert promising URLs into EndpointPlan templates with path params, query params, pagination, and safe headers.
Replay only tiny probes in the same user-owned context when cookies/storage are required.
Mark the source access as owned_session and output label as non_public_authorized_result whenever replay depends on local state.

See ../data-acquisition-core/references/execution-adapters.md for optional runtime adapters, including Jacob Padilla's Stealth-Requests and Google-Colab-Selenium projects.