Scraper contract

Scraper Contract

A scraper rarely fails by crashing. It fails by returning nothing, or half a record, while the pipeline downstream carries on as if the data were real. A week later you're emailing "Hi {first_name}" because one field came back null and nobody decided that was a failure.

This skill writes the contract that stops that. You describe what you're scraping and from where, it produces a contract that classifies every failure, treats empty as broken, and says which fields you're allowed to trust. It's the wrapper I put around every scraper before it goes near a pipeline.

Build the contract, then stop.

Inputs (ask for whatever is missing)

• The target (required): the site/source and the fields you need off it.
• How you're fetching: Apify / Playwright / a fetch-and-parse / an API, and whether the source is one you control.
• Optional: a real example of a good record, and the worst page you've hit (paywalled, JS-rendered, captcha'd).

The method

1. Define the failure taxonomy. A closed enum of failure codes, each mapped from what you can observe. The set that has covered every real case for me:

   • unreachable — DNS/connection failure, 404/502/503/504, ENOTFOUND, ECONNRESET.
   • blocked — 403, captcha, "access denied", a bot wall.
   • timeout — the request or render exceeded its budget.
   • rate-limited — 429, or the source's throttle response.
   • invalid-input — the URL/target was malformed before you even left.
   • empty-content — the fetch succeeded but the thing you came for isn't there.
   • unknown — everything else, logged with the raw cause so the taxonomy can grow. Write the classify rule: status code and message substrings in, one code out. Every failure gets a code; "it just didn't work" is not allowed.

2. Make empty a hard failure. This is the one everyone skips. A response with no markdown, an empty list, a missing required field is empty-content, and it throws or returns a failure — never a success with blank data. "No content" being a normal outcome is how garbage enters the pipeline silently.

3. Tier every field by stability. For each field you extract, mark how fragile it is:

   • stable — a reliable selector / API field unlikely to move.
   • extracted — pulled by the model from prose; resilient to layout change but needs a sanity check.
   • volatile — a brittle selector or a layout-dependent grab that will break on the next redesign. The volatile fields are your decay risk. They get the canary in step 5.

4. Set the minimum identifier. State the smallest set of fields a record must have to be accepted at all — for a person, at least a profile URL or an email; for a company, at least a name or domain. A record missing the minimum is dropped, not patched with guesses. Validate this before the record is allowed into the pipeline.

5. Add a decay canary per volatile field. A cheap assertion that fires when the shape changes: a field that's suddenly null across the whole batch means the site changed, not that everyone lacks it. Pair it with the escalation rule — on blocked/rate-limited, back off (and escalate to a proxy/stealth fetch only if it's worth it); don't hammer.

6. Keep the batch alive. Wrap each record in its own try/catch so one malformed row never kills the run. Collect the failures with their codes instead of throwing at the first one.

7. Define a broken run. The threshold that flips the whole run from "fine" to "stop and alert": e.g. ">30% empty-content or any blocked = the source changed or shut us out, halt and tell me." Without this line, a scraper that returns 90% nulls looks like a successful run.

8. Request only what you need. Pull the expensive formats (rendered HTML, screenshots) only when a field actually requires them, and let an optional extra (a screenshot) fail soft — warn and proceed — without failing the core record. Cost and fragility both drop.

Output

Produce the scrape contract as: a field table (field, stability tier, required-for-minimum?, canary check), the failure enum with its classify rules, the broken-run threshold, and the fetch economy notes (which formats, what degrades soft). Flag every volatile field with no canary as a decay risk to fix.

Then stop. The contract is the deliverable. The scraper you build against it will tell you when it breaks, instead of quietly feeding you nothing.