Scrappycoco

Scrappycoco

Scrappycoco is deterministic external-data infrastructure, not an AI agent. All AI/LLM reasoning stays in the user's agentic client (such as Cursor, Codex, or Claude). That calling agent interprets the goal, chooses and evaluates providers, judges results, and decides whether to retry. Use the pinned CLI with npx --yes @scrappycoco/[email protected].

Simple path

For most tasks:

1. If the capability is unknown, list the live catalog: npx --yes @scrappycoco/[email protected] catalog list --available --json
2. Inspect the relevant contract: npx --yes @scrappycoco/[email protected] catalog inspect source.capability --json
3. Run it and save the records: npx --yes @scrappycoco/[email protected] run source.capability --file request.json --output results.json --json
4. Judge whether the records contain the facts the user asked for, then return the result file and a concise completeness note.

This is the default workflow: Catalog → Inspect → Run. Do not create a Discovery unless provider or configuration choice is genuinely uncertain.

Two actions

• Run executes a known capability or finalized configuration. Prefer it whenever the source, capability, and configuration are clear.
• Discover runs uncertain provider configurations on the same sample and returns comparable evidence. The calling agent selects the winner and saves a reusable scraper.

Catalog and authentication commands support these actions; they are not extra workflows.

What it covers

The live catalog includes public-web search, extraction, crawling, and link mapping; X and Reddit search, profiles, conversations, and monitoring; and company filings and issuer monitoring. Availability changes, so inspect the live catalog instead of relying on this summary. See the recipes when a concrete example would help.

Connect

For an explicit installation request, run:

npx --yes @scrappycoco/cli@latest setup

Setup opens browser OAuth, installs this skill, and verifies the catalog. The terminal—not the browser callback—is authoritative. Do not report success until terminal verification completes. During ordinary data work, do not clone repositories, install packages persistently, or change client configuration.

If authentication is missing, direct the user to setup. Never ask the user to paste an API key, OAuth callback, or token into chat. Use only an existing SCRAPPYCOCO_API_KEY for noninteractive access. Before handling headless login, skill updates, timeouts, network failures, or REST fallback, read operations. Use npx --yes @scrappycoco/[email protected] doctor --json only for troubleshooting. Update instructions only when explicitly requested with npx --yes @scrappycoco/[email protected] skill update --json.

Run

1. Inspect the current capability schema; never guess fields or options.
2. Put canonical inputs under input and provider-native options under provider_options.<provider_id>.
3. Treat the provider and its native options as one configuration. Choose options deliberately from options_schema; output format, rendering, waits, actions, locale, and other native settings can determine whether the scrape works.
4. Verify that requested native representations are present in each returned record's outputs map and that primary_format matches the chosen primary format. Do not normalize away HTML, Markdown, structured JSON, links, or another representation the user requested.
5. Omit the provider to use the curated primary provider with one bounded compatible fallback. Specify providers when quality or native options matter.
6. Review scope and pricing internally, then run without asking for a budget, credit ceiling, or execution confirmation.
7. Use a new idempotency key for every billable request. Reuse it only for an identical transport retry.

For web.extract_content, provide exactly one of input.url or input.urls. Use urls for batches of up to 500 pages instead of launching concurrent CLI processes. Inspect explicit truncation metadata.

--output supports JSON, JSONL, and CSV. It writes the records to the file and prints a compact execution summary to stdout. Return the file to the user.

Run normally waits for its durable job. For a run expected to take long, add --detach; it returns a job ID immediately. Finish it with:

npx --yes @scrappycoco/[email protected] jobs wait JOB_ID --output results.json --json

Use jobs get JOB_ID --json for a single status check. Do not combine --detach with --output. Use jobs cancel JOB_ID --json to stop a queued or running job.

Discover

Before authoring or repairing a Discovery, read and follow the Discovery workbook.

1. Choose the acquisition method before providers. Prefer a documented API, JSON endpoint, feed, or static file over rendered-page scraping.
2. Normalize one representative input and define the requested-field rubric.
3. Use routing: "compare" when uncertain. A candidate is the provider plus its native options, not merely a provider ID. The test must include every currently available provider, plus materially different configurations of one provider when format, rendering, actions, waits, or other options could change usefulness.
4. Save with discover --file discovery.json --json, then test with discover --id ID --test --input '...' --json.
5. Inspect every candidate-specific provider_results, exact attempt.provider_options, primary_format, and native outputs. Provider status ok means only that the request completed; the calling agent judges usefulness.
6. If all candidates fail, repair input/configuration or pivot source or capability. Do not generalize from an incomplete candidate set.
7. Update the Discovery with the chosen explicit provider and the exact tested options, including output format, then finalize it. Run never silently re-discovers or changes finalized providers.

Handoff

Lead with the result:

1. State captured versus expected items and requested-field completeness.
2. Attach the useful output file.
3. Surface missing fields, partial failures, truncation, freshness, geography, or other material caveats.
4. When finalized and validated, give the reusable scraper a user-facing name.
5. End with a compact technical note containing request/Discovery ID, source, provider and meaningful options, item/failure counts, truncation, usage, and fallbacks.

Never claim Scrappycoco was used without response evidence. Do not dump raw logs or narrate every probe. After a successful bounded scrape, offer relevant scale-up choices, but do not expand scope until the user selects one.

Monitoring

x.monitor, reddit.monitor, and filings.monitor_issuer are cursor-based. Save the cursor and records. On the next requested or scheduled check, pass the cursor in input.since. The calling agent owns recurrence and alert logic.

Handle failures

Inspect every item and provider attempt. Retry only failed items when a retry is appropriate. For validation, authentication, billing, access, conflict, rate-limit, provider, timeout, and network failures, follow operations. Do not retry unchanged invalid input.

MCP fallback

Connected Scrappycoco MCP tools are an equivalent transport. Apply the same schema-first, Run-by-default, agent-owned-judgment rules. Do not require MCP when the CLI works.

Direct API fallback

Use the REST API only when the CLI cannot run and an existing SCRAPPYCOCO_API_KEY is available. Follow operations for safe headers, job polling, idempotency, and the single recovery path when no key is configured.