Apify scraper

Apify Scraper

Scrape websites and platforms using the Apify CLI. Discover actors (pre-built scrapers), inspect their input schemas, run them, and retrieve structured JSON results.

Mental Model

• Apify Store is a marketplace of 4,000+ pre-built scrapers (called "Actors") for every major platform — Instagram, TikTok, Amazon, Google, LinkedIn, Reddit, and more.
• apify CLI is your interface. It lets you inspect actors, run them remotely, and download results — all from the terminal.
• Each actor has its own input schema defining what parameters it accepts. You must read the schema before running any actor — even pre-vetted ones.

The Store is the catalog. Actors are the workers. The CLI is your remote control.

CLI Command Map

All outputs are JSON. Pipe through jq for extraction. Use 2>/dev/null to suppress CLI noise.

Phase	Command	Purpose
Discover	apify actors info <id> --input 2>/dev/null	Get actor's input schema as JSON
Execute	apify actors call <id> -i '<json>' --json 2>/dev/null	Run actor, wait, return run metadata JSON
Execute	apify actors start <id> -i '<json>' --json 2>/dev/null	Start actor async, return immediately
Retrieve	apify datasets get-items <datasetId> 2>/dev/null	Download a run's output dataset as JSON
Monitor	apify runs info <runId>	Check status of a run
Monitor	apify runs log <runId>	View logs from a run

Decision Tree

Read references/actors.yaml — is the target platform listed?
├─ YES → Read references/popular-actors.md for usage details
│        └─ STILL run apify actors info <id> --input first — schemas can change
│
├─ NO → Search the Apify Store (apify.com/store) for that platform
│   └─ Found an actor → apify actors info <id> --input → apify actors call
│
└─ General web page or search query?
    └─ Apify may not be the best tool — consider curl, Supadata, or Firecrawl

The Scraping Workflow

1. Identify the Target

What platform or website? What data type? (posts, profiles, comments, products, reviews)

2. Find the Right Actor

First: Check references/actors.yaml for the approved actor list. Then read references/popular-actors.md for detailed usage docs (example inputs, gotchas, output fields). If the target platform has an approved actor, use it.

If the platform isn't covered: Browse the Apify Store at apify.com/store or ask the user which actor to use.

3. Read the Input Schema

You must always run this before calling any actor. Even for actors listed in references/popular-actors.md — input schemas change over time, and calling with wrong input silently returns zero results or errors.

apify actors info apidojo/instagram-comments-scraper-api --input 2>/dev/null

This returns the full input schema as JSON. Pipe through jq to extract what you need:

List all properties with types and descriptions:

apify actors info apidojo/instagram-comments-scraper-api --input 2>/dev/null | jq '.properties | to_entries[] | {name: .key, type: .value.type, description: .value.description}'

Show just required fields:

apify actors info apidojo/instagram-comments-scraper-api --input 2>/dev/null | jq '{required, properties: .properties | map_values({type, description})}'

When reading the schema, pay attention to:

• Required vs optional fields — don't guess optional values, let defaults work
• Field formats — some actors want ["https://instagram.com/p/xyz"] (URL array), others want "username" (plain string)
• Limits — look for resultsLimit, maxItems, resultsPerPage to control output volume

4. Run the Actor

Build input JSON from the schema you just read. Run synchronously and extract the dataset ID:

apify actors call apidojo/instagram-scraper -i '{"startUrls":["https://www.instagram.com/username/"],"resultsPerPage":30}' --json 2>/dev/null | jq -r '.defaultDatasetId'

This returns just the dataset ID (e.g., namLUw6hWg3fAcC3R). Use it in step 5.

For the full run metadata (status, timing, cost):

apify actors call apidojo/instagram-scraper -i '{"startUrls":["https://www.instagram.com/username/"],"resultsPerPage":30}' --json 2>/dev/null | jq '{status, datasetId: .defaultDatasetId, duration: .stats.runTimeSecs}'

Asynchronous (for large crawls — returns immediately):

apify actors start apidojo/instagram-scraper -i '{"startUrls":["https://www.instagram.com/username/"],"resultsPerPage":30}' --json 2>/dev/null | jq -r '.id'

Returns the run ID. Check progress with apify runs info <runId>.

From a file (when input JSON is complex):

apify actors call apidojo/instagram-scraper -f input.json --json 2>/dev/null | jq -r '.defaultDatasetId'

5. Retrieve Results

Fetch the output dataset:

apify datasets get-items <datasetId> 2>/dev/null

Extract specific fields with jq:

apify datasets get-items <datasetId> 2>/dev/null | jq '[.[] | {url, caption, likeCount, commentCount}]'

Pagination for large datasets:

apify datasets get-items <datasetId> --limit 100 --offset 0 2>/dev/null

Alternative formats:

apify datasets get-items <datasetId> --format csv
apify datasets get-items <datasetId> --format xlsx

Monitoring Async Runs

Check on a run:

apify runs info <runId>

View logs (useful for debugging failures):

apify runs log <runId>

Common Pitfalls

Calling actors without checking the schema. Actor inputs are not standardized. One Instagram actor wants startUrls: ["https://..."], another wants username: "handle", a third wants profiles: ["handle"]. Always run apify actors info <id> --input 2>/dev/null first — even for actors you've used before.

The startUrls format trap. For actors that use startUrls, it must be a plain string array (["url"]), NOT an object array ([{"url": "..."}]). The object format silently returns zero results.

Over-fetching data. Many actors default to scraping everything — thousands of posts, all comments, full profile histories. Set explicit limits (resultsLimit, maxItems, etc.) to avoid long runtimes and high costs. Start small (10-30 items), verify the output looks right, then scale up.

Using -i with complex JSON in shell. Single quotes in JSON conflict with shell quoting. For complex inputs, write the JSON to a file and use -f input.json instead.

Reference Files

Read these on demand — don't load everything upfront:

File	When to Read
references/actors.yaml	Always read first — canonical list of approved actors (symlinked into projects at apify/actors.yaml)
references/popular-actors.md	Detailed usage docs — example inputs, gotchas, and output fields for each approved actor