Apify core workflow a

Apify Core Workflow A — Build & Deploy a Scraper

Overview

End-to-end workflow: define input schema, build a Crawlee-based Actor, extract structured data, store results in datasets, test locally, and deploy to Apify platform. This is the primary money-path workflow for Apify.

Prerequisites

• npm install apify crawlee in your project
• npm install -g apify-cli and apify login completed
• For programmatic retrieval (Step 6), an API token in APIFY_TOKEN — read it from the environment (process.env.APIFY_TOKEN), never hard-code it
• Familiarity with apify-sdk-patterns

Instructions

Step 1: Define Input Schema

Create .actor/INPUT_SCHEMA.json:

{
  "title": "E-Commerce Scraper",
  "type": "object",
  "schemaVersion": 1,
  "properties": {
    "startUrls": {
      "title": "Start URLs",
      "type": "array",
      "description": "Product listing page URLs to scrape",
      "editor": "requestListSources",
      "prefill": [{ "url": "https://example-store.com/products" }]
    },
    "maxItems": {
      "title": "Max items",
      "type": "integer",
      "description": "Maximum number of products to scrape",
      "default": 100,
      "minimum": 1,
      "maximum": 10000
    },
    "proxyConfig": {
      "title": "Proxy configuration",
      "type": "object",
      "description": "Select proxy to use",
      "editor": "proxy",
      "default": { "useApifyProxy": true }
    }
  },
  "required": ["startUrls"]
}

Step 2: Build the Actor with Router Pattern

Use a Crawlee router that splits handling by page type: the default handler enqueues product links + pagination from listing pages, and a PRODUCT-labeled handler extracts structured fields from detail pages. The entry point wires proxy config, concurrency, a failed-request handler, and a run summary into the key-value store. Skeleton:

// src/main.ts
import { Actor } from 'apify';
import { CheerioCrawler, createCheerioRouter, Dataset, log } from 'crawlee';

const router = createCheerioRouter();
router.addDefaultHandler(async ({ enqueueLinks }) => {
  await enqueueLinks({ selector: 'a.product-card', label: 'PRODUCT' });
  await enqueueLinks({ selector: 'a.next-page', label: 'LISTING' });
});
router.addHandler('PRODUCT', async ({ request, $ }) => {
  await Actor.pushData({ url: request.url, name: $('h1.product-title').text().trim() });
});

await Actor.main(async () => {
  const input = await Actor.getInput();
  const crawler = new CheerioCrawler({ requestHandler: router, maxRequestsPerCrawl: input?.maxItems ?? 100 });
  await crawler.run(input.startUrls.map(s => s.url));
});

The full typed Actor — Product/ProductInput interfaces, proxy configuration, failedRequestHandler, and the SUMMARY key-value write — is in implementation.md, Step 2.

Step 3: Configure Dockerfile

Use the apify/actor-node:20 base with a two-stage build (compile TypeScript in a builder stage, ship only dist/ + production deps). Full Dockerfile: implementation.md, Step 3.

Step 4: Test Locally

# Create test input
mkdir -p storage/key_value_stores/default
echo '{"startUrls":[{"url":"https://example.com"}],"maxItems":5}' \
  > storage/key_value_stores/default/INPUT.json

# Run locally
apify run

# Check results
ls storage/datasets/default/
cat storage/key_value_stores/default/SUMMARY.json

Step 5: Deploy to Apify Platform

# Push to Apify (creates Actor if it doesn't exist)
apify push

# Or push to a specific Actor
apify push username/my-actor

# Run on platform
apify actors call username/my-actor

Step 6: Retrieve Results Programmatically

From any client, use the apify-client SDK to call the deployed Actor, list its dataset items, and download results (JSON/CSV). The token comes from process.env.APIFY_TOKEN — never hard-code it. Full retrieval code: implementation.md, Step 6.

Output

• Deployable Actor with typed input schema
• Router-based crawler handling listing + detail pages
• Structured product data in default dataset
• Run summary in default key-value store
• Failed requests tracked with error messages

Error Handling

Error	Cause	Solution
Actor build failed	Dockerfile/deps issue	Check build logs on platform
Selector returns empty	Page structure changed	Update CSS selectors
maxRequestsPerCrawl hit	Too many pages enqueued	Increase limit or filter URLs
Proxy errors	Anti-bot blocking	Switch to residential proxy
TIMED-OUT status	Actor exceeded timeout	Increase timeout or reduce scope

Examples

A quick example — seed a local input, run the Actor, and check results:

mkdir -p storage/key_value_stores/default
echo '{"startUrls":[{"url":"https://example-store.com/products"}],"maxItems":5}' \
  > storage/key_value_stores/default/INPUT.json
apify run
cat storage/key_value_stores/default/SUMMARY.json

Three fuller worked scenarios live in examples.md:

• Scrape a catalog locally, then deploy — the full seed → apify run → inspect → apify push loop, with the expected SUMMARY.json output.
• Run the deployed Actor and export CSV — call the Actor via apify-client and download the dataset as CSV.
• Route through residential proxy — pass a proxyConfig group at run time to get past anti-bot blocking.

Resources

• Crawlee Quick Start
• Actor Deployment
• Input Schema Spec
• Full implementation walkthrough — complete Actor source, Dockerfile, and retrieval code
• Worked examples — three end-to-end run scenarios

Next Steps

Once your Actor is deployed and producing data, move on to dataset and key-value store management — pagination over large datasets, deduplication, exporting to external stores, and scheduling recurring runs — covered in apify-core-workflow-b.