Apify sdk patterns

Apify SDK Patterns

Overview

Production patterns for both the apify SDK (building Actors) and apify-client (calling Actors remotely). Covers Crawlee crawler selection, data storage, proxy configuration, and typed client wrappers. This skill gives you the essential skeletons inline; the full eight-pattern catalog and two worked scenarios live in references/ for progressive drill-down.

Prerequisites

• Install what you need: apify-client for calling Actors remotely, or apify + crawlee for building Actors. Both can coexist in one project.
• Set APIFY_TOKEN in the environment — read it via process.env.APIFY_TOKEN, never hard-code it. This is the only credential these patterns require (Apify uses a personal API token, not OAuth).
• TypeScript is recommended; every snippet here is typed and runs under ts-node or a compiled build.

Instructions

Use the two skeletons below to start, then reach into the reference catalog for the pattern that matches your task.

Pattern 1: Typed Client Singleton

Create one lazily-initialized, token-validated ApifyClient and reuse it everywhere. A resetClient() hook keeps it testable.

// src/apify/client.ts
import { ApifyClient } from 'apify-client';

let instance: ApifyClient | null = null;

export function getApifyClient(): ApifyClient {
  if (!instance) {
    const token = process.env.APIFY_TOKEN;
    if (!token) throw new Error('APIFY_TOKEN is required');
    instance = new ApifyClient({ token });
  }
  return instance;
}

// Reset for testing
export function resetClient(): void {
  instance = null;
}

Pattern 2: Crawlee Crawler Selection

Choose the crawler that matches the page, not the other way around:

import { CheerioCrawler, PlaywrightCrawler, PuppeteerCrawler } from 'crawlee';

// CHEERIO — Fast, lightweight, no JavaScript rendering
// Use for: static HTML, server-rendered pages, APIs
// PLAYWRIGHT — Full browser, all engines, modern API
// Use for: SPAs, JavaScript-heavy pages, complex interactions
// PUPPETEER — Chromium-only browser automation
// Use for: when you need Chromium specifically or legacy Puppeteer code

Patterns 3–8: Full catalog

The remaining six patterns are moved verbatim into patterns.md so this file stays scannable. Pick the one you need:

• Pattern 3 — Actor lifecycle with error handling: Actor.main() wrapping input validation, conditional proxy, and a failedRequestHandler.
• Pattern 4 — Dataset operations: push from inside an Actor; list/create/download from an external app.
• Pattern 5 — Key-value store operations: JSON config and binary artifacts by content type.
• Pattern 6 — Proxy configuration: datacenter vs residential vs SERP tiers.
• Pattern 7 — Router for multi-page Actors: labeled listing/detail handlers.
• Pattern 8 — Safe result wrapper: discriminated-union Result<T> around remote calls.

See patterns.md for the complete code of all six.

Output

Applying these patterns yields:

• A single reusable ApifyClient instance with fail-fast token validation.
• Actors that store structured records in the default dataset (downloadable as CSV/JSON) and named config/artifacts in key-value stores.
• Remote Actor calls that resolve to a typed Result<T> — callers branch on error instead of catching exceptions.
• Failed requests captured as { url, error, '#isFailed': true } rows rather than aborting the crawl.

Error Handling

Pattern	Use Case	Benefit
Actor.main()	Actor entry point	Auto init/exit + error reporting
failedRequestHandler	Per-request failures	Log failures without stopping crawl
Safe wrapper	External calls	Prevents uncaught exceptions
Router	Multi-page scrapes	Clean separation of page types
Proxy rotation	Anti-bot sites	Higher success rate

Examples

Two runnable end-to-end scenarios live in examples.md:

• Example A — Call a remote Actor safely from an app: composes the client singleton (Pattern 1) with the safe result wrapper (Pattern 8) to run apify/web-scraper and read its dataset without ever throwing.
• Example B — A two-tier product scraper Actor: composes crawler selection (Pattern 2), the router (Pattern 7), and dataset writes (Pattern 4) into a listing → detail → structured-record flow.

Resources

• Full pattern catalog — Patterns 3–8 with complete code
• Worked examples — two end-to-end scenarios
• Apify SDK Reference
• Crawlee Documentation
• Apify JS Client Reference
• Proxy Management Guide

Next Steps

Apply these patterns in apify-core-workflow-a for a complete build-and-deploy web scraping workflow, or pair them with apify-common-errors when hardening an Actor for production.