Crawlee skill

Crawlee OpenCode Skill

Crawlee is a scalable web crawling and scraping library for Node.js and TypeScript. It helps you build reliable crawlers that appear human-like and fly under the radar of modern bot protections even with default configuration.

Quick Start

Prerequisites

• Node.js 16 or higher

With Crawlee CLI (Recommended)

npx crawlee create my-crawler
cd my-crawler
npm start

Manual Installation

npm install crawlee playwright

import { PlaywrightCrawler, Dataset } from 'crawlee';

const crawler = new PlaywrightCrawler({
    async requestHandler({ request, page, enqueueLinks, log }) {
        const title = await page.title();
        log.info(`Title of ${request.loadedUrl} is '${title}'`);

        await Dataset.pushData({ title, url: request.loadedUrl });
        await enqueueLinks();
    },
});

await crawler.run(['https://crawlee.dev']);

Overview

Crawlee covers your crawling and scraping end-to-end and provides tools to:

• Crawl the web for links
• Scrape data from websites
• Store extracted data to disk or cloud
• Configure behavior to suit your project's needs

Key Features

• Single interface for HTTP and headless browser crawling
• Persistent queue for URLs to crawl (breadth & depth first)
• Pluggable storage of both tabular data and files
• Automatic scaling with available system resources
• Integrated proxy rotation and session management
• Lifecycles customizable with hooks
• CLI to bootstrap your projects
• Configurable routing, error handling and retries
• Dockerfiles ready to deploy
• Written in TypeScript with generics

Installation

Using CLI

The fastest way to try Crawlee is using the CLI:

npx crawlee create my-crawler
cd my-crawler
npm start

The CLI will install all necessary dependencies and add boilerplate code for you.

Manual Installation

If adding Crawlee to your own project:

npm install crawlee playwright

Note: Playwright is not bundled with Crawlee to reduce install size. You can also use Puppeteer:

npm install crawlee puppeteer

Installing Pre-release Versions

For testing new features:

npm install crawlee@next

If using Apify SDK, specify dependency overrides:

{
    "overrides": {
        "apify": {
            "@crawlee/core": "$crawlee",
            "@crawlee/types": "$crawlee",
            "@crawlee/utils": "$crawlee"
        }
    }
}

Usage

Basic Crawler

import { PlaywrightCrawler, Dataset } from 'crawlee';

const crawler = new PlaywrightCrawler({
    async requestHandler({ page, request, log }) {
        const title = await page.title();
        log.info(`Crawled: ${title}`);
        await Dataset.pushData({ title, url: request.loadedUrl });
    },
});

await crawler.run(['https://example.com']);

HTTP Crawler

For simple HTTP requests without browser:

import { HttpCrawler, Dataset } from 'crawlee';

const crawler = new HttpCrawler({
    async requestHandler({ request, body }) {
        // body is the HTML content
        await Dataset.pushData({ url: request.loadedUrl, html: body });
    },
});

await crawler.run(['https://example.com']);

Cheerio Crawler

Fast jQuery-like HTML parsing:

import { CheerioCrawler, Dataset } from 'crawlee';

const crawler = new CheerioCrawler({
    async requestHandler({ request, $, log }) {
        const title = $('title').text();
        log.info(`Title: ${title}`);
        await Dataset.pushData({ title, url: request.loadedUrl });
    },
});

await crawler.run(['https://example.com']);

Enqueue Links

Crawl multiple pages automatically:

const crawler = new PlaywrightCrawler({
    async requestHandler({ enqueueLinks }) {
        // Extract and enqueue all links from current page
        await enqueueLinks();
    },
});

Custom Link Selection

await enqueueLinks({
    selector: 'a.product-link',
    baseUrl: 'https://example.com',
});

Adding URLs Manually

// Add single URL
await crawler.addRequests(['https://example.com/page-1']);

// Add multiple URLs
await crawler.addRequests([
    'https://example.com/page-1',
    'https://example.com/page-2',
]);

API Reference

Main Crawler Classes

PlaywrightCrawler

Headless browser crawling using Playwright.

import { PlaywrightCrawler } from 'crawlee';

const crawler = new PlaywrightCrawler({
    headless: true,
    browserPoolOptions: {
        maxOpenPagesPerBrowser: 10,
    },
    requestHandler: async ({ page, request }) => {
        // Your scraping logic
    },
});

PuppeteerCrawler

Headless browser crawling using Puppeteer.

import { PuppeteerCrawler } from 'crawlee';

const crawler = new PuppeteerCrawler({
    headless: true,
    requestHandler: async ({ page, request }) => {
        // Your scraping logic
    },
});

CheerioCrawler

Fast HTML parsing without browser.

import { CheerioCrawler } from 'crawlee';

const crawler = new CheerioCrawler({
    requestHandler: async ({ $, request }) => {
        // $ is Cheerio instance
        $('a').each((i, el) => {
            console.log($(el).text());
        });
    },
});

HttpCrawler

Simple HTTP requests with fast HTML parsing.

import { HttpCrawler } from 'crawlee';

const crawler = new HttpCrawler({
    requestHandler: async ({ body, request }) => {
        // body is raw HTML
    },
});

JSDOMCrawler

Browser-like environment using JSDOM.

import { JSDOMCrawler } from 'crawlee';

const crawler = new JSDOMCrawler({
    requestHandler: async ({ window, document }) => {
        const title = document.title;
    },
});

Storage Classes

Dataset

Store scraped data:

import { Dataset } from 'crawlee';

// Push data to default dataset
await Dataset.pushData({ name: 'Product A', price: 99 });

// Export to file
await Dataset.exportToCSV('output.csv');

KeyValueStore

Store key-value pairs:

import { KeyValueStore } from 'crawlee';

await KeyValueStore.setValue('state', { page: 1 });
const state = await KeyValueStore.getValue('state');

RequestQueue

Manage crawling queue:

import { RequestQueue } from 'crawlee';

const queue = await RequestQueue.open();
await queue.addRequest({ url: 'https://example.com' });
const request = await queue.fetchNextRequest();

Configuration

Basic Configuration

const crawler = new PlaywrightCrawler({
    // Concurrency
    maxConcurrency: 10,

    // Retries
    maxRequestRetries: 3,

    // Request timeout
    requestHandlerTimeoutSecs: 30,

    // Navigation timeout
    navigationTimeoutSecs: 30,
});

Proxy Configuration

const crawler = new PlaywrightCrawler({
    proxyConfiguration: new ProxyConfiguration({
        proxyUrls: [
            'http://proxy1.com:8000',
            'http://proxy2.com:8000',
        ],
    }),
});

Session Configuration

const crawler = new PlaywrightCrawler({
    sessionPoolOptions: {
        maxPoolSize: 100,
        sessionOptions: {
            maxUsageCount: 10,
        },
    },
});

HTTP/2 Configuration

const crawler = new HttpCrawler({
    http2: true,
    // Additional HTTP2 options
});

Request Handler Options

The request handler receives these parameters:

• request - Request information (URL, headers, userData)
• page - Browser page instance (Puppeteer/Playwright)
• $ - Cheerio instance (CheerioCrawler)
• body - HTML content (HttpCrawler)
• window - JSDOM window (JSDOMCrawler)
• enqueueLinks - Function to add links to queue
• log - Logger instance
• sendRequest - Make additional HTTP requests

Development

Project Structure

my-crawler/
├── package.json
├── tsconfig.json
└── src/
    └── main.ts

Running Tests

# Run all tests
npm test

# Run tests in watch mode
npm run test:watch

# Run e2e tests
npm run test:e2e

Building

# Build project
npm run build

# Build for production
npm run ci:build

Linting

# Run linter
npm run lint

# Fix linting issues
npm run lint:fix

Formatting

# Format code
npm run format

# Check formatting
npm run format:check

Docker Deployment

Create Dockerfile:

FROM node:18-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
CMD ["npm", "start"]

Troubleshooting

Bot Protection

Crawlee includes anti-blocking features:

const crawler = new PlaywrightCrawler({
    // Browser-like headers are automatic
    // TLS fingerprinting is automatic
    // Human-like behavior is automatic
});

Rate Limiting

Use session and request delay:

const crawler = new PlaywrightCrawler({
    navigationTimeoutSecs: 30,
    maxRequestRetries: 3,
    sessionPoolOptions: {
        maxPoolSize: 50,
    },
});

Memory Issues

Reduce concurrency:

const crawler = new PlaywrightCrawler({
    maxConcurrency: 5,
    browserPoolOptions: {
        maxOpenPagesPerBrowser: 5,
    },
});

Timeout Errors

Increase timeout values:

const crawler = new PlaywrightCrawler({
    requestHandlerTimeoutSecs: 60,
    navigationTimeoutSecs: 60,
});

Storage Location

Default storage is ./storage directory. Change via:

import { Configuration } from 'crawlee';

Configuration.set('storageDir', './my-storage');

Resources

Official Resources

• Documentation: https://crawlee.dev
• GitHub: https://github.com/apify/crawlee
• NPM: https://www.npmjs.com/package/crawlee
• Discord: https://discord.gg/jyEM2PRvMU
• Stack Overflow: https://stackoverflow.com/questions/tagged/apify

Related Tools

• Apify Platform: https://apify.com
• Apify SDK: https://sdk.apify.com
• Crawlee for Python: https://github.com/apify/crawlee-python

Examples Repository

Check out the docs/examples/ directory in the GitHub repo for code examples covering:

• Basic crawling
• Multiple URL crawling
• Link extraction
• File downloads
• Forms submission
• Sitemap crawling
• And more...

Best Practices

1. Start Simple: Begin with HTTPCrawler or CheerioCrawler for faster scraping
2. Use Session Management: Enable sessions to avoid IP blocking
3. Set Timeouts: Configure appropriate timeouts for target sites
4. Handle Errors: Implement proper error handling and retries
5. Respect Robots: Follow website robots.txt and rate limits
6. Use Proxies: Rotate proxies for large-scale scraping
7. Monitor Resources: Track memory and CPU usage
8. Clean Data: Validate and clean scraped data before storage

License

Apache License 2.0 - See LICENSE.md for details.