Scrape

Skill onfire7777/universal-ai-skills-library/skills/scrape

Scrape any webpage as clean markdown via Bright Data Web Unlocker API. Bypasses bot detection and CAPTCHA. Requires BRIGHTDATA_API_KEY and BRIGHTDATA_UNLOCKER_ZONE environment variables.From its SKILL.md

Install

npx -y skills add onfire7777/universal-ai-skills-library --skill scrape

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

14 stars14 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its file declares

Copied from the file, not written here

The file declares its own license as Unspecified. That is the author’s claim about this one file, and it is not the same thing as the license GitHub reports for the repository, which is listed with the other numbers below.

SKILL.md

1.5 KB, 321 tokens by cl100k_base, as published. Nobody here has run it

Bright Data - Web Scraper

Scrape any webpage and get clean markdown content using Bright Data's Web Unlocker API. Automatically bypasses bot detection and CAPTCHA.

Setup

1. Get your API Key: Get a key from Bright Data Dashboard.

2. Create a Web Unlocker zone: Create a zone at brightdata.com/cp by clicking "Add" (top-right), selecting "Unlocker zone".

3. Set environment variables:

export BRIGHTDATA_API_KEY="your-api-key"
export BRIGHTDATA_UNLOCKER_ZONE="your-zone-name"

Usage

bash scripts/scrape.sh "url"

Parameters:

url (required): The webpage URL to scrape

Examples:

# Scrape a news article
bash scripts/scrape.sh "https://example.com/article"

# Scrape a product page
bash scripts/scrape.sh "https://shop.example.com/product/123"

Output Format

Returns clean markdown content extracted from the webpage:

# Page Title

Main content of the page converted to markdown format...

## Section Heading

More content...

Features

Bot Detection Bypass: Automatically handles anti-bot measures
CAPTCHA Solving: Bypasses CAPTCHA challenges
Clean Markdown: Returns well-formatted markdown content
JavaScript Rendering: Handles JavaScript-heavy pages

Dependencies

curl - For API requests

What ships with it: 1 file

864 B alongside SKILL.md, 1 of them executable

scripts/

scrape.shruns864 B