Crawlora

Crawlora — structured public web data

Crawlora is a hosted API that turns public websites into clean, normalized JSON. One API key gives you 733 endpoints across 61 platform groups — search engines, marketplaces, social and video, finance and crypto, maps, app stores, media, and reviews — so an agent can fetch real data without running a browser, proxies, or HTML parsers.

When to use this skill

Use Crawlora when the user asks for live data that lives on a public website, e.g.:

• "What's the price of X on Amazon / eBay / Shopify?"
• "Pull the transcript / comments of this YouTube video."
• "Get the latest Google / Bing results for …" or "what's trending on Google Trends?"
• "Reviews and ratings for this iOS / Android app."
• "Yahoo Finance quote / financials for NVDA", "top CoinGecko gainers".
• "TikTok / Instagram / Reddit posts about …", "Trustpilot reviews for …".
• Any task where the alternative would be scraping HTML or maintaining a crawler.

Prefer a more specific Crawlora skill if one is installed (e.g. product-price-research, youtube-research, app-review-mining, serp-keyword-research). This umbrella skill covers everything else.

Setup (one-time)

• Get a free Crawlora API key (2,000 credits/mo, no card) at https://crawlora.net.

• Export it so the helper and any curl calls can read it:

  export CRAWLORA_API_KEY=sk_your_key_here
  

• All requests go to https://api.crawlora.net/api/v1 with the header x-api-key: $CRAWLORA_API_KEY. A missing/invalid key returns 401.

How it works

1. Pick the endpoint. Open reference/catalog.md — it lists every endpoint grouped by platform, with method, path, and parameters. Match the user's job to a platform group, then to an endpoint.
2. Call it with scripts/crawlora.sh (see below). It handles auth and prints JSON.
3. Parse the JSON with jq and answer. Most endpoints return paginated lists; pass page/count params where documented.

Calling the API

# GET with query params (key=value):
scripts/crawlora.sh /amazon/search k=laptop | jq '.'
scripts/crawlora.sh /google/suggest q="web scraping" | jq '.'
scripts/crawlora.sh /youtube/transcript/dQw4w9WgXcQ | jq '.'

# POST endpoints take a JSON body (note the -X POST):
scripts/crawlora.sh -X POST /google/search '{"searchOption":{"q":"web scraping api"}}' | jq '.'
scripts/crawlora.sh -X POST /google/trends/explore/interest-over-time '{"keywords":["bitcoin"]}' | jq '.'

Raw curl fallback (no helper):

curl -fsS -H "x-api-key: $CRAWLORA_API_KEY" \
  "https://api.crawlora.net/api/v1/bing/search?q=web%20scraping" | jq '.'

The catalog marks each endpoint's HTTP method. Path params like {id} are substituted into the URL; GET params go in the query string; POST params go in the JSON body.

Endpoint reference

See reference/catalog.md for the full list of all 733 endpoints (method, path, params, description) grouped by platform.

Examples

• Product price: scripts/crawlora.sh /amazon/search k="standing desk" → read .results[].price to compare listings.
• YouTube transcript: resolve the video id from the URL, then scripts/crawlora.sh /youtube/transcript/<id> and summarize.
• Trend check: scripts/crawlora.sh -X POST /google/trends/explore/interest-over-time '{"keywords":["electric bikes"]}'.
• Finance: scripts/crawlora.sh /yahoo-finance/ticker/NVDA/quote (see the Yahoo Finance group in the catalog for the exact path/params).

Notes & limits

• Credits / pay-on-success: billed only on 2xx responses; failures aren't charged. Free tier is 2,000 credits/mo. Get a key at https://crawlora.net.
• Public data only. Crawlora returns publicly available web data; no third-party account credentials are involved. Respect each source's terms and rate limits.
• Security: keep the key in CRAWLORA_API_KEY only — never hardcode it, never put it in a URL query param, never commit it.
• Some search endpoints return 503 when the upstream serves a challenge page; retry or fall back to another search provider (Google ↔ Bing ↔ Brave).