Cloud browser automation

Cloud Browser Automation

Use cloud browser services (Browserbase, Browser Use Cloud) for web scraping when local tools fail due to Cloudflare protection, JavaScript rendering requirements, or anti-bot measures.

Usage Strategy

CRITICAL: On-demand only, close immediately

Always follow this pattern:

1. Create session ONLY when needed
2. Do the task (scrape/navigate/extract)
3. Close session IMMEDIATELY after done
4. Never leave sessions running idle

Why:

• Save credits (pay per minute)
• Avoid hitting concurrent session limits (e.g., Browserbase max 3)
• Clean resource usage
• No waste

When to Use

Escalation ladder (always try cheapest/fastest first):

1. Try Hermes web_fetch first
   ↓ (if fails or empty content)
2. Try direct curl with proper headers (Pattern 0 — works for most static sites)
   ↓ (if blocked or needs JS)
3. Try Hermes browser
   ↓ (if blocked by Cloudflare or sandbox issues)
4. Use cloud browser (Browserbase)

Use cloud browser when:

• ✅ Cloudflare Turnstile blocking requests
• ✅ JavaScript-heavy sites (SPA, dynamic content)
• ✅ Anti-bot detection (fingerprinting, TLS checks)
• ✅ Complex multi-step automation (login, navigate, extract)

DON'T use cloud browser when:

• ❌ Simple HTTP requests (use web_fetch)
• ❌ Static content (use web_fetch)
• ❌ Cost-sensitive tasks (cloud = pay per minute)

Browserbase Setup

API Configuration

# Credentials (saved in memory)
API_KEY="bb_live_O6tVgdzl8B6WquBSj1XpuaC5hMc"
PROJECT_ID="a5008864-bbaa-4966-96e2-2272497b003d"
BASE_URL="https://www.browserbase.com/v1"

# Limits
MAX_CONCURRENT_SESSIONS=3
SESSION_TIMEOUT=5  # minutes (default)

Session Management

CRITICAL USER PREFERENCE:

"gini jadi lo pakai browserbase saat dibutuhkan aja, nah sesi langsung close saat beres"

Translation: Use on-demand only, close immediately after done.

Why:

• ✅ Save credits (pay per minute)
• ✅ Avoid hitting concurrent limit (max 3)
• ✅ Clean resource usage
• ✅ No waste

Pattern:

# 1. Create session ONLY when needed
session = create_session()

# 2. Do the task
data = scrape_page(session)

# 3. Close IMMEDIATELY after done
close_session(session)

DON'T:

• ❌ Create session at start of script "just in case"
• ❌ Leave sessions running between tasks
• ❌ Reuse sessions across multiple unrelated tasks
• ❌ Forget to close sessions

Workflow

Step 1: Create Session

curl -X POST "https://www.browserbase.com/v1/sessions" \
  -H "X-BB-API-Key: $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "projectId": "'$PROJECT_ID'",
    "browserSettings": {
      "viewport": {"width": 1920, "height": 1080}
    }
  }'

Response:

{
  "id": "session-id-here",
  "status": "RUNNING",
  "connectUrl": "wss://connect-v2.usw2.browserbase.com/?signingKey=...",
  "seleniumRemoteUrl": "http://connect-v2.usw2.browserbase.com/webdriver",
  "expiresAt": "2026-05-03T13:00:00.000Z"
}

Step 2: Automate (Requires SDK)

Option A: Playwright (Python)

from playwright.async_api import async_playwright

async with async_playwright() as p:
    browser = await p.chromium.connect_over_cdp(connect_url)
    page = await browser.new_page()
    await page.goto("https://example.com")
    data = await page.evaluate("() => document.body.innerText")
    await browser.close()

Option B: Selenium (Python)

from selenium import webdriver

options = webdriver.ChromeOptions()
driver = webdriver.Remote(
    command_executor=selenium_remote_url,
    options=options
)
driver.get("https://example.com")
data = driver.find_element(By.TAG_NAME, "body").text
driver.quit()

Option C: Manual (Fallback)

# Get debug URL
curl "https://www.browserbase.com/v1/sessions/$SESSION_ID/debug" \
  -H "X-BB-API-Key: $API_KEY"

# Open debug URL in browser
# Navigate manually
# Screenshot/extract data manually

Step 3: Close Session

curl -X POST "https://www.browserbase.com/v1/sessions/$SESSION_ID/stop" \
  -H "X-BB-API-Key: $API_KEY"

ALWAYS close, even if task fails:

try:
    data = scrape_with_browserbase()
finally:
    close_session()  # Always close!

VPS Constraints

Problem: Ubuntu 24.04 strict package management blocks pip install playwright/selenium.

Solutions:

Option 1: Use Docker (Recommended)

# Run Playwright in container
docker run -it mcr.microsoft.com/playwright/python:v1.40.0-jammy \
  python3 /tmp/scraper.py

Pros: Isolated, no pip conflicts Cons: Need Docker (~500MB)

Option 2: Manual Fallback

# Create session
SESSION_ID=$(curl -X POST ... | jq -r '.id')

# Get debug URL
DEBUG_URL=$(curl ... | jq -r '.debuggerFullscreenUrl')

# User opens URL manually
echo "Open: $DEBUG_URL"
echo "Navigate to target site"
echo "Screenshot and send to me"

# Close session
curl -X POST ".../sessions/$SESSION_ID/stop" ...

Option 3: Delegate to Subagent

# If automation blocked, delegate research
delegate_task(
    goal="Research [topic] via web search",
    toolsets=["web", "terminal"]
)

Common Patterns

Pattern 0: Direct Curl for Static Sites (No Browserbase)

Challenge: Site returns HTML but might have Cloudflare or anti-bot checks

Solution:

# Direct curl with proper headers (often bypasses basic protection)
curl -s -L -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" \
  -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" \
  -H "Accept-Language: en-US,en;q=0.5" \
  -H "Connection: keep-alive" \
  -H "Upgrade-Insecure-Requests: 1" \
  "https://example.com" > /tmp/page.html

# If response is gzip-encoded, check file type first
file /tmp/page.html  # Should show "HTML document" not "gzip compressed"

# Extract data with grep/sed/python
grep -oP '<title>\K[^<]+' /tmp/page.html

When to use:

• ✅ Static HTML sites (marketing pages, landing pages)
• ✅ Sites with basic Cloudflare (not Turnstile)
• ✅ Public content (no login required)
• ✅ VPS without Docker/Playwright

When to escalate:

• ❌ Cloudflare Turnstile challenge appears
• ❌ JavaScript-heavy SPA (empty body in curl response)
• ❌ Login-required content

Pattern 0b: TinyFish Browser for Cloudflare Bypass (Recommended)

Challenge: API returns Cloudflare challenge page, need full browser automation

Solution:

import requests
import asyncio
import websockets
import json
import time

TINYFISH_API_KEY = "your-api-key"

async def fetch_via_tinyfish(url):
    # 1. Create browser session
    headers = {"X-API-Key": TINYFISH_API_KEY, "Content-Type": "application/json"}
    payload = {
        "url": url,
        "timeout_seconds": 300,
        "browser_profile": "stealth",
        "proxy_config": {"enabled": True, "type": "tetra", "country_code": "US"}
    }
    resp = requests.post("https://api.browser.tinyfish.ai", headers=headers, json=payload, timeout=60)
    
    info = resp.json()
    session_id = info["session_id"]
    base_url = info["base_url"].rstrip("/")
    
    # 2. Wait for Cloudflare bypass (CRITICAL: 30 attempts × 6 seconds)
    page_ws_url = None
    for attempt in range(30):  # Not 20!
        time.sleep(6)  # Not 5!
        try:
            pages_resp = requests.get(f"{base_url}/pages", timeout=20)
            pages = pages_resp.json()
            
            if pages:
                page = pages[0]
                title = page.get("title", "")
                url_check = page.get("url", "")
                
                # Check BOTH title AND url (not just title!)
                if "Just a moment" not in title and url in url_check:
                    page_ws_url = page.get("webSocketDebuggerUrl")
                    break
        except:
            pass
    
    if not page_ws_url:
        requests.delete(f"https://api.browser.tinyfish.ai/{session_id}", headers={"X-API-Key": TINYFISH_API_KEY})
        raise Exception("Failed to bypass Cloudflare after 30 attempts")
    
    # 3. Execute via CDP
    async with websockets.connect(page_ws_url, max_size=20_000_000) as ws:
        counter = 0
        
        async def send(method, params=None):
            nonlocal counter
            counter += 1
            msg_id = counter
            await ws.send(json.dumps({"id": msg_id, "method": method, "params": params or {}}))
            while True:
                msg = json.loads(await ws.recv())
                if msg.get("id") == msg_id:
                    return msg
        
        await send("Runtime.enable")
        
        result = await send("Runtime.evaluate", {
            "expression": """
                fetch('/api/endpoint')
                .then(r => r.json())
                .catch(e => ({error: String(e)}))
            """,
            "awaitPromise": True,
            "returnByValue": True
        })
        
        data = result.get("result", {}).get("result", {}).get("value")
    
    # 4. Close session immediately
    requests.delete(f"https://api.browser.tinyfish.ai/{session_id}", headers={"X-API-Key": TINYFISH_API_KEY})
    
    return data

# Usage
data = asyncio.run(fetch_via_tinyfish("https://example.com"))

CRITICAL Cloudflare Bypass Settings:

• 30 attempts (not 20) — Cloudflare can take 2-3 minutes
• 6 second delay (not 5) — Faster polling causes false negatives
• Check both title AND url — Title alone insufficient for SPA sites
• max_size=20_000_000 for websocket — Large responses need bigger buffer

When to use:

• ✅ Cloudflare Turnstile v2 (most reliable bypass)
• ✅ JavaScript-heavy SPA (full browser context)
• ✅ Login-required content (can handle auth flows)
• ✅ Production monitoring (stable, proven)

When to escalate to Browserbase:

• ❌ Need persistent sessions (TinyFish auto-closes)
• ❌ Need screenshots/video recording
• ❌ Complex multi-page workflows (TinyFish better for single-page scrapes)

Real-world example: RPOW2Swap API monitoring (see references/rpow2swap-cloudflare-bypass.md)

Pattern 0a: Analyzing Minified JS Bundles

If curl returns a SPA with minified JavaScript, you can often extract API patterns without Browserbase. See references/minified-js-analysis.md for grep-based techniques to find:

• API endpoints and base URLs
• Data structures (orderbook, pricing, etc.)
• Authentication patterns (localStorage tokens, headers)
• WebSocket URLs

This works for most modern SPAs (React, Vue, Next.js) and costs zero resources vs Browserbase.

Pattern 1: Twitter/X Scraping (No Browserbase)

Challenge: Need to read public tweet content

Solution:

# Direct curl scraping (faster + free)
curl -s -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" \
  "https://x.com/USERNAME/status/TWEET_ID" > /tmp/tweet.html

# Extract JSON data
grep -o 'window.__INITIAL_STATE__={.*}' /tmp/tweet.html | \
  sed 's/window.__INITIAL_STATE__=//' > /tmp/tweet.json

# Parse with jq or Python
jq '.entities.tweets.entities' /tmp/tweet.json

When to use:

• ✅ Public tweets (no login required)
• ✅ Simple text/media extraction
• ✅ Cost-sensitive tasks

When to escalate to Browserbase:

• ❌ Rate limited (429 errors)
• ❌ Login-required content
• ❌ Need to interact (like, retweet, reply)

See references/twitter-scraping-fallback.md for full technique.

Pattern 1: OpenSea NFT Scraping

Challenge: OpenSea blocks direct HTTP (403 Cloudflare)

Solution:

# 1. Create session
session = create_browserbase_session()

# 2. Navigate with Playwright
page = await connect_to_session(session)
await page.goto("https://opensea.io/collection/xyz")
await page.wait_for_selector('[data-testid="floor-price"]')

# 3. Extract data
floor_price = await page.text_content('[data-testid="floor-price"]')
volume = await page.text_content('[data-testid="volume"]')

# 4. Close immediately
await browser.close()
close_browserbase_session(session)

Pattern 2: Cloudflare-Protected Sites

Challenge: Site uses Cloudflare Turnstile

Solution:

# Browserbase automatically solves Cloudflare challenges
# Just navigate normally, no special handling needed

page = await connect_to_session(session)
await page.goto("https://cloudflare-protected-site.com")
# Browserbase handles Turnstile automatically
await page.wait_for_load_state("networkidle")
content = await page.content()

Pattern 3: JavaScript-Heavy SPAs

Challenge: Content loaded via JavaScript, web_fetch returns empty

Solution:

# Wait for content to load
page = await connect_to_session(session)
await page.goto("https://spa-site.com")
await page.wait_for_selector('.content-loaded')  # Wait for JS
data = await page.evaluate("() => window.appData")

Error Handling

Rate Limit (429)

Error:

{
  "statusCode": 429,
  "error": "Too Many Requests",
  "message": "You've exceeded your max concurrent sessions limit (limit 3, currently 3)"
}

Solution:

# List running sessions
curl "https://www.browserbase.com/v1/sessions" \
  -H "X-BB-API-Key: $API_KEY" \
  | jq '.[] | select(.status=="RUNNING") | .id'

# Close old sessions
for sid in $(list_running_sessions); do
  curl -X POST ".../sessions/$sid/stop" ...
done

Session Expired

Error: Session expires after 5 minutes (default)

Solution:

• Create session right before use (not at script start)
• Use keepAlive: true for longer tasks
• Close and recreate if task takes > 5 min

SDK Not Available (VPS)

Error: pip install playwright blocked by Ubuntu 24.04

Solution:

• Use Docker (Option 1)
• Use manual fallback (Option 2)
• Delegate to subagent with web search (Option 3)

Cost Optimization

Browserbase pricing: Pay per minute of session time

Optimization strategies:

1. Create session late

   # ❌ BAD: Create at start
   session = create_session()
   # ... 2 minutes of prep work ...
   scrape(session)  # Wasted 2 minutes!
   
   # ✅ GOOD: Create right before use
   # ... prep work ...
   session = create_session()
   scrape(session)  # No waste!
   close_session(session)
   

2. Close immediately

   # ❌ BAD: Leave running
   session = create_session()
   scrape(session)
   # ... session still running ...
   
   # ✅ GOOD: Close ASAP
   session = create_session()
   scrape(session)
   close_session(session)  # Immediately!
   

3. Batch operations

   # ❌ BAD: One session per page
   for url in urls:
       session = create_session()
       scrape(url, session)
       close_session(session)
   
   # ✅ GOOD: One session for all pages
   session = create_session()
   for url in urls:
       scrape(url, session)
   close_session(session)
   

4. Use escalation ladder

   # Try cheap methods first
   try:
       data = web_fetch(url)  # Free, fast
   except CloudflareError:
       try:
           data = hermes_browser(url)  # Free, slower
       except SandboxError:
           session = create_browserbase_session()  # Paid, last resort
           data = scrape_with_browserbase(url, session)
           close_session(session)
   

Pitfalls

1. Forgetting to Close Sessions

Problem: Sessions left running, hit rate limit, waste credits

Solution:

• Always use try/finally to ensure closure
• Close in error handlers
• Verify closure with API call

2. Creating Sessions Too Early

Problem: Session created at script start, expires before use

Solution:

• Create session right before scraping
• Not at script initialization

3. Not Checking Rate Limits

Problem: Hit 429 error, script fails

Solution:

• Check running sessions before creating new one
• Close old sessions if at limit
• Implement retry with exponential backoff

4. Using for Simple Tasks

Problem: Use Browserbase for static sites, waste money

Solution:

• Always try web_fetch first
• Only escalate to Browserbase when needed
• Follow escalation ladder

5. Not Handling SDK Unavailability

Problem: Assume Playwright/Selenium available, script fails on VPS

Solution:

• Check if SDK available before using
• Have fallback plan (Docker, manual, delegate)
• Document VPS constraints in skill

Verification

After using Browserbase:

# Verify session closed
curl "https://www.browserbase.com/v1/sessions" \
  -H "X-BB-API-Key: $API_KEY" \
  | jq '.[] | select(.status=="RUNNING")'

# Should return empty or sessions you intentionally left running

# Check data extracted
cat /tmp/scraped_data.json
jq '.' /tmp/scraped_data.json  # Validate JSON

# Verify files cleaned
ls /tmp/browserbase* /tmp/scrape*
# Should be empty or cleaned up

References

• Browserbase Docs: https://docs.browserbase.com
• Playwright Python: https://playwright.dev/python/
• Selenium Python: https://selenium-python.readthedocs.io/
• TinyFish Integration: references/tinyfish-integration.md (REST API, not MCP)