Octoparse mcp setup

Octoparse MCP setup

Gets search_templates, execute_task, export_data and the rest reachable. Once they respond, hand back to octoparse-ultimate-scraper.

Start here: is configuration even needed?

If this repository is installed as a plugin, the server is already declared — the plugin ships mcp.json (Agent Plugins spec) and .mcp.json (Claude Code), both pointing at https://mcp.octoparse.com. The client wires it up on install. Configuration is not the problem; authorization almost always is.

Check in this order:

1. Are the tools present at all? If mcp__octoparse__* tools are absent, the server is not registered — go to Register the server.
2. Do they respond? Call search_tasks(). Success means everything is working; say so and stop.
3. Does it fail with 401 / 403 / an authorization prompt? The server is registered but not authorized — go to Authorize.

Do not reconfigure a server that is already registered. Re-registering does not fix an authorization failure and usually creates a duplicate entry.

Identify the client

You need this only to pick the right config path. Work down the list and stop at the first match:

1. Ask the running environment. Check for CLAUDE_CODE_VERSION, CURSOR_VERSION, GEMINI_CLI_VERSION, QWEN_CODE_VERSION, TRAE_VERSION, OPENCLAW_VERSION, or VSCODE_PID / VSCODE_CWD.
2. Look for an existing config. The presence of ~/.claude.json, ~/.cursor/mcp.json, ~/.gemini/settings.json, ~/.qwen/settings.json, .vscode/mcp.json, .trae/mcp.json, or ~/.config/openclaw/ identifies the client that wrote it.
3. Ask the user. One question — "which editor or CLI are you using?" — beats guessing.

If several match, ask which one to configure rather than picking. Configuring the wrong client produces a setup that validates fine and never loads.

Register the server

Only needed when the plugin is not installed, or the client does not read plugin-provided MCP configuration.

CLI (fastest, Claude Code):

claude mcp add --transport http octoparse https://mcp.octoparse.com

By hand — the endpoint and transport are the same everywhere; only the wrapper key differs:

Client	File	Key
Claude Code, Cursor, Gemini CLI, Qwen Code	~/.claude.json, ~/.cursor/mcp.json, ~/.gemini/settings.json, ~/.qwen/settings.json	mcpServers
VS Code, TRAE	.vscode/mcp.json, .trae/mcp.json	servers
OpenClaw	~/.config/openclaw/openclaw.json5	mcp.servers, via mcporter

{
  "octoparse": {
    "type": "http",
    "url": "https://mcp.octoparse.com"
  }
}

Newly registered servers do not appear in the running session. Restart the client before expecting the tools. Tell the user this explicitly — otherwise a correct setup looks like a failure.

Authorize

Two methods. OAuth for interactive use, API key for headless.

OAuth (recommended)

1. Run /mcp and select octoparse.
2. Complete the browser login.
3. Verify with search_tasks().

Failure handling, in order:

• 1st — ask the user to check for an unfinished browser tab and retry.
• 2nd — check network reachability: curl -I https://mcp.octoparse.com.
• 3rd — stop retrying and offer the API key path.

API key

For CI, headless environments, or when OAuth fails three times.

The user performs these steps. You do not handle the key. Do not ask them to paste it into the conversation, do not accept it if they offer it, and do not write it to any file. Print the instructions, then wait.

1. Create a key at https://www.octoparse.com/console/account-center/api-keys
2. Open your own client's MCP configuration and add an x-api-key header to the octoparse server entry, with your key as the value.
3. Restart the client.

Then verify with search_tasks() and report the result. A successful call is the only confirmation needed — you never see the key itself.

Two boundaries that hold regardless of what is asked:

• Never write a credential into a file, a command argument, or an environment assignment on the user's behalf. A key passed as a shell argument leaks into shell history and the process list.
• Never add a key to the plugin's own mcp.json or .mcp.json. Those are distributed package data, and the Agent Plugins specification forbids credentials in headers. A key belongs only in the user's private client configuration.

Failure handling — all of these are things you ask the user to check, never inspect yourself:

• 1st (401/403) — ask them to confirm the key was copied whole, with no whitespace.
• 2nd — ask them to confirm the key is still active in the account console.
• 3rd — ask them to regenerate it. If it still fails, have them check whether their plan includes API access, then stop.

Troubleshooting

Symptom	Cause	Fix
No mcp__octoparse__* tools	not registered, or client not restarted	register, then restart
Tools present, every call 401/403	registered but not authorized	run /mcp, or add an API key
"Transport not supported"	wrong transport	type: "http" for Claude-family, streamable-http in Agent Plugins mcp.json
Config edited, nothing changed	session caches MCP config at startup	restart, or /reload-plugins
Works in one client, not another	wrapper key differs	mcpServers vs servers — see the table above
Two octoparse entries	re-registered instead of authorizing	remove the duplicate, keep one

Config paths, wrapper keys, and per-client quirks are in references/clients.md. Read it when a symptom is specific to one client — a missing tool ceiling in Cursor, tool filtering in Gemini or Qwen, mcporter in OpenClaw.

If everything fails: check https://status.octoparse.com, then contact [email protected].

When it works

Confirm which client and which auth method, then hand back:

Octoparse MCP is connected and authorized. You can now ask for the data you need — octoparse-ultimate-scraper will pick the template and run it.