Skip to main content
Use this tutorial if you do not yet know which Data App to use, or if you want the agent to find a capability for each task on its own. The general connection adds the Data Hub discovery and run tools to your agent. Once connected, you describe the data you need, and the agent searches for a suitable Data App, reviews its parameters, and then confirms and runs it.
The general connection does not require picking an app first. This tutorial and the “Codex: connect a specific app” and “Claude Code: connect a specific app” tutorials are two different usage patterns. A specific-app connection suits a clear, long-term, fixed capability. The general connection suits changing needs or cases where the app has not been chosen yet.

​
Understand the two connection methods

Both methods call the same Data Apps on Data Hub. The only difference is where you choose the app. You can keep both kinds of connection, but give them recognizable names so the agent does not pick the wrong tool.

​
What you can do afterwards

After this tutorial, you can do the following inside your agent, in order:
  1. Search for Data Apps by keyword, platform, or scenario.
  2. Review an app’s purpose, input parameters, output fields, and billing.
  3. Confirm the app and parameters, then start a run.
  4. Check the status of an asynchronous task and get the final result.
For the full parameters and handling rules of search_data_apps, run_data_app, asynchronous status checks, and the large-result handoff, see Data Hub MCP capabilities. This page focuses on connecting and first use.

​
Before you start

Have the following ready:
  • An Octoparse account you can sign in to.
  • An agent client that supports Streamable HTTP MCP, such as Claude Code, Codex, or Cursor.
  • If you choose the API key method: create an Octoparse API key in advance.
  • A simple data target for testing, such as “find a Data App that can enrich company information”.
An API key is an account credential. Never commit a real key to a code repository, shared config, public screenshot, or group chat. In OAuth mode, never copy a temporary access token from a browser session into a local config.

​
Step 1: open the general MCP connection

1

Go to the Data Hub Open Platform

Sign in to the Octoparse website and open the Data Hub menu at the top, then click Data Hub Open Platform.
2

Open MCP connection

In the left navigation of the Open Platform, click MCP connection to open the Data Hub MCP Server page. The default toolset shown on the page can search for and run any Data App on Data Hub. You do not need to choose an app from the catalog first.
The Integration section of any Data App also has an MCP connection link at the bottom that leads here. That link is a shortcut. It does not mean you have to choose that app first.

​
Step 2: choose an authentication method

The general connection supports both API key and OAuth. The choice affects only authentication. It does not change how Data Apps are searched or run. Best for long-term stable use, command-line clients, and automation. The configuration generated by the page contains:
Bearer is simply the header format used to pass the API key. Use an Octoparse API key, not a temporary access token from a signed-in browser session. If you do not have a key yet, create one in the Octoparse account center. An API key is usually shown in full only once, at creation time. Store it in a trusted password manager.

​
Option 2: OAuth sign-in

Best for interactive clients that support MCP OAuth. When you choose OAuth, the configuration contains no API key. The first time the client connects, it opens a browser where you sign in to Octoparse and confirm the authorization. The session may expire, in which case you authorize again.

​
Step 3: copy the general install prompt

On the MCP connection page, choose your agent client and authentication method, then click Copy install prompt. Copy MCP URL only gives you the server address. Copy the complete content the page generates right now. Do not type the server address, tool scope, or headers from memory.
The general server address shown on the page is based on https://mcp-v2.octoparse.com. The exact configuration and tool parameters may change, so always use what the Open Platform currently generates.

​
Step 4: let the agent complete the configuration

The example below uses Claude Code. Codex and Cursor place the buttons differently, but the core steps are the same: paste the install prompt, choose the authentication method, allow changes to the current user’s MCP configuration, then reload the client.
1

Send the install prompt to the agent

Start a new conversation and paste the complete prompt you copied. The agent should ask for the authentication method before writing the configuration. If it tries to write credentials into a project file, stop it and ask for the current user’s local MCP configuration instead.
2

Confirm the authentication method

If you chose API key, provide the key securely when the agent asks. If you chose OAuth, provide no key and let the agent write a configuration without credentials.
3

Reload the client

After configuration, reload the MCP servers or restart the client. With OAuth, the first connection may show “Needs authentication”. That is expected.

​
Step 5: complete OAuth authorization (OAuth only)

If you chose API key, skip to the next step.
1

Trigger the connection in the client

In Claude Code, run /mcp and select the Data Hub server you just added, named octoparse_datahub by default. When it shows that authentication is needed, choose to authenticate. Other clients show a Connect or similar button in their MCP settings.
2

Sign in and authorize in the browser

The browser opens the Octoparse identity page. Check the domain and the current account, read the requested scope, and confirm. Follow the page prompt to return to the client. Some clients return automatically.
3

Confirm the server is enabled

Back in the client, confirm the “Needs authentication” state is gone and the server is enabled. If it still shows as unauthenticated, reload the client and try again.

​
Step 6: search first, do not run yet

The key to the general connection is letting the agent discover Data Apps first. For the first use, explicitly ask it to search and compare only, and not to create a billable task yet. For example:
The agent usually calls search_data_apps to search the catalog, then get_data_app_details to read the full contract of each candidate.
The number, names, and platform coverage of the apps an agent finds change as the Data Hub catalog changes in real time. This documentation does not provide a fixed list. Rely on what search_data_apps actually returns.

​
Step 7: confirm the app, then run

Choose one app from the candidates. Have the agent restate the parameters and planned actions, then run a small test:
1

Check inputs and cost

Confirm the required parameters, data scope, returned fields, and billing unit. If anything is unclear, have the agent call the details tool again instead of guessing parameters.
2

Run a small test

Only after confirmation, let the agent call run_data_app. A synchronous app returns results directly. An asynchronous app needs follow-up status checks.
3

Get asynchronous results

For asynchronous tasks, have the agent use the status and result tools to wait for completion, then return the final data. Do not treat “task submitted” as a successful extraction.
4

Check the results

Check the task status, actual record count, and key fields. A missing field in a single record may reflect a difference in the source data. If most records do not match expectations, switch apps or adjust the parameters.

​
FAQ

No. The general connection provides the search and run tools first. After connecting, the agent uses search_data_apps to find capabilities. Only the specific-app connection requires choosing an app in Data Hub first.
Prefer API key for long-term use, command-line clients, and automation. Choose OAuth when you want to sign in through the browser and your client explicitly supports MCP OAuth. OAuth sessions may expire and need re-authorization.
Confirm you used the general configuration from the Open Platform MCP connection page, not the restricted configuration of a specific app. Copy the current prompt again and reload the client.
Trigger the connection in the client’s MCP settings, complete the browser sign-in and authorization, then return to the client. Check whether the browser blocked the redirect, and confirm that the authorization page belongs to the official Octoparse identity service.
Write “search and compare only, do not run yet” explicitly in the prompt. When cost or large data volumes are involved, ask the agent to wait for confirmation before calling run_data_app.
The app is probably asynchronous. Have the agent keep checking the task status and fetch the results when it completes. Do not resubmit the same task.

​
Checklist

  • You copied the general install prompt from the MCP connection page of the Data Hub Open Platform.
  • You chose either API key or OAuth and did not mix the two credentials.
  • The agent loaded the Data Hub general tools and can use search_data_apps.
  • You searched and reviewed app details before confirming a run.
  • You completed one real call with the smallest data volume and checked the final result.

​
Already know which app to use?

If your business uses a fixed app long term, you can narrow the connection scope:

Codex: connect a specific app

Pick a specific app first, then add it to Codex as a fixed tool.

Claude Code: connect a specific app

Pick a specific app first, then add it to Claude Code as a fixed MCP server.
The actual number of apps, their names, publishers, and prices returned by a search change as the Data Hub catalog changes. Rely on what the tool returns in each call.