How cloud tasks are split into subtasks

Octoparse can divide an eligible cloud task into smaller subtasks. Each subtask handles part of the input and uses one cloud node while it is running. Several subtasks can run at the same time, so a suitable task can finish sooner than one sequential run. Splitting is based on the task’s workflow and input structure. It is not a promise that every task will use every node in your account. Octoparse splits the first eligible list it encounters in the workflow. A later list is not split again as a separate layer, so nesting several list loops does not multiply the number of subtasks indefinitely.

​

Which tasks can be split

The most common splittable loop modes are: Single-element loops normally cannot be divided into independent work:

• Single element: the loop contains one element to process.

Variable List uses a fixed three-subtask split rather than scaling the number of subtasks with the list size. If you need more parallelism and the item structure is stable, consider converting the workflow to a Fixed List or a URL List. If a task has no splittable loop, it runs as one execution unit even when the account has several cloud nodes.

​

How splitting and node allocation work

The process is:

1. Octoparse identifies an eligible loop and its input items.
2. It creates subtasks for portions of those items.
3. Each available cloud node takes one subtask at a time.
4. When a node finishes, another waiting subtask can use it.
5. The run finishes after all subtasks complete or stop.

The number of subtasks can be greater than the number of available nodes. For example, a list with 100 URLs may produce many subtasks while a six-node account runs only up to six of them concurrently. The remaining subtasks wait for a node to become available.

​

Subtask limits and input limits

Current 9.x configurations use different subtask limits for different task types: For OTD tasks, the input method also affects the split limit. Enter manually supports up to 200 subtasks. Import from file, Batch generate, and Import from task can use up to 1,000 subtasks. Enter-manually, file-import, and batch-generated URL input is limited to 1 million URLs; Import from task can exceed that input size but remains limited to 1,000 subtasks. More nodes can increase parallel capacity, but they do not automatically make every task faster. The actual result also depends on the number of input items, task concurrency, page loading time, waits, login state, target-site limits, CAPTCHA, and queue availability.

​

What splitting changes for a run

Splitting affects both speed and troubleshooting:

• Speed: independent subtasks can process different inputs at the same time.
• Capacity: each active subtask consumes one cloud node, leaving fewer nodes for other tasks.
• Progress: subtasks can start and finish at different times.
• Recovery: one stopped subtask may affect only part of the input, but its records still need to be checked before restarting it.
• Output: duplicate handling and incomplete subtasks can make the final saved count lower than the number of records attempted.

Splitting does not remove dependencies inside a subtask. If each page must be opened after the previous page, or all items require one shared login session, parallel execution may provide little benefit or may make the target site less stable.

​

How to make a task more suitable for splitting

Use independent inputs whenever possible:

1. Build a URL List, Text List, or Fixed List loop with clear items.
2. Keep each item self-contained so a subtask can process it without another subtask’s state.
3. For a multi-page workflow, consider scraping detail-page URLs first, then using those URLs in a separate detail task.
4. If the three-subtask Variable List limit is the bottleneck, convert it to a Fixed List or URL List only when the item positions or URLs are known and stable.
5. Test the task with a small input list before increasing the volume or parallelism.

Do not add artificial splitting to a workflow that depends on strict ordering, a shared session, or a target site that limits concurrent requests.

​

Check how a run was split

After starting the task in the cloud:

1. Open the cloud run window from the Task List.
2. Open Subtask Status to see the subtasks, their status, timing, and data extracted.
3. Use the event log to check how many subtasks were created and whether they were assigned gradually.
4. Compare subtask output and elapsed time to find slow or stopped portions.

The number of subtasks is not the same as the number of active nodes. Use Cloud resources and queues to understand node capacity and queueing, and Cloud run status to interpret the run-level result.

​

When splitting does not improve the result

Review the task instead of simply adding more nodes when:

• Most subtasks spend time waiting for the same page or API response.
• The target site starts returning CAPTCHA, rate-limit, or blocked responses.
• Login cookies are not valid in parallel sessions.
• Subtasks stop after producing no data.
• The task has too few independent input items to keep nodes busy.
• The run is queued because other tasks already consume the account’s nodes.

Run a small cloud test, compare the Data List with a known-good result, and check stopped subtask logs and screenshots before scaling the workload.

​

Related pages