Cloud resources and queues

Cloud Extraction runs tasks on a pool of managed cloud resources. Your plan sets the available capacity. Your task structure determines how that capacity is used. The key rule is: each running subtask uses one cloud node. A task can use one node or several. More nodes can shorten one task’s runtime, but leave fewer nodes for other tasks. When no eligible capacity remains, new work waits in the queue.

​

Resource model

​

How tasks consume nodes

Every cloud task needs at least one node. An unsplit task runs as one execution unit and uses one node while active. For a splittable loop, Octoparse divides the loop items into subtasks. Each active subtask uses one node, so several portions can run in parallel. URL lists, text lists, and fixed lists can use the available parallel capacity. A Variable List is split into three subtasks. A single-element loop normally runs as one execution unit. With six available nodes, the allocation might look like this: The actual allocation depends on task structure, account settings, plan entitlements, and current availability. Check the subtask view to see how many nodes a run is using. Cloud nodes are shared and assigned dynamically. A node can be returned to the shared pool after its subtask finishes, while a node marked for release may stop accepting new subtasks and be removed after its current work ends. The node count shown for a plan is therefore an upper capacity limit, not a set of permanently reserved machines.

​

Why a cloud run is queued

A task can enter or remain in the queue when:

• Other tasks or subtasks use all available nodes.
• The account has reached its maximum number of tasks running in parallel.
• A splittable task uses several or all account nodes.
• Temporary cloud capacity is below the plan’s upper limit.

Queued work starts automatically when the required capacity is released. A queued status does not necessarily indicate an error.

​

How priority affects the queue

Priority determines which queued work is considered first when capacity becomes available. Values range from 1 (lowest) to 99 (highest). Use higher values for time-sensitive production runs. Use lower values for backfills, experiments, and other work that can wait. Priority changes queue order; it does not add nodes or speed up the workflow. Keep a clear difference between urgent and deferrable work. Setting every task to the highest value removes that distinction.

​

Single-task speed vs account throughput

Cloud capacity can optimize for either one task’s completion time or the number of tasks completed together. More nodes do not make every task faster. Parallel acceleration requires independent work that Octoparse can split. Sequential page dependencies, login state, site rate limits, response time, and anti-bot controls can limit the gain. More parallel traffic can also make a target site less stable. Use subtask details, elapsed time, output volume, and errors to evaluate the result. Choose the allocation that meets the deadline without starving higher-priority work or overloading the target site.

​

Plan limits

Cloud Extraction is the execution foundation for scaling data scraping in Octoparse. More cloud nodes give your account more parallel capacity, so you can run more tasks at the same time and finish scraping sooner. This makes cloud nodes one of the most important resources for large-scale extraction. Standard provides up to 6 nodes for small-team and recurring workloads. Professional provides up to 20 nodes for professional teams that run more tasks or need faster parallel extraction. If you manage hundreds of tasks or have high-volume data scraping needs, Enterprise lets you work with Octoparse to customize node capacity for your workload. If your runs regularly queue or large tasks need more parallelism, upgrading to a plan with more nodes provides the capacity to scale. These are upper limits, not guaranteed task counts. Actual capacity depends on how many nodes each task consumes, your parallel-task setting, and current availability. For Enterprise, use the capacity assigned to your account.

​

What to check when tasks are queued

1. Open the cloud run view and check the number of running subtasks.
2. Identify whether one splittable task is using several nodes.
3. Check the account’s maximum parallel-task setting.
4. Review priorities among queued tasks.
5. Pause or reschedule lower-priority work if one task must finish sooner.
6. If queues remain long, review task splitting and plan capacity.

For task-splitting behavior, see Standard vs boost mode. For product instructions, see Why does the Cloud not start all the tasks I run? and How can I scrape data faster in Cloud? in the Help Center.