Q: Why there are only a few tasks running when I set 10 tasks to run concurrently in the cloud?

 

 

     A:

 

1. The professional edition allows you to have 10 tasks being executed in parallel in the cloud.

    Since there are only 10 cloud servers for all your tasks. So the situations are:

 

If you have 10 tasks (which are not split) to be executed in parallel in the cloud, then only one cloud server will be allocated to one task;

If you have  2 tasks (which are split into sub-tasks) to be executed concurrently in the cloud, then 10 cloud servers will be allocated to your two tasks, unevenly.

 

2. Perform a similar task: in the cloud(Cloud Extraction) VS on local machine(Local Extraction).

 

If you perform a task (which is not split), the local extraction will faster than using cloud extraction.

Because only one cloud server will allocated to the unsplit task and your machine configuraion works better than one cloud server.

 

If you perform a task (which is split into sub-tasks) in the cloud, then your task will split into 10 sub-tasks and 10 cloud servers will allocated to these sub-tasks.

Thus executing tasks in the cloud will speed up the extraction and in this case have better performance than Local Extraction.

 

To speed up the extraction using Cloud Extraction, you can

1.  set the maximum number of tasks being executed in parallel (no more than 2 at first). 

2.  set priority for your task. Octoparse will execute the tasks in order from the beginning by default. (See the screenshot below)

For example, if the max number of tasks being executed in parallel is two, then 10 Octoparse cloud servers will allocated to these two tasks according to the extracted content of each task (e.g. 6 servers for task 1 and 4 servers for task 2).

After the extraction of these two tasks is completed, you can execute other tasks.

 

 

 

If you have any question, we'd happy to help.

Octoparse Support Team

btn_sidebar_use.png
btn_sidebar_form.png