Step-by-step tutorials for you to get started with web scraping

Download Octoparse

What are concurrent runs?

Thursday, August 16, 2018

Tasks that are running concurrently means they are being executed simultaneously either in the cloud or on your local machine.

1. Local concurrent runs basically mean executing more than one task on the local machine. For this specific feature, the Free Plan is limited to two concurrent local runs while all the other plans allow for unlimited concurrent runs.

2. Cloud concurrent runs can be checked from the dashboard by filtering for “Running in the Cloud”. 

Data Scraping with Octoparse - Concurrent Runs

3. Question: When a plan comes with 6 cloud servers, does it mean there should always be 6 concurrent runs in the cloud?

     Answers: Not exactly. It is possible to have 6 concurrent runs when each executing task only takes up one cloud server to run. However, when a task is set to run in the cloud, Octoparse will first check to see if the task can be further split into smaller sub-tasks then assign each of the sub-task to run on a cloud server for speeding up the extraction. In this case, one task can take up more than one if not all of the cloud servers (Learn more about task splitting).

 

Tips!

  • If task spitting is not intended, check "Disable task split" from "Setting" (for Cloud Extraction)
         Scraping with Octoparse - Concurrent Runs

  • You can limit the number of task running concurrently by accessing Account Setting. Having fewer tasks running in parallel will free up more cloud servers to the limited tasks hence faster the extraction (for Cloud Extraction)

    Scraping with Octoparse - Concurrent Runs

  • When multiple tasks are running concurrently, you can set higher or lower priority to each task (for Cloud Extraction).

         Web Scraping with Octoparse - concurrent runs

 

Related Articles:

What's new in Octoparse 7.X? 

Dealing with AJAX 

Select items in a drop-down menu 

Use lists to extract 

Extract multiple pages through pagination 

 

 

Download Octoparse to start web scraping or contact us for any
question about web scraping!

Contact us Download
btn_sidebar_use.png
btn_sidebar_form.png