Set up IP Rotation in Octoparse
Wednesday, July 20, 2016 11:43 PMThere are lots of websites that has advanced anti-scraping techniques. Usually, some websites ban or block your computer's IP address from your further access to their websites when you extract the sites' data too frequently.
Octoparse's cloud service provides many anonymous HTTP proxy servers and you don't need to manually connect to the proxy's IP address. But if you use Octoparse’ s Local Extraction, you need to add external proxy addresses manually for automatic rotation.
In this article, I’ll show you how to set IP addresses for Local Extraction.
Step 1. Get some public proxy addresses.
Step 2. After you finished configuring the extraction rule, choose either Local Extraction to run the task on your own computer or use our cloud service - Cloud Extraction. Here we choose Local Extraction.
Step 3. Data extracted would be shown in the pop-up window. Here, we will click the red button at the bottom of the window to stop this task.
Step 4. In the Extraction Options, we choose the “Use Web-Proxy (HTTP)” option and then click the “Set Proxy” Link.
Step 5. In the “Proxy Settings” window, you can set the switching interval. Here we enter 30 seconds, and Octoparse will switch IP address every 30 seconds from the list of IP addresses. In the IP Proxies box, enter a list of IP addresses and separate them by putting one IP address per line. Then, hit the “OK” button.
Step 6. Click the “Start” button at the bottom of the window to continue the process of data extraction.
Happy Data Hunting!
Author: The Octoparse Team
For more information about Octoparse, please click here.
Sign up today.
Author's Picks
Octoparse 6.0 is Now Available
What A Price Monitor Can Help you?
Examples of Businesses Who Use Data Scraping
