Web Scraping Feature Study | Infinitely ScrollingSaturday, October 08, 2016 5:44 AM
What it is？
When do you want to use it?
For websites with infinitive scrolling, contents on the web page will get loaded continuously as you scroll down the page. Scraping data from sites with such infinite scroll feature can be somewhat challenging from first look but the actual steps to accommodate is actually not tricky at all.
How to use it?
To accommodate infinitive scrolling in the scraping process, what we do is simply instruct Octoparse to keep scrolling down until the page gets fully loaded before moving to the next extraction step.
To do this,
- Navigate to the page needed to be scrolled.
- Under "Advanced Options", check for "Scroll down to the bottom of the page when finished loading"
- Fill in "Scroll times" , "Time Interval" and the "Scroll way"
Here I choose "Scroll down for one screen", scroll times = 3 and time interval = 1 second. This means that, Octoparse will keeps on scrolling on a single page for 3 times total and will wait 1 second before executing the next scroll down.
Then, we can proceed to the next extraction step, including building a loop list.
For more details steps, refer to case tutorial: Web Scraping Case Study | Scraping data from Jabong.com
Start running your task
After setting up your scraping task, follow the indications to start running your task.
- After saving your extraction configuration，click “Next”
- Select “Local Extraction”
- Click “OK” to run the task on your computer.
Octoparse will automatically extract all the data selected. Check the "Data Extracted" pane for the extraction progress
Check the data and export
The data extracted will be shown in "Data Extracted" pane.
Click "Export" button to export the results to Excel file, databases or other formats and save the file to your computer.
Now you've learn the explicit steps to scrape from websites with infinitive scrolling, check out how this technique is applied to scrape Jabong ( case tutorial: Web Scraping Case Study | Scraping data from Jabong.com)
Or learn more about how Octoparse works:
For more information about Octoparse, please click here.
Sign up today!