Web Scraping Feature Study | Infinitely Scrolling

Saturday, October 08, 2016 5:44 AM

What it is?

Infinite-scrolling,also know as "endless scrolling" is a technique used most often by websites with JavaScript or AJAX to load additional content dynamically as user scroll down to the bottom of the webpage.

 

When do you want to use it?

For websites with infinitive scrolling, contents on the web page will get loaded continuously as you scroll down the page. Scraping data from sites with such infinite scroll feature can be somewhat challenging from first look but the actual steps to accommodate is actually not tricky at all. 

 

How to use it?

To accommodate infinitive scrolling in the scraping process, what we do is simply instruct Octoparse to keep scrolling down until the page gets fully loaded before moving to the next extraction step. 

To do this, 

  • Navigate to the page needed to be scrolled. 
  • Under "Advanced Options", check for  "Scroll down to the bottom of the page when finished loading"
  • Fill in "Scroll times" , "Time Interval" and the "Scroll way"

Here I choose "Scroll down for one screen", scroll times = 3 and time interval = 1 second. This means that, Octoparse will keeps on scrolling on a single page for 3 times total and will wait 1 second before executing the next scroll down. 

 

Then, we can proceed to the next extraction step, including building a loop list. 

For more details steps, refer to case tutorial: Web Scraping Case Study | Scraping data from Jabong.com

 

Start running your task

After setting up your scraping task, follow the indications to start running your task.

  • After saving your extraction configuration,click “Next”
  • Select “Local Extraction”
  • Click “OK” to run the task on your computer.

Octoparse will automatically extract all the data selected. Check the "Data Extracted" pane  for the extraction progress 

 

 

Check the data and export

The data extracted will be shown in "Data Extracted" pane.

Click "Export" button to export the results to Excel file, databases or other formats and save the file to your computer.

 

Done!

Now you've learn the explicit steps to scrape from websites with infinitive scrolling, check out how this technique is applied to scrape Jabong ( case tutorial: Web Scraping Case Study | Scraping data from Jabong.com)

 

Or learn more about how Octoparse works:

How to Scrape Websites With Infinite Scroll? (Quora, Facebook,Twitter)

Web Scraping Case Study | Scraping Data from Yelp

Web Crawling Case Study | Scraping ASTA with Pagination (2) - No "Next Button" Found

Scrape Scientific America with Pagination Issue

 

For more information about Octoparse, please click here.

Sign up today!

30 Free Web Scraping Software

Collect Data from Amazon

Top 30 Free Web Scraping Software

- See more at: http://www.octoparse.com/tutorial/pagination-scrape-data-from-websites-with-query-strings-2/#sthash.gDCJJmOQ.dpuf
btn_sidebar_use.png
btn_sidebar_form.png