Deal with Infinite Scrolling on a Dynamic Website

Wednesday, September 27, 2017 8:44 AM

Websites can be static and dynamic. A static site is one of which the content does not change, for example a yellow page of a company. It could only be updated with knowledge of website development.  A dynamic website contains information that changes very frequently, usually generated by users. This sort of websites may have infinite scrolling techniques such as clicking to load more or scrolling down, like Facebook or Twitter.

 

In this short tutorial, I'm going to show you how to deal with infinite scrolling or clicking to load more on a dynamic website.

Let me give you a for-instance - Twitter, which load infinite content if you keep scrolling down to the bottom of the screen. This case can be easily handled by setting "Scroll Down" of "Go To Web Page" action with Advanced Mode.

 

You can set the scroll times, time interval and scroll way (scroll to the bottom or scroll one screen) according to the website you extract. (See the example tutorial here)

 

 

As for websites such as news webs, the content changes daily. What if you want the everyday top news of a week? It is definitely not a good idea to run the task every day by yourself. Octoparse allows you to schedule an extraction task to run at any time, hourly, daily, weekly etc. on cloud platform. And what you need to do is just export all the data after the extraction is done. (See the example tutorial here)

 

 

The scheduled extraction based on cloud platform is only for premium users. Try our Pro Plan to experience this function!

 

Octoparse have many useful functions you may not know! Check the following tutorials to find more:

Cloud Extraction: Speed up your extraction

Automatic IP rotation: Prevent IP from being blocked

API: Connect data with your own system

 

 

Author: The Octoparse Team

Download Octoparse Today

For more information about Octoparse, please click here.

Sign up today!

 

btn_sidebar_use.png
btn_sidebar_form.png