How to Scrape Data from A Website with A “Load More” Button (Example: Kickstarter)

Wednesday, May 04, 2016 7:01 AM

Welcome to Octoparse tutorial. In this tutorial, I’m going to show you how to scrape a website with a “Load More” button.

Some websites have a “Load More” button or “Display more data” button to load more content. This kind of websites also use a lot of Ajax. But you can easily scrape any information from these websites with Octoparse. I will take Kickstarter for example. In this video I’ll use a little X path and Ajax. So please watch the tutorials of X path and Ajax first, if you don’t know what it is.

 

First, open the page in the built-in browser. Go to the webpage.

 

Wait until the page loaded, drop a “Loop item” action into the workflow designer.

 

Then choose an "Loop Mode" under "Advanced Options". Choose "Single element".

 

Locate the “Load More” link by using firebug extension in fire fox browser.

Copy the X path and paste it in the textbox. Click "Save".

 

Then drop an “Click item” action into the "Loop item".

Select "Click loop items". Choose "Load page with Ajax" and set ajax time out. Click "Save".

Now it will automatically click on the “Load More” button.

 

Next, create a list of item. Click the first project. Select "Create a list item". Add current item to the list. Continue to edit the list.

Click the second project. Add current item to the list. Finish creating list. Click "Loop".

 

Then drag out the inside “Loop Item”.

 

Then extract data. Extract the title. Select "Extract text".

Extract name. Now we’re done configuring extraction. Click “Next”.

 

Then run the local extraction and the data would be extracted.

 

The process of the first "Loop item" won’t be very fast because there are hundreds of thousands of item to load. After finishing the first "loop item", Octoparse will automatically extract all the data loaded.

 

Now you know how to scrape a website with a “Load More” button.

Download Octoparse now and try it for yourself.

 

 

 

Author: The Octoparse Team

 

 

 

Download Octoparse Today

 

 

For more information about Octoparse, please click here.

Sign up today.

  

 

Author's Picks

 

Octoparse Smart Mode -- Get Data in Seconds

Get Started with Octoparse in 2 Minutes

Smart Mode No Coding No Training

Scrape Job Postings from Glassdoor 

Scrape Job Postings from Indeed.com

Scrape Job Postings from Monster.com

Scrape Content Details from Freelancer.com

Get Updated Data with Clicks

 

 

 

 

 

Contact
us

Leave us a message

Your name*

Your email*

Subject*

Description*

Attachment(s)

Attach file
Attach file
Please enter details of your issue and we will get back to you ASAP.