How to Scrape Data from A Website with A “Load More” Button (Example: Kickstarter)Wednesday, May 04, 2016 7:01 AM
Welcome to Octoparse tutorial. In this tutorial, I’m going to show you how to scrape a website with a “Load More” button.
Some websites have a “Load More” button or “Display more data” button to load more content. This kind of websites also use a lot of Ajax. But you can easily scrape any information from these websites with Octoparse. I will take Kickstarter for example. In this video I’ll use a little X path and Ajax. So please watch the tutorials of X path and Ajax first, if you don’t know what it is.
First, open the page in the built-in browser. Go to the webpage.
Wait until the page loaded, drop a “Loop item” action into the workflow designer.
Then choose an "Loop Mode" under "Advanced Options". Choose "Single element".
Locate the “Load More” link by using firebug extension in fire fox browser.
Copy the X path and paste it in the textbox. Click "Save".
Then drop an “Click item” action into the "Loop item".
Select "Click loop items". Choose "Load page with Ajax" and set ajax time out. Click "Save".
Now it will automatically click on the “Load More” button.
Next, create a list of item. Click the first project. Select "Create a list item". Add current item to the list. Continue to edit the list.
Click the second project. Add current item to the list. Finish creating list. Click "Loop".
Then drag out the inside “Loop Item”.
Then extract data. Extract the title. Select "Extract text".
Extract name. Now we’re done configuring extraction. Click “Next”.
Then run the local extraction and the data would be extracted.
The process of the first "Loop item" won’t be very fast because there are hundreds of thousands of item to load. After finishing the first "loop item", Octoparse will automatically extract all the data loaded.
Now you know how to scrape a website with a “Load More” button.
Download Octoparse now and try it for yourself.
Author: The Octoparse Team
For more information about Octoparse, please click here.
Sign up today.
If this video tutorial is not available for you, you can click here to see the corresponding graphic tutorial.