Web Scraping Feature Study | Scraping from multi-pages: pagination without "Next" button

Wednesday, September 28, 2016 8:32 AM

 

What it is?

Pagination action is usually used when the content we want to scrape spans over different pages of a website. Octoparse mimics human browsing behaviors, so just as you would click to the next page as you browse through a website, Octoparse does the same when you use pagination feature.

Sometimes, we may encounter such situation: there is no "Next" button for us to loop click for pagination. In this case, we should modify XPath to locate the next page on our own.

 

When do you want to use it?

If you would be extracting data from more than one page, then use pagination to enable page flipping.

Specifically, we should find locate next pages by ourselves if we encounter such situation where there is no "next" button for us to turn pages.

              

 

Download my extraction task of this tutorial of scraping data with pagiantion HERE just in case you need it.

 

How to use it?

 

Step 1. To define a loop click action for turning pages , we need drag a "Loop" item into Workflow designer and select "Single Element" in the "Loop Mode" first.

 

Step 2. To make sure you locate the first page so that you could get the data from all the pages, then modify XPath to locate the next page.

 

Step 3. To loop click all the items in the cycle pages, we need to select "Click Items in the loop" to scrape data within each page.

 

 

Now you've learned how to flip through pages to scrape data without "Next" button. Let’s look into how pagination works with this example.

Or, learn more about pagination related topics:

 

 

Author: The Octoparse Team

 

 

 

Download Octoparse Today

 

 

For more information about Octoparse, please click here.

Sign up today!

 

 

Author's Picks

 

Octoparse Smart Mode -- Get Data in Seconds

Get Started with Octoparse in 2 Minutes

Pagination Scraping: Configure “Loop click next page” When It Can’t Be Detected

Scrape Data from Website with Pagination - Infinite Scrolling

Collect Data from eBay

Top 30 Free Web Scraping Software

30 Free Web Scraping Software

Collect Data from Amazon

Top 30 Free Web Scraping Software

- See more at: http://www.octoparse.com/tutorial/pagination-scrape-data-from-websites-with-query-strings-2/#sthash.gDCJJmOQ.dpuf
Request Pro Trial Contact
us

Leave us a message

Your name*

Your email*

Subject*

Description*

Attachment(s)

Attach file
Attach file
Please enter details of your issue and we will get back to you ASAP.