Web Scraping Feature Study | Scraping from multi-pages: pagination without "Next" button

Wednesday, September 28, 2016 8:32 AM


What it is?

Pagination action is usually used when the content we want to scrape spans over different pages of a website. Octoparse mimics human browsing behaviors, so just as you would click to the next page as you browse through a website, Octoparse does the same when you use pagination feature.

Sometimes, we may encounter such situation: there is no "Next" button for us to loop click for pagination. In this case, we should modify XPath to locate the next page on our own.


When do you want to use it?

If you would be extracting data from more than one page, then use pagination to enable page flipping.

Specifically, we should find locate next pages by ourselves if we encounter such situation where there is no "next" button for us to turn pages.



Download my extraction task of this tutorial of scraping data with pagiantion HERE just in case you need it.


How to use it?


Step 1. To define a loop click action for turning pages , we need drag a "Loop" item into Workflow designer and select "Single Element" in the "Loop Mode" first.


Step 2. To make sure you locate the first page so that you could get the data from all the pages, then modify XPath to locate the next page.

(Note: For more details about how to modify XPath, you can check out the tutorials:

  Modify XPath Manually in Octoparse

  How to Use XPath in Octoparse 

  and a list of XPath relevant study tutorials by visiting Octoparse official website: www.octoparse.com)



Step 3. To loop click all the items in the cycle pages, we need to select "Click Items in the loop" to scrape data within each page.




Now you've learned how to flip through pages to scrape data without "Next" buttonLet’s look into how pagination works with this example.

Or, learn more about pagination related topics:



Author: The Octoparse Team




Download Octoparse Today



For more information about Octoparse, please click here.

Sign up today!



Author's Picks


Octoparse Smart Mode -- Get Data in Seconds

Get Started with Octoparse in 2 Minutes

Pagination Scraping: Configure “Loop click next page” When It Can’t Be Detected

Scrape Data from Website with Pagination - Infinite Scrolling

Collect Data from eBay

Top 30 Free Web Scraping Software

30 Free Web Scraping Software

Collect Data from Amazon

Top 30 Free Web Scraping Software

- See more at: http://www.octoparse.com/tutorial/pagination-scrape-data-from-websites-with-query-strings-2/#sthash.gDCJJmOQ.dpuf
We use cookies to enhance your browsing experience. Read about how we use cookies and how you can control them by clicking cookie settings. If you continue to use this site, you consent to our use of cookies.
Accept decline