undefined

Web Scraping Feature Study | Scraping from multi-pages: pagination with "Next" button

Wednesday, September 28, 2016 7:54 AM

For the latest tutorials, visit our new self-service portal. Sharpen your skills and explore new ways to use Octoparse.

 

Pagination, also known as paging, is the process of dividing a document into discrete pages, either electronic pages or printed pages. Setting up pagination is a crucial process to get plenty of data from multiple pages. Common ways of pagination include paging with a "Next" button, a "Load More" button, or with an infinite scroll.

In this tutorial, we will introduce the way to deal with pagination with a "Next" button in Octoparse.

 

There are two ways to set up pagination with a "Next" button:

1) Use the auto-detect algorithm to set it up

2) Set up the pagination manually

 

You may need this example link to follow through:

https://www.amazon.com/s?k=sunglass&ref=nb_sb_noss_2

 

1) Use the auto-detect algorithm to set it up

 

  •  Click the "Auto-detect web page data" on the Tips panel. 
  •  Click the "Check" button to inspect which button is identified as a "Next" button.

The button selected will be highlighted on the page.

check next button

If Octoparse fails to recognize the correct next page button, click "Edit" to select the next button manually. 

 

  • Uncheck the "Add a page scroll" and click "Create workflow" on the Tips panel and you will see a workflow with pagination created.

create workflow

 

 

  • Test the workflow by clicking "Pagination" and "Click to paginate" to see whether Octoparse is able to move to the next page.

If you find Octoparse is able to go to the next page, that means your pagination is successfully set up.

 

2) Set up the pagination manually

If the auto-detection does not find the next page button, you can create the pagination manually.

 

  • Click the next button, then choose "Loop click next page" on the Tips panel

loop click next button

 

Tip!

 If there is no "Loop click next page" option, you can select any option with "Loop click" to generate a pagination loop, like "Loop click single element", "Loop click single button", and "Loop click single link".

 

If you find Octoparse is able to go to the next page, that means your pagination is successfully set up. Test the workflow by clicking "Pagination" and "Click to paginate" to see whether Octoparse is able to move to the next page.

 

If you need any assistance with your data project, feel free to submit a request here to contact our support team anytime!

 

Happy Data Hunting!

Author: The Octoparse Team

Download Octoparse Today

 

For more information about Octoparse, please click here.

Sign up today. 

 

We use cookies to enhance your browsing experience. Read about how we use cookies and how you can control them by clicking cookie settings. If you continue to use this site, you consent to our use of cookies.
Accept decline