Pagination Scraping: Configure “Loop click next page” When It Can’t Be DetectedMonday, May 30, 2016 8:16 AM
In this tutorial, I’ll take realtor.com for example to show you how to configure pagination scraping rule when the “Next” button can't be detected. The URL we use for this example is
Usually you find “Loop click next page” when you configure pagination by clicking on “Next page”.
But sometimes you can’t find the “Loop click next page” option as usual.
List features covered
Some features that we will touch upon include:
- Modify XPath
Now, let’s get started!
As we open the target webpage, we can observe that there is no "Next" button for us but a right arrow to go to the next page. If we click the arrow, we will find no "Loop click the item" choice. In this case, we need to configure pagination scraping rule in another way.
Step 1. Make a loop item
To loop click next page, we still use loop action for pagination functionality.
- To do this, drop an “Loop” item into Workflow designer.
- Choose an "Loop Mode" under "Advanced Options".
- Select "Single Element" option.
Step 2. Modify the XPath for pagination
- Enter the XPath of the right arrow button: //*[@class='ra ra-chevron-right'] (See how to get Xpath with Firefox here)
- Click “Save”.
Step 3. Click to Paginate
- Drag a “Click element” action into the “Loop item”
- Choose “Click Loop items” under "Advanced Option".
- Click "Save"
Then, Octoparse will click the pagination link in sequential loop.
Good job for completing this tutorial!
Now, learn more about pagination and list building (including tips and troubleshooting):
Now you’ve configured pagination crawling. Feel free to contact us at firstname.lastname@example.org
If you like this video, please thumb up and subscribe our channel.
If this video tutorial is not available for you, you can click hereto see the corresponding graphic tutorial.