Step-by-step tutorials for you to get started with web scrapingDownload Octoparse
Why does Octoparse stop after clicking “Next” ?Thursday, August 16, 2018
While scraping data from the web, Octoparse takes the reloading as the signal to execute the action, such as "Click item" and "Click to paginate". For the web page using AJAX, it updates new contents without reloading. As there is no reloading, Octoparse doesn't receive the signal to act and would be stuck in the last step. As a result, we may get zero, or much fewer extracted data than we expect.
To fix this issue, you will need to set up “AJAX Load” timeout under “Advanced Options” of the “Click Item” action.
AJAX timeout setting in Octoparse is based on the time it takes for web pages to load. You can also set up a higher value as long as there is enough time for the next page to load.
In fact, many websites update new data with AJAX configuration such as Amazon, eBay, Google etc. Sometimes, if Octoparse takes so long to scrape data after clicking an item, you should also configure AJAX timeout to make sure there’s enough time for detail pages to load.
If you do have questions about AJAX, read more details here
- Most popular tutorials
- Scrape product information from Amazon
- How to download images from a list of URLs?
- Extract multiple pages through pagination
- Scraping info from Craigslist
- Scraping search results from Google Scholar