Step-by-step tutorials for you to get started with web scrapingDownload Octoparse
How to deal with data missing on cloud extraction?Thursday, August 16, 2018
Data missing on Cloud Extraction could occur when:
1. Tasks executed with cloud extraction are split-table and working too fast hence some elements may skip.
Tasks with "Fixed List", "List of URLs" and "Text List" loop mode are split-table. The main tasks will be split into sub-tasks executed with multiple cloud servers simultaneously. So in this case, every step of the task will work very fast hence some pages may not be loaded completely before moving to the next step.
2. The website you are after is actually multi-regional.
A multi-regional website could have different page structures for the content provided to visitors from different countries. When a task is set to run in the cloud, it is executed with our IP's based in America. In this case, for tasks targeted websites outside America, some data may be skipped as it can’t be found on the website opened in the cloud.
3. When the task has both 1 and 2 situations.
Here are common solutions to deal with data missing on cloud extraction.
- To ensure the web page to be loaded completely in cloud, you could try to
1. Increase timeout for "Go To Web Page“ step
Advanced Options > Timeout
2. set up Wait before execution
All steps created in the workflow are able to set up a waiting time, except Go To Web Page.
Advanced Options > Wait before execution
- To identify if the website is multi-regional, you could
- Test the task with local extraction. If there's no data missing like it does on the cloud extraction, then the website is most likely a multi-regional. In this case, as the targeted content can only be found when opening the website with your own IP, we suggest you Local Extraction to get the data instead.
- extract the outer HTML of the whole page. By checking the extracted HTML, you could find what has caused the data missing by the prompt in the source code like "Access denied".
- Most popular tutorials
- Scrape product information from Amazon
- How to download images from a list of URLs?
- Extract multiple pages through pagination
- Scraping info from Craigslist
- Scraping search results from Google Scholar