Key Points of The Latest Web Data Collection Technology

Cloud Service

Cloud service is the key point to measure the technical capacity of collection technology. One of the main benefits of using any cloud service is time saving. Everything is done in the cloud. All site page views, data standardization and file downloading are done by the cloud servers. You can take your time to focus on the things that matter, rather than on the boring data acquisition.

Companies like Octoparse, Import.io, Mozenda, CloudScrape and ParseHub, all offer cloud service.

The web-based web collection technology is mainly divided into two types. One is based on the software system, and one is a browser-based plug-in. Octoparse and Mozenda are based on Windows, while Import.io is based on a browser-based plug-in.

Advanced functionality and more interactive operation can be achieved with the based-system software. While browser-based software can be more lightweight and compatible with any operating system. API functions are available in Octoparse and Import.io, you can use either one if you need a large-scale cloud servers to collect data and export data by API.

Cost

Octoparse has free trial and you would only be charged for cloud service. You can just use the free edition of Octoparse if your PC could meet your needs. But if you need to collect massive amount of data for 24*7 hours per day, you may need Octoparse’s cloud servers to collect the data. And in this case you would be charged for cloud service.

Mozenda is a shareware software. It offers a time-limited trial and charged only when the users are satisfied. The mode of Import.io is basically the same with that of Octoparse.

Complex Page Structure

If the data (product list) shares the same structure in a number of different pages, pagination is the fastest way to access all the data from the subsequent pages . You just configure the rule for the first page and the rest will be collected in accordance with the same rule. 

Regular Expression

Writing your own regular expression enables to extract data from the string section inside. Of course, this is assuming you understand how to use Regular Expression. Octoparse has built-in Regular Expression tool. We make some specific examples into simple features for you to quickly write regular expressions.

The function of Regular Expression is available in Import.io, Mozenda, ParseHub, and etc.

(If you couldn’t write a regular expression, W3Schools Online Web Tutorials will guide you to learn everything you want to know.)

Infinite Scrolling

It’s tricky to collect data from pages loaded with AJAX, for example infinite scrolling. But in this respect we have achieved a breakthrough. All you need is to configure in Octoparse and set the pagination number, so you can open a web page completely and extract the data inside. Cloudscrape, Apifier, ParseHub, Data-Miner.io and import.io can handle the technology as well.

URLs

Many sites follow a particular URL pattern. In these cases, you can generate a specific list of URL to extract data. Octoparse, Import.io, Diffbot, CloudScrape, ParseHub, FiveFilters and Data-Miner.io allow you to paste URLs. However, only Apifier allows you to both generate a URL and run them through your API. You can also use Google tables or Excel to generate the list.

API

API function is available in many cloud services so that you can schedule to run the API every minute, hourly, daily, weekly, monthly. Octoparse, Diffbot, Mozenda, CloudScrape and Scrapinghub provide the same function, but with a different limit. You can collect data by using our cloud servers and real-time APIs.

Download Options

If you want to download the data, you can choose to save it in several formats such as CSV, HTML, etc. We have listed the file formats that supported by the free editions of the companies.

• CSV
  Octoparse, Import.io, Mozenda, CloudScrape,ParseHub, Data-Miner
• JSON
  Import.io, Diffbot, CloudScrape, Semantics3, ParseHub, TrooclickAPI
• HTML
  Octoparse, Import.io, Apifier

Octoparse has more than 180,000 users in China market and achieves a breakthrough in these key functions. We are committed to providing a simple, easy-to-use software with super powerful functions. Without knowing programming knowledge, you can also bulk collect data from the web. 

We are glad to help and make our product even better for you. if you find any missing feature, please feel free to contact us.support@octoparse.com