undefined
Blog > Post

Access to The Data Extracted in the Cloud via API

Thursday, April 21, 2016

Cloud Extraction is one of Octoparse's outstanding features.

Octoparse provides you with multiple cloud servers to extract web data simultaneously. You can access to the extracted data in the cloud via API.

(Note: Here is a video about Cloud Extraction. It is not available in the Free Edition.)

You can access to the extracted data and download them via the URL,

“http://dataapi.octoparse.com/SkieerDataAPI/GetData?key=89051c84-1f82-428a-9db6-6f67a87fec35&from={BeginTime}&to={EndTime}”.

To get this URL, click the “Create an API” option, as shown below.

 

Then, you’ll see the URL will be displayed in the pop-up screen.

 

Click your rule file, and choose “Start Cloud Extract” option.

Immediately the cloud management system will read the rule and send out a command to assign cloud servers to extract data in accordance with the rule.

During the process, you can see the data extraction progress clearly.

If you are using Standard Edition, you can get four cloud servers to help you extract data at the same time.

If you are using Professional Edition, you can get 10 cloud servers.

(If you want more cloud servers, please feel free to contact our support team: support@octoparse.com

 

In the process of data extraction, you can choose “View Data” option to check out the data extracted.

And of course, you can also access them via API as described above.

 

An API URL includes three parameters, KEY, BeginTime, and EndTime.

API Key is the unique identifier of each task for verifying ID and authority. (Don’t leak or modify the key. )

The begin time and EndTime are set by yourself. But both time are using the universal time, UTC-05:00.

The format of API is XML.

Before you use the API URL, you need to modify the content in the two brackets (BeginTime and EndTime) manually.

BeginTime: the starting datetime of Extraction. Format: yyyy-mm-dd hh:mm:ss

EndTime: the ending datatime of Extraction. Format: mm-dd-yyyy hh:mm:ss

 

Author: The Octoparse Team 

Octoparse Download

 

More Resources

 

Top 20 Web Scraping Tools to Scrape the Websites Quickly

Top 30 Big Data Tools for Data Analysis

Web Scraping Templates Take Away

How to Build a Web Crawler - A Guide for Beginners

Video: Create Your First Scraper with Octoparse 7.X

 

Download Octoparse to start web scraping or contact us for any
question about web scraping!

Contact Us Download