Step-by-step tutorials for you to get started with web scraping

Download Octoparse

How to export extracted data to a database?

Thursday, August 16, 2018

Besides data export to commonly used formats such as CSV, XLS, HTML etc, Octoparse also supports data export to databases, including SqlServer, MySql, and Oracle.


Tips!

Octoparse currently does not support data export to cloud databases such as AWS database, Azure, Cloud SQL database, etc.

 





Export extracted data to a database manually

1) Access data extracted in the cloud or locally

  • Click "More Actions"
  • Click "View Data"
  • Select "Cloud Data" or "Local Data"

Scraping with Octoparse - database export

2) Once the extracted data gets loaded,
  • Click the "Export" button from the lower right corner
  • Select "Export to database"
  • Confirm and click "Export data" 

Scraping with Octoparse - database export

3) Click "Next" to proceed

web scraping with octoparse - automatic data export to database

4) On the configuration interface,
  • Select a previously saved configuration setting if there's any
  • From the drop-down menu, select the type of database
  • Fill in the database information, such as server, username, password etc.

web scraping with octoparse - automatic data export to database

5) Test the connection

  • Next to "Database Name", select the database to export the data
  • Click "Test Connection"
  • When the connection is tested available, click "Next" 

 

Scraping with Octoparse - Database Export
6) Proceed with the data export setup
  • Select an existing table from the "Data Table" drop-down options.
  • Match the source data fields (data fields extracted) to the target data fields from the designated table.
  • Select any source data fields to use as primary keys. 
  • Click "Next" 

web scraping with octoparse - export data to database 

 

Tips!

  • Once a data field is selected as a primary key, subsequent data records with the same primary key will overwrite the existing ones in the database.
  • Once the configuration is completed, it can be saved for later use. 

 

7) Click "Next" and wait for the data export to finish. 





Automatic data export to a database  

You can also set a schedule for data export to any database ahead of time (only for data extracted in the cloud). Data will be exported to the designated database according to the pre-set schedule. 

1) Access data extracted in the Cloud

2) Select “Export to database (Automatic)"

web scraping with Octoparse - automatic data export to database

3) Click "Export data"

web scraping with octoparse - automatic export to database

4) Click "Next" to proceed

web scraping with Octoparse - automatic data export to database

5) On the configuration interface,
  • Select a previously saved configuration setting if there's any
  • From the drop-down menu, select the type of database
  • Fill in the database information, such as server, username, password etc.

web scraping with octoparse - automatic data export to database

6) Test the Connection

  • Next to "Database Name", select a database to export the data
  • Click "Test Connection"
  • If the connection is tested available, click "Next" 

 

 

7) Proceed with the data export setup
  • Select an existing table from the "Data Table" drop-down options
  • Match the source data fields (data fields extracted) to the target data fields from the designated table.
  • Select any source data fields to use as primary keys
  • Click "Next" 

web scraping with octoparse - export to database

Tips!

  • Once a data field is selected as a primary key, subsequent data records with the same primary key will overwrite the existing ones in the database.
  • Once the configuration is completed, it can be saved for later use.  

 

8) Set a schedule

  • Enter the name of the schedule into the text box next to "Scheduled Export"
  • Select at what time interval you would like to export the data to the database, ie. the export is scheduled to run every 4 hours in the screenshot. 
  • Click "Next" 

web scraping with Octoparse - auto export to database

9)Click "Finish" to complete the Auto-export Setup

web scraping with octoparse

10)From the pop-up window, you can start/stop/delete the scheduled auto export to the database.

web scraping with octoparse - auto export

 

Tips!

The Scheduled Export to Database windows is also accessible from "Tools" from the side navigation. 

web scraping with Octoparse - auto export to database 

 

Related Articles:

What's new in Octoparse 7.X? 

Dealing with AJAX 

Select items in a drop-down menu 

Use lists to extract 

Extract multiple pages through pagination 

  

 

Download Octoparse to start web scraping or contact us for any
question about web scraping!

Contact us Download
btn_sidebar_use.png
btn_sidebar_form.png