undefined

Step-by-step tutorials for you to get started with web scraping

Download Octoparse

How to export extracted data to a database?

Wednesday, November 24, 2021

The latest version for this tutorial is available here. Go to have a check now!

 

Besides data export to commonly used formats such as CSV, XLS, HTML etc, Octoparse also supports data export to databases, including SqlServer, MySql, and Oracle.


Tips!

Octoparse currently does not support data export to cloud databases such as AWS database, Azure, Cloud SQL database, etc.

 


Export extracted data to a database manually

1) Access data extracted in the cloud or locally

  • Click "More Actions"
  • Click "View Data"
  • Select "Cloud Data" or "Local Data"
2) Once the extracted data gets loaded,
  • Click the "Export" button from the lower right corner
  • Select "Export to database"
  • Confirm and click "Export data" 

3) Click "Next" to proceed

 

4) On the configuration interface,
  • Select a previously saved configuration setting if there's any
  • From the drop-down menu, select the type of database
  • Fill in the database information, such as server, username, password etc.

 

5) Test the connection

  • Next to "Database Name", select the database to export the data
  • Click "Test Connection"
  • When the connection is tested available, click "Next" 
6) Proceed with the data export setup
  • Select an existing table from the "Data Table" drop-down options.
  • Match the source data fields (data fields extracted) to the target data fields from the designated table.
  • Select any source data fields to use as primary keys. 
  • Click "Next" 

 

 

Tips!

  • Once a data field is selected as a primary key, subsequent data records with the same primary key will overwrite the existing ones in the database.
  • Once the configuration is completed, it can be saved for later use. 

 

7) Click "Next" and wait for the data export to finish. 


Automatic data export to a database  

You can also set a schedule for data export to any database ahead of time (only for data extracted in the cloud). Data will be exported to the designated database according to the pre-set schedule. 

1) Access data extracted in the Cloud

2) Select “Export to database (Automatic)"

 

3) Click "Export data"

 

4) Click "Next" to proceed

 

5) On the configuration interface,
  • Select a previously saved configuration setting if there's any
  • From the drop-down menu, select the type of database
  • Fill in the database information, such as server, username, password etc.

 

6) Test the Connection

  • Next to "Database Name", select a database to export the data
  • Click "Test Connection"
  • If the connection is tested available, click "Next" 

 

7) Proceed with the data export setup
  • Select an existing table from the "Data Table" drop-down options
  • Match the source data fields (data fields extracted) to the target data fields from the designated table.
  • Select any source data fields to use as primary keys
  • Click "Next" 

 

Tips!

  • Once a data field is selected as a primary key, subsequent data records with the same primary key will overwrite the existing ones in the database.
  • Once the configuration is completed, it can be saved for later use.  

 

8) Set a schedule

  • Enter the name of the schedule into the text box next to "Scheduled Export"
  • Select at what time interval you would like to export the data to the database, ie. the export is scheduled to run every 4 hours in the screenshot. 
  • Click "Next" 

 

9)Click "Finish" to complete the Auto-export Setup

 

10)From the pop-up window, you can start/stop/delete the scheduled auto export to the database.

 

Tips!

The Scheduled Export to Database windows is also accessible from "Tools" from the side navigation. 

web scraping with Octoparse - auto export to database 

Related Articles:

What's new in Octoparse 7.X? 

Dealing with AJAX 

Select items in a drop-down menu 

Use lists to extract 

Extract multiple pages through pagination 

  

 

Download Octoparse to start web scraping or contact us for any
question about web scraping!

Contact Us Download
We use cookies to enhance your browsing experience. Read about how we use cookies and how you can control them by clicking cookie settings. If you continue to use this site, you consent to our use of cookies.
Accept decline