Use Regular Expression to Reformat Captured Data
Friday, August 18, 2017 3:54 AMFor the latest tutorials, visit our new self-service portal. Sharpen your skills and explore new ways to use Octoparse.
During your web scraping project, you may want to clean the data fields while doing the web scraping. Octoparse offers 8 data cleaning options for turning the extracted data into the format you need.
When should I refine the extracted data?
If you have a desired data format for a certain field, you can use our "Clean Data" function to refine the field within Octoparse. Octoparse would scrape and refine it directly during the scraping process. No need to re-format the field after exporting the data into an excel file.
How to refine the extracted data in Octoparse?
To access these features in Octoparse, you should follow the 4 steps below:
1. Select the data field to refine
2. Click on the "..." icon and select "Clean data".
3. Click "Add step"
4. Select an operation to re-format your data
|
Tips! In programming, a "string" basically refers to a collection of characters like letters, numerals, symbols, and punctuation marks. For example, " " (space) is a string; "Octoparse" is a string; and "Hello 2 *% World!" is also a string. A string can consist of no character as well. In other words, a string that contains no character is empty. If you replace a word with an empty string, colloquially, it is equal to say that you delete the word. You would see the word "string" in a lot of function instructions of Octoparse's data reformat options. If you see the word "string" there, that means you can use the corresponding options to deal with a variety of character types in the data extracted, such as letters, words, sentences, numbers, spaces, symbols, and punctuation marks. |
9 Data reformat options
2. Replace with regular expression
3. Match with regular expression
7. Reformat extracted date/time
1. Replace
Function: Replace the specific string/s in the extracted data with the new string/s that you want.
2. Replace with regular expression
Function: Use a specific regular expression to replace the matched string/s in the extracted data with the string/s that you want.
You can learn more about regular expression in W3schools 
.
3. Match with regular expression
Function: Use a specific regular expression to pick up the matched string/s from the extracted data.
You can learn more about regular expression in W3schools .
4. Trim spaces
Function: Remove the unwanted space/s from the start and/or the end of the data extracted.
If you want to delete the spaces amid the data, you can use Replace or Replace with regular expression.
5. Add a prefix
Function: Add a string or strings to the front of the data extracted.
6. Add suffix
Function: Add a string to the end of the data extracted.
7. Reformat extracted date/time
Function: Shift the extracted date/time into one of the built-in formats, or into your own customized format.
8. Timestamp conversion
Function: Shift the Unix timestamp into your own customized format.
The Unix timestamp is a sequence of numbers that represents a specific date and time. This function will convert Unix time to a format that we can understand easily.
9. HTML transcoding
|
Tips! All the steps added can be edited and deleted here by clicking the |
icons.
Octoparse Regex Tool
Octoparse also offers a RegEx Tool to auto-generate the regular expression that you need. Let's have a quick look at how to use Octoparse's RegEx Tool to generate and apply a regular expression. For example, here we want to pick up the numeral of star-rating from the outer HTML extracted.
· Click "Try RegEx Tool"
· Enter the match criteria: start with "src="", end with " " "
· Click "generate" to produce regular expression
· Click "Match" to pick up the matched strings
· Click "Apply"
· Click "Confirm" to save the settings
Click the link here for more information about the use of the Regex tool.
If you have questions, you are welcome to submit a request here. Our support team will get back to you later.
Happy Data Hunting!
Author: The Octoparse Team
For more information about Octoparse, please click here.
Sign up today.
