Blog > Post

The Primary Use of Regular Expression in Data Processing

Monday, August 1, 2016

Previously, I introduced the usage, the pros and cons of Regular Expressions in extracting HTML content.


Using Regular Expression to Match HTML


Advanced Text - Recommendations to Handle HTML With Regular Expression


Regular expressions can be used:


  1. Data validation. (A string complies with email rule or is a phone number, etc.)


  1. Replace text. (Replace the characters we don’t want with other characters.)


  1. Extract substrings from a string based on a pattern match. (Search specific text in the document.)


In data processing, we mainly use Regular Expressions to replace text and extract substrings.


Examples: Remove all non-alphanumeric characters other than invalid characters (@-.) from the string, then return a string.


Sample code (C#)


using System;

using System.Text.RegularExpressions;

void Main()




static string CleanInput(string strIn)


   // Replace invalid characters with empty strings.

   return Regex.Replace(strIn, @"[^\w\.@-]", "");



The output







We can see the characters (* # $ %) which we believe to be invalid characters are replaced by an empty string, which removed these invalid characters


Examples: get the domain name of URL


Sample code (C#)


using System;

using System.Text.RegularExpressions;

void Main()


    string url = "http://www.octoparse.com/pricing/";

    Regex r = new Regex(@"((\w)+\.)+\w+",RegexOptions.IgnorePatternWhitespace);




The output





From the code above, we see that Regular Expression can help you extract strings with characteristics of domain names from the URL.

For hundreds of thousands of URL like this, you can extract their domain names with only a regular expression.




Author: The Octoparse Team




Download Octoparse Today



For more information about Octoparse, please click here.

Sign up today.



Author's Picks


About Octoparse

Octoparse 6.0 is Now Available

What A Price Monitor Can Help you?

Examples of Businesses Who Use Data Scraping

Collect Data from Facebook

Collect Data from Craigslist




Download Octoparse to start web scraping or contact us for any
question about web scraping!

Contact Us Download
We use cookies to enhance your browsing experience. Read about how we use cookies and how you can control them by clicking cookie settings. If you continue to use this site, you consent to our use of cookies.
Accept decline