The Primary Use of Regular Expression in Data Processing

Previously, I introduced the usage, the pros and cons of Regular Expressions in extracting HTML content.

Using Regular Expression to Match HTML

Advanced Text – Recommendations to Handle HTML With Regular Expression

Regular expressions can be used:

1. Data validation. (A string complies with email rule or is a phone number, etc.)
2. Replace text. (Replace the characters we don’t want with other characters.)
3. Extract substrings from a string based on a pattern match. (Search specific text in the document.)

In data processing, we mainly use Regular Expressions to replace text and extract substrings.

Examples:

Remove all non-alphanumeric characters other than invalid characters (@-.) from the string, then return a string.

Sample code (C#)

using System;

using System.Text.RegularExpressions;

void Main()

{

    Console.WriteLine(CleanInput(“@octoparse*.123#facebook$%-JS”));

}

static string CleanInput(string strIn)

{

   // Replace invalid characters with empty strings.

   return Regex.Replace(strIn, @”[^\w\.@-]”, “”);

}

The output

@octoparse.123facebook-JS 

We can see the characters (* # $ %) which we believe to be invalid characters are replaced by an empty string, which removed these invalid characters

Examples:

get the domain name of URL

Sample code (C#)

using System;

using System.Text.RegularExpressions;

void Main()

{

    string url = “http://www.octoparse.com/pricing/”;

    Regex r = new Regex(@”((\w)+\.)+\w+”,RegexOptions.IgnorePatternWhitespace);

    Console.WriteLine(r.Match(url).Captures[0].Value);

}

The output

www.octoparse.com

From the code above, we see that Regular Expression can help you extract strings with characteristics of domain names from the URL.

For hundreds of thousands of URL like this, you can extract their domain names with only a regular expression.