Extract data from source code

1) Extract data from inner HTML 

HTML is the standard markup language for creating web pages. When we extract the inner HTML of an element on the page, we will get the HTML markup contained within the element. So, for the information shown in the form of a picture or icon, we can capture its inner HTML first, then further extract the target data from the extracted code by using data reformat tools.

Take the star rating of a restaurant on Yelp.com as an example.

• Locate "star-rating" on the webpage and click it
• Select "Extract inner HTML of the selected element" from the Action Tips panel

Switch to the Workflow Mode by toggling the Workflow switch . The extracted inner HTML had been added to the "Data field" section,

  <img class="offscreen" src="https://s3-media2.fl.yelpcdn.com/assets/srv0/yelp_design_web/9b34e39ccbeb/assets/img/stars/stars.png

" alt="4.0 star rating" height="303" width="84">

Notice the number value of the star rating (4.0) is included in the code extracted though it was not directly available on the web page. Now we have the code, we can further pinpoint "4.0" from it via reformatting the data with Regular Expression (learn more about reformatting HTML in Part 3). 

2) Extract data from outer HTML 

Outer HTML is an element property that includes the opening and the closing tags as well as the content. So, capturing the outer HTML can technically provide more information than inner HTML. If the information needed cannot be found in the inner HTML, it is still possible to locate it in the outer HTML.

The steps to extract outer HTML is similar to that of inner HTML:

• Click the data needed
• Select "Extract outer HTML of the selected element" from "Action Tips"

3) Reformat data with RegEx tools

Data reformat tools are very helpful to process the data extracted, to clean the data. There are 8 built-in data reformat tools in Octoparse. For the purpose of this tutorial, we'll cover two HTML-related reformat tools.

To access the data reformat tools,

• Select the data field to reformat
• Click on  to customize the field
• Click "Refine extracted data"
• Click "Add step"

1. HTML Transcoding

Once you have the inner/outer HTML code extracted, you can convert the HTML tags into plain text using "HTML transcoding". For example, transcode "&gt" into ">" and "&nbsp" into a space.

• Select "HTML transcoding"
• Click "Evaluate" and confirm the output
• Click "OK" to save the settings

2. Match with Regular Expression

• Select "Match with Regular Expression"
• Click "Try RegEx Tool"
• Enter the match criteria: start with " alt=" ", end with "star rating"
• Click "generate", then "Match", you will see the number value of star rating (4.0) is matched. 
• Click "Apply" 
• Click "OK" to save the settings