XPath Introduction -- Use XPath to Scrape Web Data
Monday, September 05, 2016
XPath is a language used for locating certain elements in XML documents. One of the main reasons for using XPath is when you can’t directly find certain data on the web page. Since HTML is part of XML family, you can use XPath in HTML as well.
In the context of web scraping, XPath is a useful tool that helps you get a path to a certain place of HTML and extract whatever you can find there.
In this article, I will show you how you can use XPath to scrape websites and extract valuable data that you can use it for whether SEO campaign, social media campaign, content marketing, etc.
Find XPath Using Firefox & Chrome
The first thing that you need to do before anything is to install Firefox or Chrome.
If you are using Firefox browser, you need to install the plugin - Firebug, in order to see the XPath.
(Note: FireBug is used to easily look up the HTML/SCC of any element on a web page so that makes it very easy to debug and develop web pages.)
If you open a web page in Firefox, click Firebug button and click an element in the page to inspect. It will bring out all of the XPath.
Or you can simply right click on the page and we have an option “Inspect in FirePath”
Right click the line and choose “Copy XPath”.
If you are using Chrome, right click the web page and choose “Inspect”. It will bring out the HTML. Each line of this HTML has its own XPath. You can expand or contract each line of this.
Then simply right click on the line and choose “Copy XPath”.
Use XPath to Scrape Specific Data
One of the really awesome things is that you can run XPath directly within Octoparse.
If you would like to extract this line, copy and paste the XPath in built-in XPath tool.
So you can easily scrape specific data you want.
Have A Tip for Using XPath?
If you have any tip for scraping data using XPath, drop us a message here.
We really would welcome your thoughts, suggestions, recommendations and any feedback that you can give us. Every one of which we will take most seriously!