Blog > Octoparse > Post

Python Web Crawler? Create Your Own Crawler in 4 Step!

Tuesday, April 12, 2016

The HTMLParser module for Python can help you parse the HTML tag or other elements inside, and is truly an easy way to deal with HTML. What if I tell you there is an automation tool that can parse HTML even more efficiently? Octoparse, a free and easy-to-use web data extractor, can parse any web pages and extract HTML elements. You can totally nail it within 3-5 minutes if you learn to use Octoparse for a while.


I’ll show you how to use Octoparse to parse Amazon. Let’s just say how to build an Amazon crawler using Octoparse.


Step 1.

Enter the URL which you want to extract data from.



Step 2.

Choose the part of content you want to scrape. Usually Octoparse will be pick up a whole piece of data before you can extract more specific information. 



When you choose the second part which shares the same layout with the previous part, Octoparse will automatically get all the parts with similar layout.



Step 3.

Select what you want to extract. Here we will extract product name, price, brand, picture, and etc.



Step 4.

Configure pagination. In most cases, we need to extract data from multiple web pages.



Now your web crawler has been created! Run the task on your own computer and Octoparse will crawl and parse data from multiple pages. Certainly, you can export the structured data you just extracted from these web pages to different data formats like Excel, Text, HTML and etc, or import the data into you own database. Octoparse API and cloud service will definitely make your own crawler more efficiently and stably.



Octoparse can be used for many other purposes like price comparison and market strategy. So, how long would it take to create such a useful crawler? Less than 5 minutes! Unbelievable, right? Actually it will take you more than 5 minutes unless you spend 10 minutes watching Octoparse tutorials firstly and use two Octoparse modes (Wizard Mode and Advanced Mode) by following the prompts. Sign up now to see if you can create your own crawler in Octoparse in 5 minutes. 





Author: The Octoparse Team




Download Octoparse Today



For more information about Octoparse, please click here.

Sign up today.



Author's Picks


About Octoparse

Octoparse 6.0 is Now Available

What A Price Monitor Can Help you?

Examples of Businesses Who Use Data Scraping

Collect Data from Facebook

Collect Data from Craigslist

Collect Data from LinkedIn




Download Octoparse to start web scraping or contact us for any
question about web scraping!

Contact Us Download