Octoparse vs. Scrapinghub (Portia) - Which is a better scraper bot?

Tuesday, March 06, 2018

Portia, one of the platform of Scrapinghub, is a visual web scraping tool. In this article I will put it head to head with Octoparse to see how these two tools compare (check here for another comparison between Octoparse and import.io).

Features Comparison

Feature	Octoparse	Portia
Authoring environment	Desktop app for Windows (available fo MAC with virtual machine)	Web based application
Selecting elements	Point-and-click, XPath	Point-and-click, CSS selectors, XPath
Pagination	Clicking on pagination links or manually entering the XPath(websites without "Next page" links)	Exclusively by exploration
Scraper logic	Variables, loops, conditionals, function calls (via RegEx, XPath)	Selecting and extracting only
Pop-ups, infinite scroll, hover content,drop downs, tabs	Yes	With external libraries
Signing in to accounts	Yes	Yes
Entering into search boxes	Yes	No
Javascript, Ajax and dynamic content	Yes	Yes, when subscribed to Splash
Debugging	No	Visual debugger and server snapshots
Transforming data	Regex expressions	Partial annotations
Speed	Fast parallel execution	Fast parallel execution
Hosting	Hosted on cloud of Octoparse servers if subscribed to Octoparse plans or on local machine with free version	Hosted on cloud of ScrapingHub servers
IP Rotation	Included in paid plans or manual IP proxy in free plan	With Crawlera plan
Scheduling runs	With a premium Octoparse account	With a Scrapy Cloud plan
Cloud service	Yes	With a Scrapy Cloud plan
Data export	CSV, Excel, Txt, Databases, API	CSV, JSON, XML, API
Support	Professional support, tutorials, community support	Community support

�

What can Octoparse do for you?

With its simple point-and-click UI, extracting data with Octoparse can be rather easy. Octoparse, a visual web scraper works by mimic human browsing behaviors and can be instructed to interact with the website in various ways, thus allowing scraping dynamic and more complex websites.

Some of the more advanced features worth mentioning include scraping behind a login, select the different options from a dropdown menu, searched based extraction as well as dealing with infinite scrolling, etc. Octoparse is also pretty neat by having a workflow showing all the different steps for any extraction task and I did find it useful for sorting out all the logic behind the extraction.

Octoparse

Furthermore, the built-in RegEx tool and XPath tool come in handy if one is looking to customize data extracted.

Octoparse supports multi-step extractions and eventually combines the data together in one output. An extraction task can be set up quickly with only a few steps: open web page - select items - extract data - get data - export data.

For more detailed information you could check out Octoparse Tutorials.

What can Portia do for you?

Making a crawler in Portia is very similar to that in Octoparse. Like Octoparse, Portia can automatically detect similar items on any page. Portia will find items that are structured the same as the sample you have created and this step will continue until you either tell it to stop, reach the limit of your ScrapingHub plan or if the software finishes checking every page.

The way Portia gets data can lead to unexpected or unwanted data. To make up for this issue, Portia provides regular expression to narrow down its search. But still, big sites like Amazon are hard to be navigated this way. Check below for a simple example of how Portia crawler works.

What is the difference between Octoparse and Portia?

As previously mentioned, Portia can only get data from pages that have the exact same layout, but going between search results and more detailed product description pages is not possible. Portia can't interact with dropdown menus, pop-up windows, infinite scrolling pages, or pagination unless you use external libraries. It can't deal with captcha, which is quite common for most web pages. And you would not know which pages Portia gets its data from as the scraper can't be controlled with any regular expressions. As for transforming the data in the regular expression or modifying the XPath, there are no tools available for you will need to master XPath and regular expression if you want to explore more on Portia.

According to my test, there’s no difference in the extraction speed of a Portia scraper running on Scrapinghub cloud and an Octoparse crawler running on my local machine. However, with the Octoparse cloud service which enables extraction in the cloud, running an extraction can be faster than that on Portia.

Cost Comparison

There’s no doubt that Octoparse has overwhelming advantages. Both Portia and Octoparse provide free versions, but their pricing structures are quite different.

Portia’s Pricing

Portia’s pricing depends on the number of ScrapingHub cloud units you purchase, and the use of extra libraries in ScrapingHub. Buying additional ScrapingHub cloud units would make your crawling faster. Also, if you buy one cloud unit, you could save your data for 120 days on the cloud.

ScrapingHub cloud unit price is $9 each. Check below for details.

ScrapingHub Cloud

If you want to make full use of Portia, you need to subscribe to ScrapingHub’s other paid services, IP rotation service Crawlera and JavaScript friendly browser Splash. Crawlera’s paid plans ranging from $25/month to $500/month are limited by the number of monthly requests and concurrent requests, while Splash’s plans ranging from $25/month to $100/month are based on the different speed.

Crawlera Monthly Plans

Splash Monthly Plans

Octoparse’s Pricing

Unlike Portia, which is similar to infrastructure-as-a-service(IAAS), Octoparse offers more conventional software-as-a-service (SAAS) packages with free, basic, standard and professional plans. Price ranges from $19/month to $249/month with three different kinds of subscriptions including monthly, quarterly and annual. You can see a summary of Octoparse's pricing plans below.

Octoparse Pricing

Both Octoparse and ScrapingHub provide custom solution if you need a more customized web scraping plan. They both also provide data service for data on demand.

Conclusion

While Portia needs to work with other platforms of Scrapinghub at a higher level, Octoparse has most of the features bundled together for easier implementation. For entry-level users, Octoparse offers the same level of web scraping power and scale of Portia in a much easier-to-use package. It’s not difficult to start an Octoparse crawler or Portia scraper, but it would take you quite a while if you want to explore more.

I wanted this comparison to be as fair as possible. However, since Portia is an open-source project while Octoparse is a UI tool, I can’t help if my bias for Octoparse may have lead to some unfair criticism of Portia. If you find something wrong with the information above or if you are an experienced Portia user, please contact me here. Thanks.

Octoparse vs. Import.io comparison: which is best for web scraping?

Octoparse vs. Content Grabber comparison: which is better for web scraping?

Web Scraping Service vs. Automatic Web Scraper: Which is the best option for web scraping?

Top 20 Web Crawler Tools to Scrape the Websites

What is data extraction?