Blog > Web Scraping > Post

eCommerce pricing strategy: how to build a price comparison model

Monday, November 23, 2020


Table of Contents

0:15 Why they track prices?    

0:40 What can you get with price data?

1:07 How to generate more profits? 

1:40 Why web scraping?

2:10 How to use Octoparse to get price data on Wayfair?

3:29 Conclusions









Why you should track prices

Almost all of our eCommerce users are tracking product prices online. Price data is a critical source for them to monitor competitors and optimize their price strategy.


Track your competitors 

First of all, knowing how your competitors price is an important part of the game. When you get a structured database for this, you can see a bigger picture of the market landscape.


In fact, many eCommerce businesses pay to get a competitor price monitoring/comparison service. Acting upon your competitor’s moves is a good way to adapt to market changes. If you are not spying on them, believe me, they are very likely spying on you.


You know what, there are price comparison websites doing this too. The owner uses web scraping to gather price information and offer consumers such service to compare prices and buy at the lowest cost. Hence, even consumers would know better than you about your product pricing.

Note: Actually when you are using web scraping to get price data of products that you are interested in, you could easily get more than prices. You can also get product name, stock number, images, customer reviews and rankings, product description and sometimes sales.



Adapt your pricing strategy intelligently

One of the difficulties of competing in a market is that the information gap exists and you have competitors who may be better informed to react to the dynamic changes.


Using the web scraping technique, you can get product prices of both direct and indirect rivals to:

  • measure your competitiveness in the market
  • deal with the product pricing to win over more sales 
  • maximize your margins 
  • in the long run, develop a dynamic pricing strategy that adapts to market fluctuations


Most of the time, price and sales are correlated differently. When you get enough data points and find out the pattern of it, you can apply the best price to get the best sales. However, you should take into consideration the costs as well. As the third dimension cost is added, you will figure out the best mix to get the highest profits.

How to use Octoparse to scrape from Wayfair

Step 1: Enter the URL

Enter the keyword in Wayfair and copy paste the product page you want to scrape from on Octoparse. In the built-in browser, click and select one of the product card and choose "select all" - "Extract the URLs" in the Tips Panel. 


Step 2: Add pagination

Follow the Tips Panel to add pagination in the crawler - pagination is a type of loop that can configure the crawler to repeat scraping action across pages. There are different types of pagination structured in websites.

Learn more about them:

Pagination with a "Load more" button

Pagination with a "Next" button

Pagination without a "Next" button

Pagination: infinitive scroll


Step 3: Select data you want to scrape 

Go back to the workflow, and click the loop (and the built-in browser will show what items are there in the loop). Now you can select what sub-elements you want to scrape in each loop item by clicking and selecting the data field in the first loop item.

After you have gotten all data fields you want, you can run the task.


Note: You can also check this guide if you are using Octoparse 7.3. You can find more tips on Help Center by searching keywords.




Almost all visible price data online can be scraped down using Octoparse. Web scraping feeds your customized price comparison model with accurate data and makes competitor monitoring and pricing strategy optimization easier.


Author: Cici

Octoparse News: Customer Stories


How Dealogic Gets Empowered with Content Aggregation

Ecommerce Product Tracking for Successful Reselling

Web Scraping In Marketing Consultancy

Web Scraping Manages Inventory Tracking in Retail Industry

Video:3 Easy Steps to Boost Your eCommerce Buiness

We use cookies to enhance your browsing experience. Read about how we use cookies and how you can control them by clicking cookie settings. If you continue to use this site, you consent to our use of cookies.
Accept decline