Skip to main content
Market and competitive research is one of the most common uses of web scraping. Prices, stock, reviews, assortment, and store footprints are scattered across the public web, and scraping is a way to collect them continuously and turn them into a comparable form. The point isn’t to collect once. It’s to monitor the same targets over time and accumulate the data in a form where you can track change — that’s what gives market and competitive research its value.

​
Types of data to gather

What you collect depends on the objective. Pricing strategy centers on price and stock; demand sensing on reviews and volume; catchment analysis on stores and location.

​
Choosing data sources

Choose sources based on the market you’re studying.
  • E-commerce and retail: collect products, prices, and reviews from Amazon, Rakuten, and marketplaces. See How to scrape Amazon and How to scrape Rakuten Ichiba.
  • Local and store markets: collect store distribution, ratings, and status from map services. How to scrape Google Maps is a good starting point.
  • Food and services: read area-level competition from review and rating sites. See How to scrape Yelp.
  • Social and topic volume: track interest over time from post counts and engagement.
Combining multiple sources in one study — price, reputation, and volume together — gives a fuller, multi-dimensional picture.

​
Designing point-in-time monitoring

Market research data is collected repeatedly and compared, not once.
  1. Define the research target (categories, area, competitor set).
  2. Decide the fields and comparison axes (price, rating, stock, rank).
  3. Collect on a schedule (daily, weekly — often enough to see change).
  4. Store snapshots rather than overwriting (keep the collection timestamp).
  5. Normalize for comparison (align currency, units, and formatting).
Store snapshots with a collection timestamp instead of overwriting. It lets you analyze change over time — price movement, stockout periods, rating trends, and rank shifts.

​
Extending to competitor analysis

Point-in-time data lets you track competitor moves continuously.
  • Price movement: track the timing and size of competitor price changes.
  • New products and assortment: detect new SKUs and discontinued items.
  • Rating trends: track how fast review counts grow and how average scores shift.
  • Stores and location: track new openings and competitor density by area.
  • Rank and visibility: track position changes in search results and rankings.
Defining changes as events (price drop, stockout, new product, rating spike) lets you feed alerts and automated reports.

​
What to watch for

  • robots.txt and terms of service: check crawl permission and terms per target URL.
  • Personal data: minimize personal data such as reviewer identity — don’t collect it if you don’t need it.
  • Prefer official APIs: when an official API or partner feed fits the use case, use it first.
  • Server load: on scheduled runs, keep frequency and volume modest so you don’t overload the target site.
For the criteria that determine legality, see Is web scraping legal?.

​
Implementing with Octoparse

Octoparse lets you build a workflow that separates discovery (listings, search) from extraction (detail, reviews), and running it on a cloud schedule automates point-in-time monitoring. Accumulating results as snapshots gives you time-series comparison of prices and ratings. Set up recurring runs with scheduled runs, and use the Octoparse scraping templates as a starting point for target sites.