> ## Documentation Index
> Fetch the complete documentation index at: https://www.octoparse.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Data collection for market and competitive research

> How to collect data for market and competitive research with web scraping: the types of data to gather, choosing sources, designing point-in-time monitoring, extending to competitor analysis, and what to watch for.

Market and competitive research is one of the most common uses of web scraping. Prices, stock, reviews, assortment, and store footprints are scattered across the public web, and scraping is a way to collect them continuously and turn them into a comparable form.

The point isn't to collect once. It's to monitor the same targets over time and accumulate the data in a form where you can track change — that's what gives market and competitive research its value.

## Types of data to gather

| Data type               | What it tells you                                 | Common sources                 | Key fields                              |
| ----------------------- | ------------------------------------------------- | ------------------------------ | --------------------------------------- |
| Price and stock         | Price bands, price moves, availability            | E-commerce, marketplaces       | Price, shipping, stock, points, sales   |
| Reviews and reputation  | Satisfaction, complaints, quality of demand       | E-commerce, review sites, maps | Score, count, text, date                |
| Products and assortment | Lineup, new products, category structure          | E-commerce, brand sites        | Product name, category, SKU, attributes |
| Stores and location     | Store distribution, catchment, competitor density | Map services, store finders    | Store name, address, category, hours    |
| Trends and volume       | Rising interest, topic shifts                     | Social media, Q\&A, reviews    | Post count, engagement, keywords        |

What you collect depends on the objective. Pricing strategy centers on price and stock; demand sensing on reviews and volume; catchment analysis on stores and location.

## Choosing data sources

Choose sources based on the market you're studying.

* **E-commerce and retail**: collect products, prices, and reviews from Amazon, Rakuten, and marketplaces. See [How to scrape Amazon](/docs/en/academy/scrape-amazon) and [How to scrape Rakuten Ichiba](/docs/en/academy/scrape-rakuten).
* **Local and store markets**: collect store distribution, ratings, and status from map services. [How to scrape Google Maps](/docs/en/academy/scrape-google-maps) is a good starting point.
* **Food and services**: read area-level competition from review and rating sites. See [How to scrape Yelp](/docs/en/academy/scrape-yelp).
* **Social and topic volume**: track interest over time from post counts and engagement.

Combining multiple sources in one study — price, reputation, and volume together — gives a fuller, multi-dimensional picture.

## Designing point-in-time monitoring

Market research data is collected repeatedly and compared, not once.

1. Define the research target (categories, area, competitor set).
2. Decide the fields and comparison axes (price, rating, stock, rank).
3. Collect on a schedule (daily, weekly — often enough to see change).
4. Store snapshots rather than overwriting (keep the collection timestamp).
5. Normalize for comparison (align currency, units, and formatting).

<Tip>
  Store snapshots with a collection timestamp instead of overwriting. It lets you analyze change over time — price movement, stockout periods, rating trends, and rank shifts.
</Tip>

## Extending to competitor analysis

Point-in-time data lets you track competitor moves continuously.

* **Price movement**: track the timing and size of competitor price changes.
* **New products and assortment**: detect new SKUs and discontinued items.
* **Rating trends**: track how fast review counts grow and how average scores shift.
* **Stores and location**: track new openings and competitor density by area.
* **Rank and visibility**: track position changes in search results and rankings.

Defining changes as events (price drop, stockout, new product, rating spike) lets you feed alerts and automated reports.

## What to watch for

* **robots.txt and terms of service**: check crawl permission and terms per target URL.
* **Personal data**: minimize personal data such as reviewer identity — don't collect it if you don't need it.
* **Prefer official APIs**: when an official API or partner feed fits the use case, use it first.
* **Server load**: on scheduled runs, keep frequency and volume modest so you don't overload the target site.

For the criteria that determine legality, see [Is web scraping legal?](/docs/en/academy/is-web-scraping-legal).

## Implementing with Octoparse

Octoparse lets you build a workflow that separates discovery (listings, search) from extraction (detail, reviews), and running it on a cloud schedule automates point-in-time monitoring. Accumulating results as snapshots gives you time-series comparison of prices and ratings.

Set up recurring runs with [scheduled runs](/docs/en/platform/schedules), and use the [Octoparse scraping templates](https://www.octoparse.com/template) as a starting point for target sites.

## Related resources

* [Price monitoring](/docs/en/academy/price-monitoring) — a design focused on point-in-time price tracking
* [E-commerce data collection](/docs/en/academy/ecommerce-data) — collecting products, prices, and reviews
* [How to scrape Amazon](/docs/en/academy/scrape-amazon) — a data source for e-commerce market research
* [How to scrape Google Maps](/docs/en/academy/scrape-google-maps) — a data source for local and catchment analysis
* [Is web scraping legal?](/docs/en/academy/is-web-scraping-legal) — legality and scope of collection