Skip to main content
Reviews and ratings are data that reflect customer satisfaction, complaints, and the quality of demand. Web scraping is a way to collect reviews scattered across multiple sites, continuously, and shape them into a form usable for sentiment and reputation analysis. This page is a technical playbook for the data design that takes you from collecting reviews to analyzing them. For how to collect reviews from a specific site, see the site-specific guides; for designing the analysis, use this page.

​
Data you get from reviews

Review pages yield structured data you can use for analysis. Combining text with score surfaces patterns a score alone can’t — for example, “high score but the text describes a complaint.”

​
Choosing data sources

Choose review sources based on what you’re analyzing.
  • E-commerce and product reviews: collect product reviews and buyer voices from Amazon and Rakuten. See How to scrape Amazon and How to scrape Rakuten Ichiba.
  • Store and service reviews: read store ratings and area reputation from review sites. See How to scrape Yelp.
  • Map service reviews: collect local-business reputation from Google Maps reviews. See How to scrape Google Maps.
  • Social and forums: collect reputation and topic volume for products and brands from posts.
Reading reviews across multiple sources reveals differences in rating by channel and the overall shape of reputation.

​
Designing collection

To get review data that’s usable for analysis, design at the collection stage.
  1. Define the analysis target (product, store, brand, period).
  2. Decide the fields to collect (text, score, and date at minimum).
  3. Choose the collection method (bulk backfill of past reviews, or point-in-time monitoring of new ones).
  4. Deduplicate (detect duplicate and re-posted reviews).
  5. Normalize (align language, encoding, and rating scale).
Always collect the rating score and post date alongside the review text. Score serves as training or validation data for sentiment, and post date drives time-series analysis. Text alone narrows what you can analyze substantially.

​
Extending to analysis

Once collected and normalized, review data feeds a range of analyses.
  • Sentiment classification: classify text as positive/negative/neutral and track the shifting mix.
  • Topic extraction: extract recurring topics (price, quality, delivery, service) and pinpoint complaints.
  • Rating trends over time: measure the impact of initiatives or issues from shifts in score or negative rate.
  • Product and store comparison: compare competitors or your own products and stores by rating and review content.
  • Score-text correlation: read the gap between score and text to see what a score alone hides.
Monitoring new reviews and defining events (such as a spike in negative rate) lets you feed alerts and automated reports.

​
What to watch for

  • Copyright in review text: review text is highly copyright-protected. Being able to collect it is separate from being free to republish, redistribute, or reuse it. Distinguish internal use for analysis from public reuse.
  • Personal data: minimize personal data such as reviewer names — don’t collect or store it if analysis doesn’t need it.
  • robots.txt and terms of service: check crawl permission and terms per target URL.
  • Server load: keep frequency and volume modest on scheduled collection.
For the criteria that determine legality, see Is web scraping legal?.

​
Implementing with Octoparse

Octoparse renders pages in a headed, real browser, so reviews shown through incremental loading can be collected too. You can visually configure both a bulk backfill of past reviews and recurring collection of new ones, and accumulate them continuously with cloud runs. Set up recurring collection with scheduled runs, and use refine and clean extracted data to shape the reviews you collect.