Octoparse documentation

Start here
Academy
Learn web scraping — concepts, mechanics, use cases, and site guides.
Platform
Product reference — tasks, cloud runs, exports, anti-blocking, team features.
OpenAPI
REST API for custom integrations, task automation, and data export.
CLI
Command-line tools for running tasks, exporting data, and creating local tasks from URLs.
MCP server
Connect Octoparse to AI agents — ChatGPT, Claude, Cursor, Antigravity CLI / Gemini CLI.
Templates
Prebuilt scraping templates for popular websites — pick a category and start without building from scratch.
E-commerce
Amazon, eBay, Walmart, Shopify — product, price, and review data.
Social media
TikTok, Twitter/X, YouTube, Reddit — posts, comments, and engagement.
Maps & local business
Google Maps, Yelp, Yellow Pages — business names, addresses, and contacts.
Real estate & jobs
Zillow, Realtor, Indeed, LinkedIn Jobs — listings and structured details.
Browse the full template gallery or read the templates guide.
Resources
Help center
Setup guides, product documentation, and troubleshooting articles.
Template gallery
Pre-built scraping templates for common websites and use cases.
Download
Download the Octoparse desktop app.
Need help? Contact our support team.
Community & social
X (Twitter)
Product updates and news.
LinkedIn
Connect with the team.
YouTube
Tutorials and walkthroughs.
Discord
Chat with the community.
GitHub
Open-source projects and tools.


How it works

Octoparse turns web browsing actions into repeatable extraction workflows. You define what to collect, how the website should be navigated, and where the results should go. Most workflows follow four stages: build, test, run, and export.
Workflow overview
1
Build
Start from a URL, template, or custom task. Select the data fields you want and define actions such as clicking, scrolling, pagination, and opening detail pages.
2
Test
Run a small sample to confirm that Octoparse captures the right fields, records, and page sequence.
3
Run
Execute the task locally for testing or in the cloud for scheduled, unattended, and larger-scale extraction.
4
Export
Send the extracted results to files, spreadsheets, databases, cloud storage, or other connected systems.
Build the task
A task defines how Octoparse interacts with a website. You can build a task by:
Using a template
Letting Auto-detect identify page data automatically
Selecting elements manually in the no-code builder
Adding actions such as click, scroll, loop, pagination, and wait
Refining field values before export
The goal is to turn the website interaction into a reusable workflow.
Test the extraction logic
Before running a task at scale, test a small sample. Check whether:
CheckWhy it matters
Fields are correct
Prevents exporting the wrong values
Field names are clear
Makes downstream data easier to use
Pagination works
Ensures the task moves across result pages
Detail pages open correctly
Confirms nested page workflows are captured
Sample output looks clean
Reduces cleanup after export
Testing is especially important for dynamic pages, login-protected pages, infinite scroll, popups, and websites where data is loaded after user actions.
Run locally or in the cloud
Octoparse supports different run options depending on the task and your plan.
Run typeBest for
Local extraction
Testing, debugging, and tasks that rely on your local environment
Cloud extraction
Scheduled, unattended, and higher-volume extraction
Boost mode
Cloud tasks that need more speed or concurrency, when supported
The best option depends on the website, task complexity, required frequency, and whether the task needs to keep running when your computer is off.
Export the data
After a task runs, Octoparse stores the extracted results as structured records. Common export destinations include:
CSV
Excel
JSON
HTML
XML
Google Sheets
Databases
Cloud storage
For automated workflows, use scheduled export or integrations so data can move downstream without manual downloading.
Related pages
Local vs cloud extraction
Compare run environments and choose the right execution mode.
Export formats
Learn which output formats Octoparse supports.


What is Octoparse?

Octoparse is a no-code web data extraction platform. It helps you turn website content into structured data by building scraping workflows visually, running them locally or in the cloud, and exporting the results to files, databases, cloud storage, or downstream systems. Instead of writing and maintaining custom scrapers from scratch, you can use Octoparse to define what to collect, how to navigate a website, and where the extracted data should go.
What Octoparse does
Build scraping workflows
Select data on a web page, define actions such as clicks and pagination, and turn the workflow into a reusable task.
Run data extraction
Execute tasks locally or in the cloud, depending on the website, task setup, and automation needs.
Export structured data
Send extracted results to formats and destinations such as CSV, Excel, JSON, Google Sheets, databases, or cloud storage.
How it works
At a high level, an Octoparse workflow moves through three stages:
1
Open the target website
Start from a URL, template, or custom task. Octoparse loads the page in its built-in browser so you can inspect and interact with the site.
2
Define the extraction logic
Select the data fields you want, then add actions such as clicking links, handling pagination, scrolling, logging in, or refining field values.
3
Run and export the task
Run the task locally or in the cloud, monitor progress, then export the collected data to the format or destination you need.
What Octoparse is good for
Octoparse is useful when data is available on websites but not provided in a convenient structured format. Common use cases include:
Price monitoring
Lead generation
Market research
Product and review extraction
Directory and listing collection
Real estate, job, ecommerce, and social data collection
Recurring data collection for operations or reporting workflows
Octoparse is especially helpful when the target website requires browser interaction, such as clicking buttons, scrolling pages, opening detail pages, or extracting data from dynamic content.
What Octoparse is not
Octoparse is not a general-purpose analytics, BI, or database platform. It helps collect and structure web data, but downstream analysis, modeling, and reporting usually happen in other tools. It is also not the same as a hosted scraping API where you only send a URL and receive a standardized response. Octoparse tasks are workflows: you define how the website should be opened, navigated, extracted, cleaned, and exported.
Octoparse isOctoparse is not
A no-code web data extraction platform
A BI dashboarding tool
A visual workflow builder for scraping tasks
A generic RPA platform for all desktop actions
A way to run and automate extraction tasks
A one-size-fits-all hosted scraping API
A tool for producing structured web data
A replacement for data analysis or reporting tools
Browser-based extraction
Octoparse includes a built-in browser for interacting with websites while building tasks. In normal task-building mode, clicks are used to select elements and create actions. When manual interaction is needed, such as logging in, closing popups, or solving a challenge before continuing setup, Browse Mode lets the built-in browser behave more like a regular browser. This browser-based approach is useful for websites where the data is not available in static HTML or where extraction depends on page interaction.
The exact behavior of a task depends on the website structure, task settings, extraction mode, and whether the task runs locally or in the cloud.
Where to go next
Understand the core concepts
Learn the main building blocks behind tasks, fields, actions, runs, and exports.
Build tasks without code
Learn how Octoparse lets you create scraping workflows visually.
Compare local and cloud extraction
Understand when to run tasks locally and when to use cloud extraction.
Explore export options
See how extracted data can be exported for downstream use.


Octoparse academy

Learn web scraping from the core concepts to scraper mechanics, common business use cases, and site-specific playbooks.
Explore academy
Web scraping basics
What scraping is, where it fits, how no-code and AI scraping compare, and the legal landscape.
How web scrapers work
Rendering, navigation, extraction, cloud scraping, fingerprints, proxies, and anti-bot challenges.
Use cases
Where teams put scraping to work — leads, prices, real estate, e-commerce, social.
Site-specific guides
Step-by-step playbooks for Amazon, LinkedIn, Google Maps, and Yelp.


What is web scraping?

The program that does this is usually called a “scraper” or “crawler”: it visits pages, reads their HTML content, and pulls out the specific data you need — work that would otherwise mean copying information from web pages by hand.
Common use cases
Common use cases include collecting product prices from e-commerce sites, gathering news articles, pulling job listings, monitoring competitors, or building datasets for research.
How it works
A typical scraping workflow looks like this: send an HTTP request to a URL, receive the HTML response, parse it to find the relevant elements (using CSS selectors or XPath), and then store the extracted data in a structured format like CSV, JSON, or a database.
How it differs from APIs and manual collection
Compared with collecting data by hand, scraping does the same copy-and-paste work automatically — so it’s faster, repeatable, and scales to thousands of pages, at the cost of some upfront setup and ongoing maintenance when a site’s layout changes. Compared with an API, scraping reads the same HTML a browser shows rather than a dedicated data feed. When a site offers an official API, that’s usually the more reliable and sanctioned option: the data is already structured, stable, and documented. Scraping is what you reach for when no API exists, when the API doesn’t expose the data you need, or when its limits are too restrictive — with the trade-off that it depends on the page’s markup and can break when that markup changes.
Tools and libraries
Popular tools and libraries for web scraping include Python’s Beautiful Soup and Scrapy, JavaScript’s Puppeteer and Cheerio, and no-code platforms like Octoparse. No-code platforms take a different approach from script-based libraries: Octoparse, for example, runs pages in a real browser and lets you build a scraper by pointing and clicking — simulating the way a person browses and selects data, rather than writing extraction code. For pages that rely heavily on JavaScript to render content, headless browsers (like Puppeteer or Playwright) can simulate a real browser to access dynamically loaded data.
Things to keep in mind
A few things to keep in mind when scraping: always check a site’s robots.txt file and terms of service to understand what’s allowed, be respectful with request rates to avoid overloading servers, and be aware that some jurisdictions have legal restrictions around scraping certain types of data.


Anatomy of a web scraper

A web scraper isn’t a single thing — it’s a pipeline. Whether you write it in Python, drive a headless browser yourself, or build it visually in a no-code tool, the same stages happen in the same order. Knowing the stages makes it much easier to see where a scraper breaks, where it slows down, and which technique fixes what.
1
Fetch the page
Send an HTTP request, get HTML back.
2
Render the page
Run the page’s JavaScript in a real browser.
3
Locate the data
Point at elements with XPath or CSS selectors.
4
Extract and refine
Pull values out, clean them up with regex.
5
Navigate the site
Work through pagination and login walls.
6
Store the output
CSV, JSON, a database, or a downstream API.
The first five stages run in order on every page; navigate loops back to fetch for the next page, and the whole loop runs at scale — in the cloud, behind rotating proxies, past CAPTCHAs.
Fetch the page
Everything starts with a request. The scraper sends an HTTP request to a URL and receives a response — usually HTML, sometimes JSON. For simple, server-rendered pages this single step delivers everything you need. The catch is that the response is only the initial payload the server sends; on many modern sites, that’s a near-empty shell.
Render the page
When a site builds its content with JavaScript, the data you want isn’t in the initial HTML — it appears only after the page’s scripts run. Rendering is the stage that executes those scripts in a real browser so the full content materializes. The browser runtime you use here determines how faithfully the page loads and how much it costs in memory and speed, and it’s the core of scraping JavaScript-rendered pages.
Locate the data
Once the page is fully loaded, the scraper has to point at the specific pieces you want — a price, a title, a row in a table. This is done with XPath or CSS selectors, which describe where an element sits in the page structure. Good selectors are the difference between a scraper that survives small layout changes and one that breaks on every site update.
Extract and refine
Locating an element gets you its raw content; extraction pulls the value out, and refining cleans it up. A scraped price might arrive as "$1,299.00 USD" when you only want the number. Regular expressions and post-processing rules strip, split, and reformat raw text into the structured fields you actually intend to store.
Navigate the site
Most data sets span more than one page. The scraper has to walk through pagination — next-page links, “load more” buttons, infinite scroll — to reach every record. Some content also sits behind login walls, which the scraper has to authenticate past before any of the earlier stages can run.
Run it at scale
A scraper that works once on your laptop still has to survive thousands of runs. Running in the cloud keeps tasks going without tying up your machine. Rotating proxies spread requests across many IP addresses so a site doesn’t rate-limit or block you. And CAPTCHAs and anti-bot services are the obstacles this stage exists to handle.
Store the output
Finally, the extracted data lands somewhere usable — a CSV or Excel file, a JSON export, a database, or a downstream API. This is the stage that turns a scraping run into a dataset.
How Octoparse fits
With a hand-coded scraper, you wire each of these stages together yourself: a request library, a browser driver, a selector engine, cleanup code, pagination logic, proxy management, and an export step. Octoparse collapses the whole pipeline into one visual workflow. You point and click to set selectors, and rendering, pagination, proxy rotation, cloud extraction, and export are built in — configured rather than coded. The stages are the same; what changes is that you describe what to collect instead of engineering how each stage runs.


Web scraping vs web crawling

Web scraping and web crawling are often used interchangeably, but they describe two distinct jobs that work together.
Crawling: discovery
Crawling is about discovery. A crawler navigates through a website by following links, mapping out the structure, and collecting URLs. Think of it as exploring — the goal is to find all the pages that matter. Search engines are the most obvious example: they crawl the web to build an index of what exists and where.
Scraping: extraction
Scraping is about extraction. Once you know which pages contain the data you need, a scraper visits those pages and pulls out specific fields — prices, titles, descriptions, contact details, whatever your use case requires. The goal is structured output, not discovery.
Why split them into two stages
In practice, most real data collection tasks involve both. You first crawl a site to discover all relevant page URLs (say, every product listing in a category), then scrape each of those pages to extract the actual data. While it’s possible to do both in a single pass, splitting them into two separate stages is often the smarter approach — especially for complex sites. The crawling phase produces a clean list of URLs, and the scraping phase works through that list to extract content. This separation has practical benefits: you can run each stage independently, retry failures without redoing the whole job, and parallelize the scraping phase across many pages simultaneously, which significantly speeds up large-scale collection.
How Octoparse handles both
Octoparse addresses both sides of this with dedicated AI-powered modes. Its AI Crawl feature handles the discovery stage — analyzing a site’s link structure and automatically generating a workflow to collect target URLs across pagination, categories, or nested pages. Its AI Scrape templates then take those URLs and extract structured data from each page. By treating these as two connected but independent tasks, users can take advantage of Octoparse’s parallel execution infrastructure: once the URL list is ready, hundreds or thousands of pages can be scraped concurrently in the cloud rather than sequentially, turning what might be a day-long job into something that finishes in minutes.
The takeaway
Think of crawling and scraping as two phases of the same pipeline. Crawl first to build your target list, scrape second to get the data. Keeping them separate gives you more control, better error handling, and the ability to scale the extraction phase horizontally — which matters a lot when you’re dealing with sites that have thousands or millions of pages.


What is AI web scraping?

Traditional scraping relies on manually defined rules — you specify exact CSS selectors or XPath expressions, or regular expressions, to locate data on a page. AI-powered scraping, by contrast, can understand page structure and content more flexibly, reducing the manual configuration effort and handling variations between pages more gracefully.
Three ways AI is used in scraping
There are a few key ways AI is applied in web scraping today. One is automatic detection of page structure — instead of a user manually clicking on each field, the tool analyzes the page layout and intelligently identifies repeatable data patterns like product listings, article feeds, or contact directories, then generates the extraction logic on its own. Another is using AI to write complex matching rules like regular expressions, which are notoriously tricky to get right by hand. Rather than crafting regex patterns yourself, you describe what you need in plain language and the AI generates the pattern for you. A third approach involves feeding raw HTML directly to an AI model and letting it extract structured data based on a prompt or template — essentially treating the HTML as unstructured text and using language understanding to pull out the relevant fields.
How Octoparse uses AI
To give a concrete example, Octoparse incorporates all three of these approaches. Its auto-detect feature scans a target webpage and automatically generates a scraping workflow, identifying data fields and pagination without manual setup. It also offers AI-assisted regex generation for pattern-matching tasks, and provides AI-powered HTML extraction templates that can parse page content directly through a language model. These features sit alongside its traditional visual configuration tools, so users can choose the level of automation that fits their task.
Benefits and limitations
The broader benefit of AI in scraping is resilience. Traditional rule-based scrapers tend to break when a website changes its layout, since the hardcoded selectors no longer match. AI-driven approaches can often adapt to minor structural changes without manual intervention, making long-running scraping tasks more maintainable. That said, AI scraping isn’t a silver bullet — it can introduce unpredictability in edge cases, and for very precise extraction requirements, explicit rules may still be more reliable. In practice, the best results often come from combining AI automation with manual fine-tuning where needed.


AI vs traditional web scraping

AI web scraping and traditional selector-based scraping each have clear strengths, and understanding the trade-offs helps you pick the right approach — or combine them effectively.
Traditional selector-based scraping
Traditional selector-based scraping works by defining explicit rules (CSS selectors, XPath, regex) to locate data on a page. Its main advantages are precision and predictability. When you write a rule that targets a specific HTML element, you know exactly what you’ll get back every time. This makes it ideal for structured, high-volume extraction where accuracy matters — think pricing feeds, inventory monitoring, or financial data. The output is deterministic, costs are low since there’s no model inference involved, and performance is fast. The downside is maintenance: when a site redesigns or changes its DOM structure, your selectors break and someone has to go in and fix them manually.
AI-driven scraping
AI-driven scraping flips the equation. Instead of rigid rules, it uses language models or pattern recognition to understand what the data means rather than where it sits in the markup. This makes it far more resilient to layout changes and much faster to set up — you can often describe what you want in plain language rather than inspecting HTML. The trade-offs are cost (model inference adds up at scale), occasional unpredictability (the same page might yield slightly different output across runs), and the fact that for very structured, repetitive tasks, it’s simply overkill.
Combining them: AI drafts, humans control
The smartest approach isn’t choosing one over the other — it’s combining them with clear roles. Think of it as AI drafting and humans controlling. AI handles the initial heavy lifting: analyzing page structure, generating extraction logic, writing regex patterns, and producing a working first version of the scraping workflow. Then the human reviews, adjusts, and locks down the rules — fine-tuning selectors, removing false positives, and ensuring the output meets quality requirements before anything goes into production. This keeps the speed advantage of AI during setup while putting humans in charge of the final accuracy.
How Octoparse applies this
Octoparse reflects this philosophy well. Its auto-detect feature lets AI scan a page and draft a complete scraping workflow — identifying data fields, pagination, and repeatable patterns automatically. Users then step into the visual editor to review what the AI proposed, adjust selectors, add or remove fields, and refine the logic to their exact needs. AI-assisted regex generation and HTML extraction templates follow the same pattern: the AI produces a working draft, the user validates and tweaks. Once the task is locked in, it runs on deterministic selector-based logic for consistency at scale. The AI gets you to 80% in minutes; human judgment takes it the rest of the way.
The bottom line
Let AI do the drafting where speed and adaptability matter, keep humans in the loop for quality control and edge cases, and run production tasks on stable, rule-based logic. This gives you fast setup, reliable output, and a clear line of accountability at every stage.


What is no-code web scraping?

It’s a visual alternative to traditional web scraping: instead of building scripts with libraries like Beautiful Soup or Scrapy, you use point-and-click tools that handle the technical work for you.
How it works
These tools typically work by letting you open a webpage in a built-in browser, click on the data elements you want to extract (like product names, prices, or reviews), and the tool figures out the underlying patterns to scrape similar data across multiple pages. You can then export the results as a spreadsheet, CSV, or into a database.
Who it’s for
No-code scraping is a good fit for people who need data but don’t have programming experience — marketers, researchers, business analysts, or anyone who wants quick results without a development investment. It’s also useful for simpler, recurring tasks where building a custom script would be overkill.
What no-code platforms can do
Some well-known platforms in this space include Octoparse, ParseHub, and Browse AI. Taking Octoparse as an example, it illustrates what modern no-code scrapers can typically do: configure tasks through a visual interface with a built-in browser, use pre-built templates for popular sites to speed up setup, handle JavaScript-rendered pages with infinite scrolling or Ajax loading, run tasks in the cloud on a schedule so you don’t tie up your own machine, and manage anti-blocking measures like IP rotation automatically. Most mature platforms in this category offer a similar feature set, though they each have their own strengths depending on the use case.
Beyond the visual interface
It’s also worth noting that some platforms go beyond the traditional no-code interface to accommodate different user needs. Octoparse, for instance, offers multiple delivery options — not just the visual tool, but also API access and code-based SDKs for developers who want to integrate scraping into their own workflows, as well as managed data delivery services for teams that would rather receive clean, structured datasets directly without running tasks themselves. This kind of layered approach bridges the gap between fully no-code users and more technical teams, making it possible for organizations with mixed skill sets to use a single platform.
The trade-off
The trade-off with no-code tools is flexibility. Custom code gives you full control over edge cases, complex login flows, or unusual page structures, while no-code tools can sometimes struggle with highly dynamic or unconventional websites. But for the majority of common scraping tasks, they’re more than capable and significantly faster to set up.


Cloud web scraping explained

Running a scraper on your own machine is enough for many one-off jobs, and local extraction can still be accelerated with multiple processes or concurrent browser sessions. Cloud scraping exists for the moment local capacity and manual operation stop being enough: the run takes hours, the data needs to refresh every morning, the site needs many pages collected in parallel, or the task needs browser and network resources your laptop should not have to provide. At a technical level, cloud scraping moves the execution environment away from the operator’s computer and into managed servers. The scraper’s rules still define what to collect and how to navigate; the cloud supplies the compute, scheduling, concurrency, networking, monitoring, and retry behavior around those rules.
Why scrapers move to the cloud
Cloud extraction solves four common problems. First, it removes the operator from the runtime. A local scraper depends on one machine staying awake, connected, and available. A cloud scraper can run unattended, overnight, or on a recurring schedule. Second, it gives the scraper a larger execution pool. Local tools can run multiple processes, but they are still bounded by one machine’s CPU, memory, bandwidth, and uptime. In the cloud, large jobs can be split into subtasks: one category per worker, one URL range per worker, one location per worker, or one detail-page batch per worker. Many pieces can run at the same time and merge the results afterward. Third, it centralizes resources that are awkward to manage locally: browser instances, memory, CPU, queues, retries, logs, IP pools, region selection, and storage. The user configures the task; the platform decides where and when to run it. Fourth, it makes recurring data collection operational. Schedules, run history, failure alerts, automatic exports, and downstream delivery matter as soon as scraping becomes a data pipeline rather than a manual action.
What cloud extraction actually adds
Cloud scraping is not only “someone else’s computer.” A useful cloud scraping system usually provides several layers around the scraper.
Managed compute
Each run needs CPU, memory, storage, and often a browser runtime. Dynamic pages are especially expensive because every worker may need to render JavaScript, wait for network calls, scroll, click, and keep session state. Cloud extraction gives those workers a controlled environment instead of competing with the operator’s desktop apps.
Subtask parallelism
Many scraping jobs can be divided into independent units. A search results task can split by keyword. A product crawl can split by category. A detail-page extraction job can split by URL list. A multi-region monitor can split by geography. Parallelism is where cloud scraping usually creates the biggest speedup beyond what a single local machine can provide. A task that takes 10 hours sequentially may finish much faster if it can be divided across many workers. The exact gain depends on site rate limits, task complexity, browser cost, local concurrency limits, and how much coordination is needed between subtasks.
Scheduling and queues
Recurring scraping needs a scheduler. Daily price monitoring, weekly lead collection, hourly inventory checks, and periodic report exports should not depend on someone pressing “Run” at the right time. Queues also matter. If too many jobs start at once, the platform needs to assign workers, respect plan limits, throttle tasks, and keep later jobs waiting instead of failing unpredictably.
Network and region resources
The network layer becomes important at scale. Cloud systems can route runs through different regions, maintain stable IP behavior for a session, and separate traffic across workers. This does not replace responsible scraping practices, but it gives the task a more suitable operating environment than a single home or office connection. For difficult sites, cloud extraction often pairs with browser fingerprinting, human-like behavior, CAPTCHA handling, and proxy rotation. Those capabilities are easier to coordinate centrally than on every user’s laptop.
Monitoring and recovery
Long-running jobs fail in ordinary ways: a page times out, a login expires, a worker crashes, a selector returns no records, or a target site slows down. Cloud systems can retry failed subtasks, preserve logs, expose run status, and make partial failure easier to diagnose. The goal is not to make scraping failure-free. The goal is to make failures visible, bounded, and recoverable.
Local vs cloud
Local extraction still has a place. It is useful for building a task, debugging selectors, testing a new site, handling sensitive data that should stay on a controlled machine, or running small and medium jobs that fit comfortably on local CPU, memory, and bandwidth. Some platforms also support local acceleration through multiple processes or concurrent task instances. Cloud extraction is better when the job is recurring, slow, large, parallelizable, browser-heavy, or operationally important. It is also better when the person who needs the data should not have to keep a machine running just to collect it.
Use local whenUse cloud when
You are building or debugging the scraper
The task needs to run unattended
The dataset is small
The dataset spans many pages or URLs
You need direct control over the machine
You need scheduling, queues, and run history
The data must stay on your device
You need managed resources beyond local concurrency
The run is occasional
The run is part of a recurring data pipeline
Integrated scraping platforms usually separate the scraping rules from the execution location. For example, a task can be designed locally in Octoparse with point-and-click actions, run locally with acceleration when appropriate, or sent to globally deployed cloud servers for execution. The important idea is that the extraction logic does not have to be rewritten for the cloud: the same workflow that opens pages, clicks, paginates, extracts fields, and exports results can run locally for testing or move to cloud infrastructure for scheduling, queues, managed browser resources, and subtask acceleration.
What cloud does not solve by itself
Cloud extraction is not a shortcut around bad scraper design. A task still needs stable selectors, a correct pagination strategy, reasonable delays, duplicate handling, and account-safe behavior for logged-in content. Running a fragile scraper on more servers usually makes the fragility show up faster. Cloud also changes the cost model. More workers, longer browser sessions, higher concurrency, and more frequent schedules all consume resources. A good cloud scraping setup balances freshness, speed, reliability, and cost instead of maximizing concurrency by default. Cloud scraping matters because it turns a scraper from a manual script into an operational data collection system. The cloud supplies the execution layer; the quality of the scraping rules still determines whether the data is complete and reliable.


Headed vs headless browsers

When you’re picking a tool for web scraping, the decision that matters most isn’t language or browser engine — it’s whether the browser runs headless (no visible window, programmatic only) or headed (a real, visible browser window). Almost the entire ecosystem — Puppeteer, Playwright, Selenium, every cloud browser API — defaults to headless. A smaller category, the integrated scraping platforms, runs headed by design; Octoparse is the clearest current example. The two choices serve different operators, succeed against different sites, and break in different ways.
Why headless became the default
Headless browsers were built for developers. They run on servers without displays, fit cleanly into containers and CI/CD pipelines, skip the rendering overhead a human user needs (window chrome, GPU compositing, autofill, telemetry), and let one machine run many concurrent sessions. When the operator is writing code, reading logs, and describing the page programmatically, there’s no reason to see it. Everything in the browser runtime landscape above the integrated-platform tier — Puppeteer, Playwright, Selenium, Splash, every cloud browser API — assumes this posture. Headless is the silent default of code-based scraping.
What headless costs you
The cost shows up in four places, and it shows up consistently:
Bot detection signals. navigator.webdriver flips to true, the user-agent says HeadlessChrome, plugins are missing, canvas and WebGL produce anomalous fingerprints. Anti-bot services like Cloudflare and DataDome are tuned to spot these. The whole “stealth variant” sub-ecosystem — puppeteer-extra-stealth, undetected-chromedriver, nodriver, Patchright — exists because plain headless leaks them.
Viewport-driven behavior. Lazy-loaded images, intersection-observer content, visibility-gated scripts — these are designed around a page actually being rendered and “seen.” Headless can fake the viewport, but the boundary is fragile and the behaviors are easy to miss.
No human in the loop. A CAPTCHA, an unexpected modal, a session-expired login — headless is blind. The script doesn’t know it’s stuck, only that it stopped returning data.
Debugging by log, not by sight. When a selector breaks on page 42 of an overnight run, you reproduce it locally — often headed — to actually see what changed. The debugging tool is the headed posture; the production tool isn’t.
None of these are dealbreakers. They’re the steady tax that code-based, headless scraping pays.
The headed-by-design category
There’s a smaller category of scraping tools where the page isn’t an internal implementation detail — it is the interface. The operator selects elements by clicking on a rendered page; watches a task execute step by step; sees a CAPTCHA when it appears and clears it; debugs by looking. Octoparse is the clearest current example. ParseHub follows the same pattern, and the older Web Scraper.io Chrome extension shares the lineage. The economics of headed-by-design only work if the runtime is purpose-built. A stock Chrome with all the human-user machinery (extensions, syncing, autofill, full GPU compositing, telemetry) is too heavy to run at scraping scale. So integrated platforms ship runtimes built specifically for the headed-scraping case — heavy enough to be authentic, light enough to run densely.
Inside Octoparse’s two headed runtimes
Octoparse ships two purpose-matched headed runtimes, and switches between them based on what the target site demands.
Electron Chromium, stripped and optimized
The first is a customized Chromium runtime built into Electron. Rather than running a stock browser, Octoparse has stripped and optimized this runtime specifically for scraping — removing unnecessary overhead like extensions, background processes, and rendering features that a human user needs but a scraper doesn’t. The result is a lightweight engine that loads pages faster, consumes significantly less memory and CPU, and can handle many concurrent sessions without bogging down a machine. Compared to running a full browser instance through Puppeteer or Selenium, this purpose-built approach offers a noticeable performance advantage, particularly when running tasks locally or on hardware with limited resources. The tight integration with Octoparse’s visual editor also means users configure and execute tasks in the same environment — no context switching between tools.
Chrome for Testing driven by Puppeteer
The second is Chrome for Testing driven by Puppeteer. This is a full, unmodified Chrome browser controlled programmatically, behaving identically to what a real user would see. It’s the better option for sites with aggressive bot detection, fingerprinting, or compatibility checks that expect a standard Chrome environment. It’s heavier on resources than the Electron runtime, but the browser authenticity it provides is sometimes essential.
A third runtime mode is in development — a browser extension that drives your own Chrome browser directly. Because extraction runs inside a real, everyday browser with authentic fingerprints and behavior, it closely mirrors genuine user activity and significantly reduces the risk of triggering anti-bot detection.
When to use which
The key advantage of having both built in is flexibility without complexity. With standalone tools like Puppeteer or Playwright, users need to manage browser binaries, handle versioning, configure launch options, and deal with infrastructure concerns themselves. Octoparse abstracts all of that away. The optimized Electron runtime handles the vast majority of tasks efficiently, while Chrome for Testing serves as a ready fallback when full browser fidelity is needed — and switching between them is a configuration choice, not an engineering project.
Whether a task runs on your own machine or in the cloud, the runtime choice stays the same simple toggle.
When headed wins, when headless wins
Pick thisWhen
Headless code library (Puppeteer / Playwright / Selenium)
A developer owns and operates the scraper, you’re scaling on servers, and the target site isn’t heavily anti-bot
Headless cloud API (Browserless / Zyte / Browserbase)
Same as above, but you don’t want to host the browser yourself
Headless + stealth variant (puppeteer-extra-stealth, undetected-chromedriver, nodriver)
Same as above, but the target site fingerprints aggressively
Headed-by-design platform (Octoparse / ParseHub)
The operator isn’t an engineer; the target site has serious anti-bot defenses; you need to debug visually or intervene on CAPTCHAs and logins
The choice isn’t “easier” or “better” in the abstract — it’s about who the operator is and what the target site does. A developer scraping a public-data site can run plain Puppeteer and skip every other concern on this page. The case for headed grows as the target hardens and the operator moves away from code. For the full enumeration of tools on each side, see The browser runtime landscape.


The browser runtime landscape

Most articles on browser automation talk about one tool at a time. The more useful question is what the space looks like — what kinds of runtimes exist, what each kind is good at, and which axis actually matters when you’re picking one for scraping. The most decisive axis isn’t language or browser engine; it’s whether the runtime runs headless by default or headed by design. Almost the entire ecosystem sits on the headless side. The integrated scraping platforms — with Octoparse the clearest example — sit on the headed side. Knowing which side a runtime is on tells you more about how it will behave on real sites than any spec sheet.
At a glance
ToolInterfaceBrowser engine(s)Default modeTypeStrongest at
Puppeteer
Node API
Chromium
Headless
OSS library
Chrome scraping in JS
Playwright
Node, Python, Java, .NET
Chromium, Firefox, WebKit
Headless
OSS library
Cross-browser, modern code
Selenium
Most languages
Most browsers (WebDriver)
Configurable
OSS library
Widest browser & legacy support
Splash
HTTP API (Lua)
WebKit
Headless
OSS service
JS rendering inside Scrapy
WebdriverIO
Node
WebDriver / CDP
Configurable
OSS library
Test-style scraping in Node
chromedp / Rod
Go
Chromium
Headless
OSS library
Go-native scrapers
Pyppeteer
Python
Chromium
Headless
OSS library
Puppeteer-shaped API in Python
HtmlUnit
Java
Pure-Java browser
Headless
OSS library
JVM scraping without a browser binary
puppeteer-extra-stealth
Node plugin
Chromium
Headless
OSS plugin
Puppeteer + bot evasion
undetected-chromedriver
Python
Chromium via Selenium
Configurable
OSS library
Selenium + bot evasion
nodriver
Python
Chromium via CDP
Headless
OSS library
Modern stealth, no driver binary
Patchright
Node
Chromium
Headless
OSS fork
Playwright + stealth patches
Browserless
HTTP API
Chromium
Headless
Cloud / self-host
Hosted browsers behind an API
Browserbase
HTTP API
Chromium
Headless
Cloud service
Managed browsers for AI agents
Steel.dev
HTTP API
Chromium
Headless
Cloud / OSS
OSS-friendly cloud browsers
Bright Data Scraping Browser
HTTP API
Chromium
Headless
Cloud service
Browser + built-in unblocking
Zyte API
HTTP API
Chromium
Headless
Cloud service
Browser + anti-bot handling
ScrapingBee
HTTP API
Chromium
Headless
Cloud service
Simple “render this URL” API
ScrapingAnt
HTTP API
Chromium
Headless
Cloud service
Budget scraping with proxies
Apify Browser Actors
Apify platform
Chromium, Firefox
Configurable
Cloud platform
Apify-native large-scale scraping
Octoparse
Visual workflow + cloud
Electron Chromium, Chrome for Testing
Headed
Integrated platform
No-code, WYSIWYG selection, headed by design
ParseHub
Visual workflow + cloud
Chromium
Headed
Integrated platform
No-code, similar concept
The bold row is Octoparse — headed by design, with a purpose-built runtime for scraping. ParseHub shares the headed category, but Octoparse is the more widely adopted example.
Open-source automation libraries
This is where most code-based scraping starts. Puppeteer drives Chromium from Node — fast, modern, Chrome-only. Playwright, often described as Puppeteer’s successor, covers Chromium, Firefox, and WebKit across Node, Python, Java, and .NET; for a new project today, it’s usually the better default unless you specifically need Chrome-only. Selenium is the elder statesman — slower and heavier, but it speaks to nearly every browser through the WebDriver protocol, which still matters when a project needs Safari, Edge legacy, or mobile-browser bindings. Outside the big three, the field branches by language and ecosystem. Splash is the JS-rendering service that fits inside Scrapy pipelines, scripted in Lua. WebdriverIO brings a WebDriver/CDP-driven API to Node-heavy projects with a test-runner feel. In Go, chromedp and Rod are the two practical choices, with Rod often preferred for ergonomics. Pyppeteer is the Python port of Puppeteer for teams that want Puppeteer’s shape without leaving Python. HtmlUnit is the outlier — a pure-Java browser implementation, no Chromium binary involved, useful when the JVM ecosystem matters more than JS-engine fidelity. All of these run headless by default. They can run headed, but the friction is real — you need a display (or a virtual one like Xvfb), and most scripts in the wild don’t bother. Their normal posture is invisible.
Stealth and anti-detection variants
When a target site fingerprints the runtime, plain Puppeteer or Selenium gets caught quickly — navigator.webdriver, missing plugins, the headless Chrome user-agent, canvas / WebGL anomalies. The stealth variants patch those leaks. puppeteer-extra-stealth is the most established: a Puppeteer plugin that ships a stack of evasions for the common headless fingerprints. undetected-chromedriver does the same for Selenium-driven Chrome and is the go-to in the Python anti-bot space. nodriver is a newer, driver-less CDP approach from the same author, designed to look like an organic browser session from the network up. Patchright is a Playwright fork with similar stealth patches baked in, for teams already on Playwright. These don’t change the headless/headed posture — they’re still headless by default. They reduce the gap between headless and “real,” but they’re playing defense against a continuously updated detection layer.
Cloud browser APIs
Instead of self-hosting browsers, you call an HTTP endpoint and get a rendered page or a controllable session back. Browserless is the most established — works as managed cloud or self-hosted, drop-in Puppeteer/Playwright endpoint. Browserbase and Steel.dev are newer entrants oriented toward AI agents (Steel is OSS-friendly). Bright Data Scraping Browser and Zyte API bundle browser execution with anti-bot handling and unblocking infrastructure — you pay more, and you get a higher success rate against hard targets. ScrapingBee and ScrapingAnt are simpler “render this URL” APIs aimed at smaller teams. Apify is a platform of its own, with Browser Actors that combine cloud-hosted Chromium with Apify’s queueing and storage. All of these run the browser somewhere on a server with no display attached. Headless is the only mode that makes economic sense in this category — you can’t see what they’re doing, only what they return.
Integrated scraping platforms
This category is structurally different from everything above. Instead of a library you call from code, or an API you POST URLs to, an integrated platform gives you a visual workflow editor with a browser embedded inside it. You build the scraper by clicking on the page, not by writing selectors. Octoparse is the clearest example. It runs two runtimes — a stripped, optimized Electron Chromium for everyday tasks, and Chrome for Testing driven by Puppeteer for sites that need a fully authentic browser. Crucially, both are headed by design: the browser window is visible because the visible page is the editor. ParseHub sits in the same category with a similar approach. Older entries like the Web Scraper.io Chrome extension share the headed lineage too — a browser extension can only operate inside a headed Chrome window. This is the only category where headed is the default rather than a configuration option. That isn’t a limitation — it’s the design choice the workflow depends on.
Headless vs headed: the axis that matters
The headless/headed split tracks who the runtime is for. Headless makes sense when a developer is the operator. You’re writing code, reading logs, scaling out on servers without displays; you don’t need to see the page because you’re describing it programmatically. The whole ecosystem above the integrated-platform line is built on this assumption. Headed makes sense when the page itself is the interface. You’re selecting elements visually, watching a task run, intervening on a login or CAPTCHA, debugging by seeing rather than logging. That’s the Octoparse posture — and it’s also why Octoparse’s stripped Electron runtime exists: headed isn’t necessarily heavy if the underlying browser is purpose-built for scraping rather than general browsing. Two practical consequences fall out of this:
Bot detection. Real headed browsers — visible window, real rendering, real input events — leak fewer of the signals anti-bot services hunt for. Headless tools have to add stealth layers; headed-by-design platforms get this largely for free.
Operator skill. Headless tools assume engineering ownership: someone maintains the script, the proxies, the captcha solver, the deploy. Headed-by-design platforms assume the operator is closer to the data — analyst, ops, growth — and the platform owns the engineering.
For a deeper look at why headed-by-design is a deliberate choice rather than a missing feature, see Headed vs headless browsers.
How to pick
A few decision rules that hold up across most projects:
Writing your own scraper in code, just need to render a page? Playwright is the default. Puppeteer if Chrome-only and you’re already in Node. Selenium only if you need a browser those two don’t support.
Code-based scraper, target site fingerprints aggressively? Move to a stealth variant (puppeteer-extra-stealth, undetected-chromedriver, nodriver) — or skip to a cloud API that bundles anti-bot handling.
Don’t want to host browsers at all? Cloud APIs: Browserless / Browserbase / Steel for plain rendering; Zyte API / Bright Data Scraping Browser for unblocking-included.
Don’t want to write code at all? Integrated platform: Octoparse (or ParseHub). Headed by design, visual selection, runtime bundled with workflow and cloud extraction.
Operator isn’t an engineer, and the target site has anti-bot defenses? This is the strongest case for headed by design — fewer detection signals to leak, and a human can step in when a CAPTCHA appears.
The runtime decision is rarely permanent. Many teams start in a cloud API for one-off rendering, move to a stealth-equipped code library for repeat jobs, and reach for an integrated platform when the operator needs to be someone other than the engineer.


How to handle pagination

Pagination is the navigation layer of a scraper. After the scraper can fetch, render, and extract one page, it still needs to answer a practical question: where is the next batch of records, and how do I know when there are no more? Most pagination failures come from treating every site like a numbered page list. In practice, a catalog might use URL parameters, a next button, infinite scroll, a load-more button, an API offset, or an opaque cursor token. Some sites combine several of these patterns.
Start with the request, not the UI
Before writing pagination logic, open DevTools and watch what changes when you move to the next batch.
Open the Network tab and filter to Fetch/XHR.
Click the next page, scroll down, or press the load-more button.
Inspect the request URL, query parameters, request body, and response.
Decide whether the scraper should follow links, interact with the page, or call an API endpoint directly.
Use the UI as a clue, but trust the network request. A button that says “Load more” might call a simple API with offset=40. A page link might actually hydrate results through JavaScript after the URL changes.
What changesWhat to try first
URL includes page=2, p=2, or /page/2
Loop through numbered URLs
An  link points to the next page
Follow the href until it disappears or becomes disabled
Content appears after scrolling
Find the XHR request; use browser scrolling only if needed
Content appears after clicking a button
Reuse the API request or click the button in a browser session
JSON includes next_cursor, endCursor, has_more, or offset
Paginate through the API response
Numbered pages
Numbered pagination is the simplest case because the next location is visible in the URL:
https://example.com/products?page=1
https://example.com/products?page=2
https://example.com/catalog/page/3

The scraper can increment the page number and stop when the response contains no items, fewer items than expected, or a known 404/empty-state page.
import requests
from bs4 import BeautifulSoup

all_products = []

for page in range(1, 100):
 html = requests.get(f"https://example.com/products?page={page}").text
 soup = BeautifulSoup(html, "html.parser")
 cards = soup.select(".product-card")

 if not cards:
 break

 for card in cards:
 all_products.append(card.select_one(".title").get_text(strip=True))

Watch for page indexes that start at 0, parameter names such as p or start, and sites that return the first page again when the page number is out of range. A repeated first page is worse than an empty page because it can create duplicate data without obvious errors.
Next links
Some sites do not expose page numbers. They only expose a “Next” link or arrow. If the element is a normal anchor, treat pagination as link following:
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup

url = "https://example.com/products"
seen_urls = set()

while url and url not in seen_urls:
 seen_urls.add(url)
 soup = BeautifulSoup(requests.get(url).text, "html.parser")

 for card in soup.select(".product-card"):
 print(card.select_one(".title").get_text(strip=True))

 next_link = soup.select_one("a[rel='next'], a.next")
 url = urljoin(url, next_link["href"]) if next_link and next_link.get("href") else None

The seen_urls guard matters. Misconfigured sites sometimes point the final “Next” link back to the current page or to page one. Also check disabled states such as aria-disabled="true", disabled, or a disabled class before trusting the link.
Infinite scroll
Infinite scroll looks like a browser-only problem, but it usually has an API underneath it. Scroll once with DevTools open and look for a request that fetches the next group of records. The useful parameters are often named offset, page, after, cursor, or limit. When the endpoint is usable, call it directly:
import requests

offset = 0
limit = 24
products = []

while True:
 data = requests.get(
 "https://example.com/api/products",
 params={"offset": offset, "limit": limit},
 ).json()

 batch = data.get("items", [])
 if not batch:
 break

 products.extend(batch)
 offset += len(batch)

Use a browser only when the API is hard to call outside the page because of authentication, signed parameters, or complex client-side state.
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
 browser = p.chromium.launch()
 page = browser.new_page()
 page.goto("https://example.com/products")

 previous_count = 0
 for _ in range(40):
 page.mouse.wheel(0, 4000)
 page.wait_for_timeout(1500)

 current_count = page.locator(".product-card").count()
 if current_count == previous_count:
 break
 previous_count = current_count

 print(page.locator(".product-card").count())
 browser.close()

For infinite scroll, do not rely only on page height. Some layouts keep changing height because of ads, images, or virtualized lists. Item count, network idle, and a maximum scroll count make a safer combination.
Load-more buttons
A load-more button is controlled infinite scroll. The page waits for a click before requesting the next batch. That makes pacing easier because the scraper can wait, validate the new item count, and retry if the request fails. If the button calls a clean API, use that API. If not, click the button in a browser loop:
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
 browser = p.chromium.launch()
 page = browser.new_page()
 page.goto("https://example.com/products")

 while page.locator("button.load-more").is_visible():
 before = page.locator(".product-card").count()
 page.locator("button.load-more").click()
 page.wait_for_function(
 "(count) => document.querySelectorAll('.product-card').length > count",
 before,
 )

 browser.close()

The important check is not just “button clicked”; it is “new records appeared.” Buttons can fail silently, become disabled, or remain visible after the final batch.
Offset and cursor APIs
Modern sites often paginate data at the API layer. Offset pagination asks for a numeric position:
/api/products?offset=40&limit=20

Cursor pagination asks for the next opaque token returned by the previous response:
{
 "items": [],
 "pageInfo": {
 "hasNextPage": true,
 "endCursor": "eyJpZCI6MTAwfQ=="
 }
}

Cursor pagination is more stable when records are added or removed while you scrape. Instead of saying “skip the first 40 rows”, the cursor says “continue after this known position.”
import requests

cursor = None

while True:
 params = {"limit": 50}
 if cursor:
 params["after"] = cursor

 data = requests.get("https://example.com/api/products", params=params).json()
 for item in data.get("items", []):
 print(item["name"])

 page_info = data.get("pageInfo", {})
 if not page_info.get("hasNextPage"):
 break

 cursor = page_info.get("endCursor")

For API pagination, handle rate limits deliberately. Respect Retry-After, retry temporary failures with backoff, and store progress if the job is large enough that restarting from page one would be expensive.
Hybrid pagination
Real sites often combine patterns:
A category has numbered pages, but each page lazy-loads more products after scrolling.
A search page starts with a load-more button, then switches to numbered links.
A tabbed interface has separate pagination for “New”, “Popular”, and “Sale”.
A listing page paginates result URLs, then each detail page has its own paginated reviews or comments.
Handle these as nested loops. Keep the outer loop responsible for the larger navigation unit, and keep each inner loop responsible for one repeated action. Track unique IDs across the whole run so duplicate records do not leak into the output.
Practical safeguards
Define a stop signal. Empty result sets, missing next links, disabled buttons, hasNextPage: false, repeated cursors, and max-iteration limits are all valid stop signals.
Detect duplicates. Infinite scroll and cursor APIs can repeat records when data changes mid-run. Store stable IDs or canonical URLs.
Throttle navigation. Add small randomized waits between batches. Browser automation should wait for content changes, not only fixed timeouts.
Log failures. If one page fails after retries, record the URL or cursor and continue when possible.
Prefer APIs when they are legitimate and stable. Direct API pagination is usually faster and easier to validate than driving a browser.
Use a visual tool when speed matters more than custom code. In Octoparse, pagination can be configured visually for common next-page, load-more, and infinite-scroll flows, then run locally or in the cloud.
Pagination is not just “go to the next page.” It is the scraper’s control loop. Once that loop has clear next-step logic, a reliable stop condition, and duplicate protection, the scraper can move through a site without silently stopping at page one or spinning forever.


Scraping JavaScript-rendered pages

Most modern websites don’t serve their content as static HTML anymore. Frameworks like React, Vue, and Angular build the page dynamically in the browser — the server sends a minimal HTML shell along with JavaScript bundles, and the actual content only appears after that JavaScript executes on the client side. This is why a basic HTTP scraper using something like Python’s requests library often comes back with a nearly empty page: it fetches the raw HTML but never runs the JavaScript that would populate it with the data you actually want. There are three approaches to solving this, and the right one depends on how the site is built. The cheapest approach wins when it applies, so check in order.
Approach 1: Render the page in a real browser
The universal solution is to run a real browser engine that executes JavaScript exactly like a user’s browser does. The page loads, scripts run, API calls fire, content renders — and only then does the scraper extract data from the fully populated DOM. This works against any JS-rendered site, but it costs more: you’re spinning up a browser instance for every page, which uses real memory and CPU. Which browser engine to run is its own choice — different libraries (Puppeteer, Playwright, Selenium), different cloud APIs, different headed-vs-headless trade-offs. The full menu is in The browser runtime landscape, and the headed-vs-headless decision that matters most for scraping is covered in Headed vs headless browsers. Runtime overhead matters at scale, so a purpose-built engine costs significantly less per page than a stock browser.
Approach 2: Intercept the underlying API
JS-rendered pages don’t conjure their content from nothing — they fetch it from backend APIs over XHR or fetch requests. If you can identify those endpoints, you can call them directly and get clean, structured JSON without rendering a browser at all. This is faster, lighter, and more reliable than full-page rendering: no DOM to wait on, no selectors to break when a layout changes, and the data arrives already parsed. The catch is that these APIs can be undocumented, require authentication, or be rate-limited, and they may change without notice. So the workflow is: open the page in a real browser with DevTools’ Network panel open, watch the requests as the content loads, and look for the XHR / fetch calls whose responses contain the data you want. Once you’ve found them, you can often reproduce them outside the browser entirely — sometimes with a single curl command. Octoparse builds this directly into its visual editor. Its built-in browser exposes the same network panel as DevTools, but it lets you select an underlying API response the same way you’d select a DOM element — point and click, and the task uses that endpoint instead of rendering the page. This collapses the typical “open DevTools, find the call, copy headers, rebuild the request” loop into a single visual step. This approach is the strongest answer to JS-rendered pages whenever it’s available — and it’s more often available than people assume. It’s worth checking before reaching for a browser.
Approach 3: Detect server-side rendering
Some sites that use client-side frameworks also implement server-side rendering (SSR) or static site generation for performance and SEO. In those cases, the initial HTML response actually does contain the full content — meaning a lightweight HTTP request may be all you need. View the page source (Cmd+U / Ctrl+U, not “Inspect” which shows the live DOM); if the data is already there in the raw HTML, you can skip the browser overhead entirely. Some sites go further and serve different content to different user-agents — pre-rendering for search-engine crawlers, for example. Setting the request’s user-agent to a known search bot can occasionally unlock the server-rendered version of a page that otherwise requires JavaScript. Use this when it works, with the usual caveats about respecting a site’s terms.
A few practical tips when a real browser is needed
When you do end up rendering in a browser, the failure modes are predictable:
Wait for the right thing, not for “load.” The browser’s load event fires when the HTML and assets are in, but client-rendered content may still be on its way. Wait for the specific selector or text you need to appear, not for the page to “be done.”
Watch for lazy loading. Content that renders only when scrolled into view will not exist until the scraper scrolls. Most browser-automation libraries can simulate scrolling; the trick is knowing you need to.
Client-side routing trips traditional scrapers. In SPAs, navigating from /products to /products/42 may change the URL without firing a new HTTP request. Logic that watches for pageload events misses the transition entirely; wait for content changes instead.
Infinite scroll and “load more” need interaction, not just observation. For deeper treatment of these patterns, see Handle pagination.
Try the cheapest approach first
A reliable strategy is to check in this order:
View source. If the content is in the raw HTML, you’re done — a single HTTP request will do.
Inspect network calls. If the page fetches its content from an API, call that API directly. Faster, lighter, more reliable.
Render in a real browser. When neither of the above applies, run the page. Pick the runtime that fits your operator profile and the target site’s defenses.
The reason this order matters: each step up is more expensive — in latency, infrastructure, and breakage risk. A site that genuinely requires browser rendering costs orders of magnitude more to scrape than one whose API you can call directly, and the cost compounds over thousands of pages. When a workflow ends up needing more than one approach across different page types — some pages SSR, others API-fetched, others fully client-rendered — a platform that lets you switch between strategies in the same task (visual selection on rendered pages, network inspection for APIs, browser-runtime choice when rendering is needed) saves the setup and integration work of stitching three different toolchains together.


XPath & CSS selectors

Selectors are the foundation of any rule-based scraper — they tell the tool exactly which elements on a page to extract data from. A well-written selector keeps working through minor site updates; a brittle one breaks the moment a developer changes a class name or rearranges the layout. Understanding how to write durable selectors is one of the most practical skills in web scraping, and the place to invest time even if you use a visual tool that generates selectors for you. There are two main selector languages: CSS selectors and XPath. Both can locate elements in an HTML document, but they work differently, and each has strengths the other lacks.
CSS selectors
CSS selectors use the same syntax web developers write in stylesheets, which makes them feel intuitive if you have any front-end experience. They select elements based on tag names, classes, IDs, attributes, and their relationships to other elements. For example, div.product-card h2 selects all h2 elements inside divs with the class product-card. CSS selectors are generally shorter, easier to read, and faster for browsers to evaluate. They’re the better default choice for most straightforward extraction tasks.
XPath
XPath is more powerful but more verbose. It treats the HTML document as a tree and lets you navigate in any direction — not just downward from parent to child, but also upward to parents, sideways to siblings, and across the document. Crucially, XPath can select elements based on their text content — //a[contains(text(), "Next Page")] — which CSS selectors cannot do. This makes XPath essential for tasks like finding a button by its label, or locating a table cell based on what it says. XPath also supports conditions, functions, and complex predicates, giving it more expressive power for unusual page structures.
When to pick which
The trade-off comes down to simplicity versus flexibility. CSS handles the majority of common cases more cleanly. XPath is the tool you reach for when the HTML structure is awkward, when you need text-based matching, or when the element you want can only be identified by its relationship to a sibling or ancestor rather than a direct parent.
Writing durable selectors
Regardless of which language you use, durability comes down to a few principles:
Avoid auto-generated class names. Frameworks like React, Vue, and Angular often produce class names that change on every build — .css-1a2b3c and friends are landmines.
Prefer semantic attributes. id, data-*, role, and aria-label are less likely to change because they carry meaning beyond styling. A selector anchored on [data-product-id] survives a stylesheet rewrite; one anchored on .flex-row__inner--lg does not.
Keep selectors short. The longer the chain of parent-child relationships, the more likely some intermediate layout change breaks the whole chain.
Avoid positional selectors. nth-child and absolute XPath paths like /html/body/div[3]/div[2]/ul/li[1] assume the element stays in an exact spot in the DOM — almost never true.
Anchor to the nearest stable landmark. Rather than traversing from the root of the document, find the closest element with a meaningful, stable identifier and select relative to it.
The Shadow DOM challenge
One emerging obstacle for selectors is Shadow DOM, a web component feature that encapsulates a section of the DOM inside a closed boundary. Standard XPath and CSS selectors cannot reach into a shadow root — which means elements inside web components are invisible to traditional scraping approaches. As more sites adopt web components for modular UI, this is becoming a real practical problem: a scraper might see the outer shell of a component but not the content rendered inside it. The fix requires a tool that can pierce shadow roots. Playwright extends standard selectors with its >> syntax for this; Octoparse extends XPath with a custom syntax that does the same — letting a generated selector reach into a shadow root the same way it would address a regular DOM subtree.
How Octoparse approaches selectors
Octoparse primarily uses XPath for element targeting, which is a deliberate choice. XPath’s tree-navigation model maps naturally to the way non-technical users think about page structure — “the price inside this product card” translates more directly into an XPath expression than a CSS selector chain. This keeps things approachable for users who aren’t developers but still need to review or adjust a selector when something changes. More importantly, Octoparse doesn’t just generate any working XPath when a user clicks on an element — it applies an intelligent attribute prioritization algorithm. The system evaluates available attributes by their semantic stability: meaningful identifiers like id, data-*, and role are preferred over volatile class names or fragile positional indices. The auto-generated XPath is designed to survive minor site changes — a reshuffled layout or an updated stylesheet won’t break a selector that anchors on a stable data-product-id attribute rather than a third-div-inside-the-second-section path. This algorithm was rebuilt from the ground up in Octoparse 10.1.0, putting auto-generated selectors in the top tier for accuracy — the generator produces a correct, durable XPath on the first click for the large majority of pages, so users spend less time hand-correcting selectors. For pages built with web components, the same generator produces selectors using Octoparse’s custom XPath extension for Shadow DOM — so an element inside a shadow root is addressable the same way an element in the regular DOM is, without dropping out of XPath to a different selector dialect. On top of the new generator, Octoparse adds two AI capabilities:
AI-assisted XPath generation that evaluates broader page context to produce more resilient selectors
AI-powered self-repair (coming soon) that detects broken selectors and rebuilds them from the changed page structure — covering both pagination (a “Next” control that moved or was renamed) and field selectors (a value that shifted within the layout), so a task that would otherwise stall keeps running
These additions reflect the direction the modern web is heading — pages are becoming more dynamic, more component-based, and more encapsulated. Scraping tools have to keep pace.
The takeaway
Invest time in selector fundamentals even if you use a visual tool. Understanding what makes a selector fragile or durable helps you evaluate auto-generated output, troubleshoot when a task breaks, and build scrapers that hold up over weeks and months — not just the first run. Once your selectors land you the raw values, the next step is cleaning and reshaping that text into the data you actually want.


Refining data with regex

Raw scraped data is rarely clean enough to use directly. Once your selectors have located the values you want, the strings they pull off the page often arrive with surrounding noise. A product price might come with currency symbols, whitespace, and trailing text. A phone number might appear in three different formats across the same site. Dates could be written as “May 15, 2026” in one place and “2026-05-15” in another. Regular expressions — regex — are the standard tool for cleaning, extracting, and reshaping this kind of messy text into consistent, structured fields. Regex works by defining a pattern that describes the shape of the text you want. The engine scans a string, finds matches, and lets you extract or replace them. You don’t need to master every edge case of regex syntax to get real value from it — a handful of common patterns cover the vast majority of scraping cleanup tasks.
Common patterns for cleanup
A small library of reusable patterns handles most real-world cases:
Prices. [\d,.]+ captures numeric values with commas and decimals, stripping currency symbols and surrounding text. Refined to (\d{1,3}(?:,\d{3})*(?:\.\d{2})?), it matches standard formats like 1,299.99 more precisely.
Phone numbers. \(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4} handles common US formats — with or without parentheses, separated by dashes, dots, or spaces.
Emails. [\w.+-]+@[\w-]+\.[\w.]+ catches most standard email addresses — one of the most well-known regex use cases.
Dates. \d{4}-\d{2}-\d{2} matches ISO format; \w+ \d{1,2}, \d{4} handles May 15, 2026 style strings.
HTML tag stripping. <[^>]+> removes tags from a string.
Whitespace normalization. \s+ collapses runs of whitespace into a single space.
Practical tips
Test against real samples. Real pages have quirks idealized examples don’t. Run the pattern against output from the actual target site before trusting it.
Use non-greedy quantifiers (*?, +?). When the engine has a choice between a short and a long match, default * and + take the longest. That’s often not what you want.
Use capture groups. Parentheses () let you extract just the part you care about from a larger match — useful when you need to match context around the data but only keep the data.
Watch locale edge cases. A price regex built for US formatting (1,299.99) will misbehave on European numbers (1.299,99) where the comma and period swap roles. The same applies to dates, phone numbers, and decimal notations.
How Octoparse approaches it
Regex has a well-earned reputation for being difficult to write and harder to read. A complex pattern can look like line noise, and a small mistake can silently match the wrong data or miss valid entries. The cost of getting it wrong isn’t a crash — it’s silently corrupted data you only notice downstream, sometimes much later. Octoparse builds regex support directly into the extraction workflow — users can apply regex transformations to any scraped field as a post-processing step, cleaning and reshaping data before export without writing standalone scripts. The pattern, the field it applies to, and the cleaned output all live inside the same task definition. For users who aren’t comfortable writing regex from scratch, Octoparse also offers AI-assisted regex generation: highlight similar elements on the page, and the AI analyzes the patterns across those examples to automatically generate a matching regex. That pattern can then be tested against the actual scraped data within the platform; you see what it matches, and adjust if needed. The AI handles the syntax; the user validates the result. This addresses the two real costs of regex in a scraping workflow: writing patterns from a blank slate, and verifying they do what you intended on real data. Both are absorbed into the visual editor rather than left as separate engineering tasks.
The takeaway
For most scraping projects, you won’t need deeply complex regex. A small library of reusable patterns for prices, emails, dates, phone numbers, and whitespace cleanup will cover the majority of cleaning tasks. Keep the patterns as simple as they can be while still matching accurately, document them if you plan to maintain the scraper long-term, and lean on AI generation when the syntax gets unwieldy. The goal is clean, consistent output — regex is the means, not the end.


E-commerce data collection

E-commerce data collection turns product pages and marketplaces into structured datasets. Retailers use it to monitor competitors. Brands use it to watch reseller activity and reviews. Market researchers use it to understand category trends. Product teams use it to identify gaps in assortment, content, and customer sentiment. The sources are familiar: Amazon, Walmart, eBay, Shopify stores, brand sites, marketplace seller pages, review pages, and category pages. The engineering challenge is that each source represents the same commercial facts with different page layouts and different anti-bot posture.
What to collect
E-commerce scraping usually starts with a product catalog.
Data typeExample fields
Product identity
Title, brand, ASIN/SKU/GTIN/UPC, model, product URL
Pricing
Current price, list price, discount, coupon, subscription price
Availability
In stock, out of stock, delivery estimate, seller availability
Seller data
Seller name, marketplace seller ID, fulfilled-by signal
Product content
Images, description, feature bullets, specifications
Reviews
Rating, review count, review text, review date, helpful votes
Ranking
Best-seller rank, search position, category rank
Variants
Size, color, pack count, style, region
Templates from Octoparse, Apify, and Bright Data commonly separate listing, detail, and review extraction. That mirrors how e-commerce sites are structured. Listing pages provide breadth; detail pages provide full product facts; review pages provide sentiment and quality signals.
Common workflows
Catalog monitoring
Scrape category pages or search results to discover products, sellers, and rankings. Store product URLs and IDs as refresh targets.
Product detail enrichment
Visit detail pages for discovered products. Collect descriptions, specs, images, variants, seller information, and availability.
Review analysis
Collect reviews separately from product facts. Review pages often paginate independently and may require sorting by newest to support monitoring.
Price and stock tracking
Refresh selected products on a schedule. Store timestamped snapshots so the team can detect price changes, promotions, stockouts, and seller changes.
Platform differences
Platform typeNotes
Large marketplaces
Rich data, heavy anti-bot defenses, many variants and sellers
Brand stores
Cleaner product structure, often Shopify or similar commerce platforms
Long-tail retailers
Less standardization, but lighter defenses
Review-heavy marketplaces
Strong sentiment value, separate review pagination
B2B catalogs
Often require login, quote requests, or region-specific pricing
Amazon is the classic example. Search and category pages expose product cards with title, price, rating, review count, image, and ASIN-like identifiers. Product pages add descriptions, feature bullets, specifications, seller details, variants, best-seller rank, and stock or delivery hints. Review pages add text, rating, reviewer signals, helpful count, and verification status. Treat each page type as a different dataset.
Data normalization
E-commerce data needs cleanup before analysis.
Normalize currency and region.
Convert pack counts into unit price.
Separate product price from shipping.
Standardize availability states.
Map variants to parent products.
Deduplicate identical products across URLs.
Preserve source timestamps.
For cross-site comparison, product matching is the core problem. Use exact identifiers where possible, then fall back to title, brand, model, pack count, size, and image similarity.
Anti-bot and scale
E-commerce sites are among the most protected scrape targets because pricing, reviews, and inventory are commercially sensitive. Expect JavaScript rendering, rate limits, CAPTCHA, IP reputation checks, fingerprinting, and page layout tests. At small scale, careful pacing and a real browser may be enough. At larger scale, use cloud extraction, subtask splitting, proxy rotation, coherent fingerprints, and retry logic. For pages like Amazon, prebuilt scrapers can save time because they already encode page-type handling and field mapping.
Compliance boundaries
Scrape responsibly. Respect robots.txt and site terms, avoid personal or sensitive data unless you have a legitimate basis, and prefer official APIs or partner feeds when they are available and suitable. For marketplaces, seller and reviewer data can raise additional policy and privacy concerns. E-commerce scraping is most valuable when it feeds a defined decision: repricing, assortment planning, review monitoring, reseller compliance, or market research. Start from that decision, then design the fields and cadence around it.


Price monitoring with web scraping

Price monitoring turns web scraping into an operating system for retail decisions. A scraper collects product prices, availability, promotions, shipping terms, seller information, and ratings across competitors or marketplaces. The business then uses that data to adjust pricing, detect MAP violations, monitor stockouts, or understand market movement. The scrape itself is only one piece. The real work is matching equivalent products, collecting at the right frequency, normalizing price signals, and alerting only when something meaningful changed.
Who uses price monitoring
Common users include:
Retailers tracking competitor prices and assortment.
Brands checking marketplace sellers, unauthorized resellers, and MAP compliance.
Marketplaces monitoring sellers, inventory, and category-level price dynamics.
Investment and research teams using price movement as a demand or inflation signal.
Procurement teams watching supplier catalog changes.
For example, a brand might monitor its top 500 SKUs across Amazon, Walmart, eBay, and specialty retailers. A marketplace seller might track one category every hour during a promotion window and daily outside that window.
What to collect
A strong price record should include more than the visible price.
FieldWhy it matters
Product title
Human-readable reference and matching clue
Product URL
Source provenance and refresh target
SKU, ASIN, GTIN, UPC, MPN
Product matching across sites
Current price
Core monitoring value
List price or original price
Discount and promotion analysis
Shipping price
Total landed cost
Availability or stock status
Price is meaningless if unavailable
Seller name
Marketplace and reseller analysis
Rating and review count
Demand and trust context
Variant attributes
Size, color, pack count, region
Collected timestamp
Change history and alerting
Amazon templates from scraping platforms often separate listing-page extraction from detail-page extraction. Listing pages are good for breadth: title, price, rating, review count, image, ASIN, and URL. Detail pages add depth: seller, description, feature bullets, specifications, best-seller rank, variants, stock signals, and reviews. That split is useful for price monitoring too: scan listings frequently, then refresh detail pages for the products that changed.
SKU matching
Product matching is the hardest part of price monitoring. Different sites describe the same item differently. Use exact identifiers when possible:
ASIN for Amazon-specific workflows
UPC, EAN, or GTIN for packaged goods
MPN for manufacturer parts
SKU for your own catalog
When identifiers are missing, combine fuzzy signals:
Normalized title
Brand
Model number
Pack count
Size or volume
Color or variant
Image similarity
Category path
Do not alert on a price difference until the match is reliable. A 2-pack and 6-pack can look similar but represent completely different unit economics.
Scrape cadence
Frequency should match business value and site stability.
ScenarioTypical cadence
High-volume marketplace prices
Hourly or several times per day
Brand MAP monitoring
Daily
Long-tail category research
Weekly
Promotion or holiday campaigns
Higher frequency during event windows
Stock availability checks
Hourly when inventory is volatile
More frequent scraping is not automatically better. It increases cost, block risk, and storage volume. Start with the business decision: if pricing changes are acted on daily, hourly scraping may only create noise.
Change detection
A price monitoring system should distinguish events:
Price dropped below a threshold.
Competitor changed price by more than X percent.
Seller changed on a marketplace listing.
Product went out of stock or came back in stock.
Promotion started or ended.
Review count jumped or rating changed.
Store snapshots instead of overwriting rows. Historical data lets you calculate volatility, average discount depth, stockout duration, and promotion timing.
Common pitfalls
Ignoring shipping. A lower item price with higher shipping may not be cheaper.
Mixing variants. Size, color, pack count, and subscription options can change the price.
Scraping only search results. Listing pages may omit seller, stock, coupon, or variant details.
Over-alerting. Small price movements can drown out meaningful changes.
Not tracking source time. A price without a timestamp cannot support trend analysis.
Template vs custom workflow
Templates are effective when the target is common and the desired fields match the standard output. Octoparse’s Amazon scraper templates, for example, cover listing pages, product details, Prime listings, and reviews; Apify and Bright Data offer similar managed approaches for Amazon and e-commerce sources. These tools reduce the work around pagination, parsing, anti-blocking, and export. Custom workflows are better when you need SKU matching across many retailers, custom alert logic, or downstream integration into pricing engines. A practical architecture is:
Collect source records.
Normalize and match products.
Store timestamped snapshots.
Compare against previous state.
Send only meaningful changes to alerts, BI, or repricing systems.
Price monitoring is successful when the scrape produces a reliable decision, not just a spreadsheet of prices.


Real estate data extraction

Real estate data changes constantly. Listings appear, prices move, homes go pending, rentals disappear, agents update contact information, and neighborhoods shift. Scraping public real estate portals helps analysts, investors, brokers, lenders, and proptech teams turn those changes into structured market intelligence. The goal is not just to collect property pages. It is to build a clean property dataset with enough history to answer questions: what is available, what changed, how fast it changed, and where the market is moving.
Common sources
Real estate scraping usually combines several source types:
SourceTypical data
Listing portals
Active listings, rentals, photos, price, bedrooms, bathrooms, size, description
Sold-history pages
Sale price, sale date, prior listing events
Agent and brokerage pages
Agent name, phone, office, service area, active listings
County or public records
Assessor data, parcel IDs, tax history, ownership records where public
Rental platforms
Asking rent, amenities, availability, lease terms
Neighborhood pages
Schools, commute, demographics, market trend summaries
Octoparse’s real estate template category and Zillow-style templates reflect the common split: search or listing pages discover properties; detail pages extract deeper property facts, descriptions, photos, dates, and agent/contact information.
Field map
Useful property fields include:
Listing URL
Property address
City, state, ZIP/postal code
Latitude and longitude
Listing type: sale, rent, sold, pending
Price or rent
Bedrooms and bathrooms
Square footage
Lot size
Property type
Year built
Days on market
Listing status
Agent and brokerage
Phone number or contact URL
Description
Image URLs
First seen and last seen timestamps
For market analysis, timestamps are as important as fields. A current listing record tells you what is visible now; a history of snapshots tells you price cuts, absorption speed, relisting behavior, and inventory changes.
Freshness and deduplication
Real estate pages are duplicate-heavy. The same property can appear on multiple portals, under slightly different addresses, with different photo sets or agent information. Deduplicate using a combination of:
Normalized address
Coordinates
Parcel ID when available
Listing URL
Agent/brokerage and price
Property attributes such as beds, baths, and square footage
Keep source-specific records even after deduplication. One portal may update status faster; another may preserve a better description or richer photos.
Example workflows
Investor market scan
Collect active listings in target ZIP codes, normalize price per square foot, compare days on market, and flag properties with recent price reductions.
Rental monitoring
Collect apartment or rental listings daily, track asking rents by bedroom count, and detect when a unit disappears or reappears.
Agent prospecting
Scrape public agent pages or listing detail pages to collect agent names, brokerage, listing volume, and service area. Use this for market mapping or B2B outreach, not for collecting private account data.
Valuation inputs
Use recent sold data, active listings, property attributes, and neighborhood signals as inputs to valuation models. Scraped data should be validated against official records where accuracy matters.
Technical challenges
Map search limits. Map interfaces often show only a limited number of pins at one zoom level. Split large geographies into smaller regions.
Dynamic pages. Many portals render listings through JavaScript and APIs.
Status changes. A page can switch from active to pending to sold while the URL stays the same.
Hidden or inconsistent fields. Lot size, HOA fees, taxes, and history may appear only on some listings.
Image-heavy pages. Photos increase bandwidth and storage costs; collect URLs unless you truly need the files.
Legal and operational boundaries
Real estate data can include personal information, ownership signals, and location-sensitive details. Scrape only data you are allowed to access, respect site terms and robots.txt, and be careful with downstream use. Public availability does not automatically make every use appropriate. Official APIs, MLS feeds, brokerage data agreements, or public-record bulk downloads may be better for high-stakes workflows. Scraping is often most useful for market research, monitoring, lead discovery, and supplementing official feeds.
Tools and templates
Apify, Bright Data, and Octoparse all provide marketplace-style options for common real estate or business-location data sources. These templates typically handle pagination, map interaction, browser rendering, retries, and output formatting. Use them when the source and fields match your use case. Build a custom scraper when you need custom geographies, unusual property types, or a multi-source deduplication pipeline. Real estate scraping works best as a monitoring system: collect snapshots, preserve history, deduplicate carefully, and treat every source as one signal rather than the entire truth.


Social media data mining

Social media data is useful because it captures language, attention, complaints, trends, creators, communities, and public reactions in near real time. It is also one of the riskiest categories of web data because platforms have strict terms, user expectations vary, and profiles can contain personal information. Use scraping carefully. Collect only public data you are allowed to use, avoid private or login-gated content, and prefer official APIs when they provide the access you need.
What teams collect
Common social data projects include:
Brand and sentiment monitoring
Creator or influencer discovery
Public review and complaint analysis
Trend detection
Hiring and company research
Community research
Competitive content analysis
The useful fields depend on the platform, but most workflows collect some mix of post text, author metadata, timestamp, engagement counts, media URLs, profile URLs, hashtags, comments, and source links.
Platform map
PlatformTypical public dataCommon use cases
Reddit
Posts, comments, subreddits, scores, timestamps
Community research, sentiment, product feedback
YouTube
Video metadata, comments, channels, views, likes
Creator discovery, review mining, trend tracking
TikTok
Public videos, captions, creator profiles, engagement
Creator research, trend monitoring
X/Twitter
Posts, profiles, repost/like/reply counts
News, sentiment, event monitoring
LinkedIn
Public profiles, company pages, jobs, posts
Recruiting, B2B research, hiring signals
Bright Data’s LinkedIn Scraper API, for example, is organized around profiles, companies, jobs, and posts. Apify’s LinkedIn ecosystem has separate actors for profiles, company data, and jobs. That reflects a general pattern: social platforms are not one dataset. Treat each page type as a separate extraction workflow with separate limits and risks.
Public data vs account data
The most important distinction is access level.
Public data is visible without logging in or by visiting a public URL.
Logged-in public data may be visible only after authentication but still belongs to public pages.
Private or restricted data includes DMs, private groups, non-public profiles, private analytics, or data behind permissions.
Avoid private or permissioned data unless you have explicit authorization. Scraping behind login walls increases legal, ethical, and account-safety risk.
Technical challenges
Social platforms are dynamic and heavily defended.
Infinite scroll and cursor APIs are common.
Posts can be deleted or edited.
Engagement counts change continuously.
Search results are personalized or region-dependent.
Login prompts and rate limits appear quickly.
Anti-bot systems look at IP, fingerprint, behavior, and account trust.
For long-running monitoring, store immutable snapshots. A post that disappears later may still matter analytically, but you need source timestamps and deletion handling.
Data quality
Social data is noisy. Build filters and context into the pipeline:
Language detection
Duplicate and repost detection
Spam or bot-account filtering
Time-window normalization
Hashtag and mention extraction
Author or community context
Engagement rate instead of raw engagement
For sentiment analysis, do not rely only on scraped text. Sarcasm, platform slang, quoted text, and brigading can distort simple models.
Compliance and ethics
Social scraping should be governed more tightly than ordinary product or directory scraping.
Respect platform terms and robots.txt.
Prefer official APIs for regulated or recurring use cases.
Avoid sensitive personal data where possible.
Minimize fields to what the project needs.
Avoid deanonymizing users or combining datasets in harmful ways.
Honor takedown, deletion, and opt-out requirements where applicable.
For research, document collection dates, query terms, sampling limits, and platform constraints. For commercial use, involve legal and privacy review early.
When templates help
Templates and managed scraper APIs help when you need structured output from common page types: LinkedIn jobs, public company pages, YouTube comments, Reddit posts, or public TikTok profiles. They handle pagination, retries, anti-blocking, and field mapping. Custom workflows are better when the research question is narrow, the platform changes often, or the analysis requires careful sampling. In social data mining, the collection method shapes the conclusion. Treat scraping methodology as part of the analysis, not just the plumbing.


Is web scraping legal?

As a general principle, scraping publicly available data that doesn’t involve personal information is broadly accepted in most jurisdictions. The landmark hiQ v. LinkedIn case in the US reinforced the idea that accessing publicly available data doesn’t violate the Computer Fraud and Abuse Act. However, several factors can push a scraping activity into legally risky territory.
Terms of Service
Terms of Service are the first consideration. Many websites explicitly prohibit automated access in their ToS. While violating ToS isn’t necessarily a criminal offense, it can expose you to civil liability, and courts have ruled differently on this depending on the case and jurisdiction.
Copyright
Copyright is another layer. The raw facts on a page (a product price, a public phone number) generally aren’t copyrightable, but the creative expression around them — articles, reviews, original descriptions — may be. Scraping and republishing copyrighted content at scale can create legal exposure.
Data privacy regulations
Data privacy regulations add significant complexity. Under GDPR in Europe and CCPA in California, personal data carries strict handling requirements regardless of whether it’s publicly visible. Scraping email addresses, names, or behavioral data from public profiles can still trigger compliance obligations around consent, storage, and the right to deletion.
Rate and method
Rate and method matter too. Aggressive scraping that degrades a site’s performance could be treated as a form of unauthorized access or even a denial-of-service issue. Respecting robots.txt, throttling request rates, and avoiding circumvention of access controls all reduce legal risk.
How Octoparse supports compliance
When evaluating scraping tools, it’s worth considering how the platform itself addresses these concerns. A standalone script has no built-in guardrails — compliance depends entirely on the developer. A platform like Octoparse bakes several safeguards into the product by default. Data security. All data transmitted between the client and Octoparse servers is encrypted via TLS. Local execution mode lets users run tasks entirely on their own machines so that sensitive or internal data never passes through third-party cloud servers — important for organizations with strict data governance requirements. Cloud extraction results can be deleted by the user at any time, and Octoparse provides automatic data cleanup mechanisms so extracted data does not persist indefinitely on the platform. Responsible collection. Built-in request throttling and rate controls help users avoid overloading target sites, reducing both legal risk and the chance of being blocked. The platform respects robots.txt directives and provides configurable delay settings between requests, making it easier to scrape responsibly without custom engineering. Its global server infrastructure lets users choose where cloud tasks run, which can help with jurisdictional considerations around data residency. Operational accountability. Task logs record what was collected, when, and how — providing an audit trail for internal compliance reviews. Users control exactly where data goes: exports are sent only to destinations the user configures, such as files, spreadsheets, databases, or cloud storage. Enterprise plans support team collaboration with role-based access, so organizations can control who builds tasks, who runs them, and who accesses exported data. Compliance education. Beyond the product, Octoparse maintains educational resources — including this article and the broader Academy — to help users understand the legal landscape and build responsible scraping practices from the start. For users who have specific legal questions about their scraping use case, the Octoparse team also offers consultation to help navigate compliance considerations.
The bottom line
No tool can make scraping legal or illegal on its own — legality is determined by the combination of your data target, your intended use, and the applicable laws. When in doubt, it’s always worth consulting legal counsel, especially when dealing with personal data, copyrighted content, or cross-border collection. The safest general practice is to scrape only public, non-personal data, respect the target site’s stated policies and server capacity, and handle any collected data in compliance with the privacy regulations that apply to your situation.