> ## Documentation Index
> Fetch the complete documentation index at: https://www.octoparse.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Incremental data scraping

> Use incremental scraping to revisit only new or changed website URLs instead of scraping the same input repeatedly.

Incremental data scraping helps a recurring task avoid processing the same website inputs again. Octoparse checks the URL used by the task against URLs it has already scraped and skips inputs that match the incremental rule.

Incremental scraping is different from duplicate removal:

* **Incremental scraping** decides whether a URL should be visited again.
* **Duplicate removal** compares the records produced by the task with existing records.

## How incremental scraping works

When you enable incremental scraping, Octoparse uses the URL from the task's extraction step as the comparison key. On a later run, a URL that matches the stored incremental history is treated as already scraped. A new URL can be scraped as a new input.

The task should contain one extraction step for incremental scraping. If a workflow has several extraction steps or changes the URL after the comparison point, confirm which URL Octoparse uses before relying on the result.

## When to use it

Incremental scraping is useful when:

* A URL list is updated regularly with new items.
* A search or detail-page task runs on a schedule.
* You need to avoid revisiting inputs that have already been processed.
* The target site exposes a stable URL or URL parameter for each item.

It is less suitable when the same URL regularly changes its content and you need to refresh that URL on every run. In that case, run the task again and use a downstream date, version, or record key to decide what to keep.

## Set up an incremental task

<Steps>
  <Step title="Choose a stable URL key">
    Identify the complete URL or URL parameter that uniquely represents the item you want to scrape.
  </Step>

  <Step title="Keep one clear extraction step">
    Make sure the task uses one extraction step for the inputs that should be tracked incrementally.
  </Step>

  <Step title="Enable incremental scraping">
    Open the task's **Settings**, go to **Run Options**, and enable **Incremental Extraction**.
  </Step>

  <Step title="Test with known and new inputs">
    Run a small sample containing one previously processed URL and one new URL. Confirm that the new input is scraped and the previous input follows the configured incremental rule.
  </Step>

  <Step title="Monitor later runs">
    Compare the run's input and output with the previous run. Check that URL changes, pagination, and filters do not unexpectedly change the comparison key.
  </Step>
</Steps>

## History and retention limits

Incremental scraping depends on the history that Octoparse still retains. If old run batches or Cloud Data are deleted, Octoparse may no longer remember URLs recorded only in that deleted history. A later run can therefore process those URLs again.

Plan retention together with your recurring workflow:

* Keep the history needed to identify previously scraped URLs.
* Do not clear Cloud Data without understanding how it affects the next incremental run.
* If data retention removes older history, use a stable downstream key to detect reprocessed records.
* Treat a reprocessed URL as a possible repeat, not automatically as a task failure.

## Incremental scraping and duplicate removal

Incremental scraping prevents some repeated inputs from being visited. It does not guarantee that every output record is unique. Octoparse also compares records from a task with its stored data and can identify records whose field values are identical.

As a result:

* A task can revisit a URL even when incremental scraping is not enabled.
* A task can scrape a URL but save fewer records after duplicate handling.
* The extracted count and saved count can differ.
* The same URL can produce a new record when its content changes.

Use the URL rule to control which inputs are visited, and use record keys or field-level checks to control what your downstream workflow considers a duplicate.

## Common cases to check

| Situation                              | Why it happens                                                  | What to check                                                                 |
| -------------------------------------- | --------------------------------------------------------------- | ----------------------------------------------------------------------------- |
| A new item is skipped                  | Its URL or selected parameter matches incremental history.      | Confirm the URL key distinguishes the new item.                               |
| An old item is scraped again           | Its previous history was deleted or is no longer retained.      | Check Cloud Data and run-history retention.                                   |
| The same URL needs refreshing          | Incremental scraping treats the URL as already processed.       | Disable incremental scraping for that run or use a separate refresh workflow. |
| More records are saved than expected   | A changed URL parameter creates a new incremental input.        | Review which URL parameters are included in the comparison.                   |
| Fewer records are saved than extracted | Duplicate handling removed records with identical field values. | Compare extracted, saved, and duplicate counts in Cloud Run Details.          |
| Pagination stops producing new records | The pagination URL or parameter is part of the incremental key. | Check the extraction step and the URL value used for comparison.              |

## Validate an incremental workflow

Use a small test before enabling a recurring schedule:

1. Run the task with a known input set.
2. Add one new URL or change one input URL.
3. Run the task again.
4. Confirm which inputs were visited and compare the output with the expected new records.
5. Check duplicate count and saved data in Cloud Run Details.

If the result is not as expected, inspect the task's extraction step, URL parameters, filters, and retained history before changing the workflow. For scheduled tasks, compare the next run with the last known good run rather than assuming that every repeated URL should produce a new record.

## Related pages

<Columns cols={2}>
  <Card title="Cloud extraction" href="/docs/en/platform/cloud-extraction">
    Understand unattended cloud execution and recurring data scraping.
  </Card>

  <Card title="Cloud run status" href="/docs/en/platform/cloud-run-status">
    Check run status, subtasks, data volume, and completed results.
  </Card>

  <Card title="Schedule cloud runs" href="/docs/en/platform/cloud-schedules">
    Run an incremental task at a recurring time or interval.
  </Card>

  <Card title="Monitor cloud runs" href="/docs/en/platform/monitor-cloud-runs">
    Compare data, duplicates, logs, and screenshots across runs.
  </Card>
</Columns>