Incremental data scraping

Incremental data scraping helps a recurring task avoid processing the same website inputs again. Octoparse checks the URL used by the task against URLs it has already scraped and skips inputs that match the incremental rule. Incremental scraping is different from duplicate removal:

• Incremental scraping decides whether a URL should be visited again.
• Duplicate removal compares the records produced by the task with existing records.

​

How incremental scraping works

When you enable incremental scraping, Octoparse uses the URL from the task’s extraction step as the comparison key. On a later run, a URL that matches the stored incremental history is treated as already scraped. A new URL can be scraped as a new input. The task should contain one extraction step for incremental scraping. If a workflow has several extraction steps or changes the URL after the comparison point, confirm which URL Octoparse uses before relying on the result.

​

When to use it

Incremental scraping is useful when:

• A URL list is updated regularly with new items.
• A search or detail-page task runs on a schedule.
• You need to avoid revisiting inputs that have already been processed.
• The target site exposes a stable URL or URL parameter for each item.

It is less suitable when the same URL regularly changes its content and you need to refresh that URL on every run. In that case, run the task again and use a downstream date, version, or record key to decide what to keep.

​

Set up an incremental task

​

History and retention limits

Incremental scraping depends on the history that Octoparse still retains. If old run batches or Cloud Data are deleted, Octoparse may no longer remember URLs recorded only in that deleted history. A later run can therefore process those URLs again. Plan retention together with your recurring workflow:

• Keep the history needed to identify previously scraped URLs.
• Do not clear Cloud Data without understanding how it affects the next incremental run.
• If data retention removes older history, use a stable downstream key to detect reprocessed records.
• Treat a reprocessed URL as a possible repeat, not automatically as a task failure.

​

Incremental scraping and duplicate removal

Incremental scraping prevents some repeated inputs from being visited. It does not guarantee that every output record is unique. Octoparse also compares records from a task with its stored data and can identify records whose field values are identical. As a result:

• A task can revisit a URL even when incremental scraping is not enabled.
• A task can scrape a URL but save fewer records after duplicate handling.
• The extracted count and saved count can differ.
• The same URL can produce a new record when its content changes.

Use the URL rule to control which inputs are visited, and use record keys or field-level checks to control what your downstream workflow considers a duplicate.

​

Common cases to check

​

Validate an incremental workflow

Use a small test before enabling a recurring schedule:

1. Run the task with a known input set.
2. Add one new URL or change one input URL.
3. Run the task again.
4. Confirm which inputs were visited and compare the output with the expected new records.
5. Check duplicate count and saved data in Cloud Run Details.

If the result is not as expected, inspect the task’s extraction step, URL parameters, filters, and retained history before changing the workflow. For scheduled tasks, compare the next run with the last known good run rather than assuming that every repeated URL should produce a new record.

​

Related pages