Smart Decision Engine

The Smart Decision Engine (SDE) helps a local extraction choose how it should run. Before the task starts, it recommends an execution browser for that task. If the selected runner later stops unexpectedly, SDE can switch the task to an alternative runner instead of repeating the same failed setup. Introduced in Octoparse 10.2, SDE moves a decision that previously depended on manual trial and error into the task-running process. You can use the recommendation or keep control by selecting a browser yourself.

​

What SDE changes

A task can behave differently across browser environments. When one environment cannot open or run a target page correctly, trying the same environment again may reproduce the same failure. SDE gives Octoparse two opportunities to make a better execution decision: This does not change the task workflow or the fields you configured. It changes the browser environment used to execute that workflow.

​

How the decision works

When you open the local extraction settings, Octoparse requests a recommendation for the current task. The decision considers the target website and the execution environments available on the current device. The result contains a recommended runner and an alternative runner that can be used if recovery is needed. Octoparse can run the task with one of these browser options: SDE only recommends an option that the current environment can use. For example, Local Browser cannot be used as a recovery option when its required browser or profile is unavailable.

​

Auto Select and Manual

In the task start dialog, the Browser setting offers two ways to use the decision: Auto Select is marked Recommended in Octoparse. Choose it when you want Octoparse to handle both the first browser decision and the recovery decision. Choose Manual when a task must use a particular local browser environment or when you want to approve any switch.

​

What happens after an unexpected stop

SDE does not retry indefinitely. For an eligible run, Octoparse makes at most one browser switch:

1. If the task was not using the recommended runner, Octoparse switches to the recommended one.
2. If the task was already using the recommended runner, Octoparse switches to the prepared alternative.
3. If a valid checkpoint exists, Octoparse uses Checkpoint resume to continue from saved progress. If no checkpoint can be used, the task starts again from the beginning.

The single-switch limit prevents the task from cycling repeatedly between browser environments. It also makes the recovery visible and diagnosable: the task either continues with a different runner or stops for you to inspect.

​

Example

Suppose a long local task starts with Chrome for Testing and the runner exits unexpectedly after 3,200 records.

• If Built-in Browser is the prepared alternative, Auto Select can relaunch the task there without waiting for you to choose another browser.
• If the task has a valid checkpoint, it can restore the saved run and work toward the remaining records instead of recollecting all 3,200.
• If you selected the original browser manually, Octoparse asks before making the switch.

The practical gain comes from combining two different recovery mechanisms: SDE changes the environment that failed, while Checkpoint resume preserves completed work when possible.

​

When Octoparse does not switch runners

No automatic switch is made when:

• You stopped the task yourself
• The task completed normally
• The task uses a JSON workflow
• The alternative resolves to the runner already in use
• The alternative requires a Local Browser environment that is not available
• The task has already used its one browser-switch attempt

In these cases, the task remains stopped or completes with its current result rather than entering an unexpected retry loop.

​

What SDE does not do

SDE is a local execution decision and recovery capability. It does not:

• Rewrite your workflow or repair extraction fields
• Switch a local task to Cloud Extraction
• Guarantee access to every website
• Treat empty data, a selector mismatch, or every page error as a runner failure
• Keep changing browsers until a task succeeds

If the workflow itself no longer matches the website, update the task or use Field Fixer to repair affected fields. If the task needs unattended, scalable execution rather than a local browser environment, compare local and Cloud Extraction.

​

Why it matters

SDE is an agentic step in Octoparse’s evolution as an AI web scraper: the product does not only help create a workflow, but can also make a bounded execution decision and act on a runtime failure. Its advantage is not hidden automation for its own sake. It removes a specific manual loop—choose a browser, run the task, diagnose an unexpected stop, choose another browser, and restart—while preserving a manual path and a clear recovery limit.

​

Related pages