> ## Documentation Index
> Fetch the complete documentation index at: https://www.octoparse.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Smart decision engine

> Octoparse's smart decision engine automatically selects the best extraction strategy for each task based on site characteristics and historical performance.

<Note>
  The smart decision engine is coming soon. This page describes the planned capabilities. Details may change before release.
</Note>

The smart decision engine (SDE) is a system-level capability that automatically determines the best way to extract data from a target website. Instead of relying on a single fixed extraction method, SDE evaluates each task and selects the strategy most likely to succeed — with the lowest cost and least risk.

## Why it matters

Different websites have different levels of complexity, anti-bot protection, and rendering requirements. A single extraction approach cannot cover all of them reliably. SDE solves this by making the strategy selection automatic — you no longer need to manually decide which approach to use.

* Websites that load fine in a standard browser get a lightweight, fast extraction
* Websites with stricter anti-bot measures automatically receive a more advanced strategy
* If one approach fails, the system falls back to an alternative without manual intervention

## How it works

SDE operates behind the scenes before and during task execution:

<Steps>
  <Step title="Analyze the target">
    SDE evaluates the target website's characteristics, including its domain, structure, and known behavior from previous runs.
  </Step>

  <Step title="Select a strategy">
    Based on the analysis, SDE automatically chooses the extraction method, browser environment, and related settings that are most likely to succeed.
  </Step>

  <Step title="Execute and monitor">
    The task runs using the selected strategy. SDE monitors the execution for issues such as blocks, empty results, or timeouts.
  </Step>

  <Step title="Fall back if needed">
    If the selected strategy encounters problems, SDE automatically switches to an alternative approach without requiring any action from you.
  </Step>

  <Step title="Learn from results">
    Execution results are fed back into SDE. Over time, the system builds a profile of what works best for each website, making future decisions faster and more accurate.
  </Step>
</Steps>

## Multi-level extraction strategies

SDE selects from multiple extraction levels depending on the target website's requirements:

| Level   | Approach                                                  | Best for                                                              |
| ------- | --------------------------------------------------------- | --------------------------------------------------------------------- |
| Level 1 | Lightweight and fast extraction                           | Simple websites with minimal anti-bot protection                      |
| Level 2 | Standard browser-based extraction                         | Websites that require JavaScript rendering or moderate interaction    |
| Level 3 | Advanced extraction with full browser behavior simulation | Websites with strict anti-bot measures or complex access requirements |

The system starts with the most efficient level and escalates only when necessary, keeping costs low and speed high.

## Automatic fallback

If an extraction strategy fails — due to a block, timeout, or incomplete data — SDE automatically retries with a different approach. This means:

* A single failed attempt does not stop the task
* The system recovers without manual intervention
* Historical failures inform future strategy selection, reducing repeat issues

## Site learning

SDE builds an internal profile for websites over time. The more tasks run against a particular site, the better the system understands which strategies work best. This means:

* New tasks on familiar websites get optimized strategies from the start
* Strategy selection improves continuously as more data is collected
* You benefit from collective learning across runs

## What you can expect

When SDE launches, you will experience:

* **Higher success rates** — tasks are more likely to complete without manual troubleshooting
* **Less manual configuration** — strategy selection is handled automatically
* **Better resilience** — automatic fallback reduces the impact of website changes or blocks
* **Continuous improvement** — extraction performance improves over time as the system learns