Skip to main content
The smart decision engine is coming soon. This page describes the planned capabilities. Details may change before release.
The smart decision engine (SDE) is a system-level capability that automatically determines the best way to extract data from a target website. Instead of relying on a single fixed extraction method, SDE evaluates each task and selects the strategy most likely to succeed — with the lowest cost and least risk.

​
Why it matters

Different websites have different levels of complexity, anti-bot protection, and rendering requirements. A single extraction approach cannot cover all of them reliably. SDE solves this by making the strategy selection automatic — you no longer need to manually decide which approach to use.
  • Websites that load fine in a standard browser get a lightweight, fast extraction
  • Websites with stricter anti-bot measures automatically receive a more advanced strategy
  • If one approach fails, the system falls back to an alternative without manual intervention

​
How it works

SDE operates behind the scenes before and during task execution:
1

Analyze the target

SDE evaluates the target website’s characteristics, including its domain, structure, and known behavior from previous runs.
2

Select a strategy

Based on the analysis, SDE automatically chooses the extraction method, browser environment, and related settings that are most likely to succeed.
3

Execute and monitor

The task runs using the selected strategy. SDE monitors the execution for issues such as blocks, empty results, or timeouts.
4

Fall back if needed

If the selected strategy encounters problems, SDE automatically switches to an alternative approach without requiring any action from you.
5

Learn from results

Execution results are fed back into SDE. Over time, the system builds a profile of what works best for each website, making future decisions faster and more accurate.

​
Multi-level extraction strategies

SDE selects from multiple extraction levels depending on the target website’s requirements:
LevelApproachBest for
Level 1Lightweight and fast extractionSimple websites with minimal anti-bot protection
Level 2Standard browser-based extractionWebsites that require JavaScript rendering or moderate interaction
Level 3Advanced extraction with full browser behavior simulationWebsites with strict anti-bot measures or complex access requirements
The system starts with the most efficient level and escalates only when necessary, keeping costs low and speed high.

​
Automatic fallback

If an extraction strategy fails — due to a block, timeout, or incomplete data — SDE automatically retries with a different approach. This means:
  • A single failed attempt does not stop the task
  • The system recovers without manual intervention
  • Historical failures inform future strategy selection, reducing repeat issues

​
Site learning

SDE builds an internal profile for websites over time. The more tasks run against a particular site, the better the system understands which strategies work best. This means:
  • New tasks on familiar websites get optimized strategies from the start
  • Strategy selection improves continuously as more data is collected
  • You benefit from collective learning across runs

​
What you can expect

When SDE launches, you will experience:
  • Higher success rates — tasks are more likely to complete without manual troubleshooting
  • Less manual configuration — strategy selection is handled automatically
  • Better resilience — automatic fallback reduces the impact of website changes or blocks
  • Continuous improvement — extraction performance improves over time as the system learns