Check The Extraction Rule When Errors Occur

Wednesday, July 20, 2016 10:52 PM

In this article, I will show you how to check the data extraction rule you configured when errors occur.

When you configured a data extraction rule but failed to get the data you want, in this case there must be something wrong with the rule you made.

It always happens especially when the site you planed to crawl is very complicated and the data you want is to huge. 

 

I thought it would be useful to divide this tutorial into three step.

 

Step 1. Click each step one by one in the “Workflow Designer” to check whether the workflow you made is right or not.

             If the order of these steps is wrong, correct the order.

 

( Note: Before you click the next step, the page must be fully loaded in the browser. )

 

Step 2. In a "Loop item" action, sometimes you may need to  click each item in “Loop Item List” to check if each data you want is chosen.

 

(Note: If you find some data is missing, that means you are not locating the right place of the data you want. In this case, you may need to reset the X path)

(In case you are looking to learn more about XPath, please follow the links below. Thanks in advance for your time.)

Video:  Get Started with XPath 1

Video:  Get Started with XPath 2

 

Step 3. When there's nothing wrong with the rule you made, you may need to check out if the page is loaded with Ajax. If it is, set Advanced options like “Wait before execution” or “Ajax Load”.

 

If  the page is not loaded with AJAX but cannot be fully loaded, choose “Wait before execution” option and set up waiting time like “4 seconds” or longer.

 

If it's loaded with AJAX, select "Load with AJAX" and select AJAX timeout like "8 seconds” or longer.

When you extract data from a drop-down menu, you have to select “Load page with AJAX”.

(In case you are looking to learn more about XPath, please follow the links below. Thanks in advance for your time.)

(Video: How to Extract Data from Webpages Loaded with AJAX Example: gumtree.com)

 

 

 Happy Data Hunting!

Contact
us

Leave us a message

Your name*

Your email*

Subject*

Description*

Attachment(s)

Attach file
Attach file
Please enter details of your issue and we will get back to you ASAP.