undefined

How to Avoid the CookieWall When Scraping the Website in Octoparse

Thursday, April 21, 2016 6:16 AM

For the latest tutorials, visit our new self-service portal. Sharpen your skills and explore new ways to use Octoparse.

 

Cookies are usually small text files, given ID tags that are stored on your computer's browser directory or program data subfolders. Cookies are created when you use your browser to visit a website that uses cookies to keep track of your movements within the site, help you resume where you left off, remember your registered login, theme selection, preferences, and other customization functions. That is, cookies can help a website to arrange content to match your preferred interests more quickly.

Most major websites use cookies. Sometimes cookie messages appear on the screen to inform users that the cookie would be created and remain in the cookie file of your browser when you access certain websites.

If you want to scrape some web pages from a website and the cookie will message would always come first when the web page is loaded in Octoparse, you can configure a rule to remove the cookie wall. Here we would take https://www.chefkoch.de/rezepte/499161144640027/Walnusseis-mit-Ahornsirup.html for instance and solve the problem by the following steps.

 

Steps:

1. Log into Octoparse and create a task with the target website, which is https://www.chefkoch.de/rezepte/499161144640027/Walnusseis-mit-Ahornsirup.html

2. After the web page is loaded, we will notice that a cookie message window has appeared on the screen. 

a typical cookie wall

3. Click Zustimmen and then select Click button from the Tips panel

4. Now Octoparse will close the cookie wall the first time it appears. 

5. If we open any subpage of the website, we would find that the cookie wall was removed, and then you can extract any data on the website.

 

Now you have successfully passed the cookie wall and may continue your task building.

 

Happy Data Hunting!

Author: The Octoparse Team

Download Octoparse Today

 

For more information about Octoparse, please click here.

Sign up today. 

We use cookies to enhance your browsing experience. Read about how we use cookies and how you can control them by clicking cookie settings. If you continue to use this site, you consent to our use of cookies.
Accept decline