This article, let’s clear that at the beginning, is not about the children’s game Duck, Duck, Goose, but an internet search engine whose name is a reference to this famous game.
DuckDuckGo holds a distinct product position in the market in contrast to other search engines like Google and Microsoft Bing, which use an algorithm to provide individualized search results. It emphasizes protecting searchers’ privacy and avoiding the filter bubble via using various APIs of other websites to show quick results rather than showing search results from content farms. According to its official website, this search engine has about 3 billion monthly searches and 6 million monthly downloads.
So, let’s find the simple steps to scrape DuckDuckGo search results and export to Excel files.
Things You Must Know about DuckDuckGo Web Scraping
Is it legal to scrape DuckDuckGo
Most of the time, web scraping is acceptable as long as no private information is taken. However, occasionally it’s possible to break the terms of service of the target website while extracting data from a platform. Breaking the terms of service does not mean it is illegal at all, but the website might be able to take legal action against you for breach of contract. To avoid unpleasant situations like these, we highly recommend you read the terms of service before launching your scraper.
Does DuckDuckGo have APIs
The answer is YES. On its help page, DuckDuckGo states that it encourages users to use its official APIs. Previously, DuckDuckGo provided an Instant Answer API. It’s not a full search results API that contains all the links. However, there are currently no additional API details available on DuckDuckGo. If you’re interested in it and want to learn more, you may need to carefully read the entire help page or get in touch with the developers via the official page.
4 Steps to Scrape DuckDuckGo Search Results
Instead of APIs, we’ll show how you can extract search results from DuckDuckGo with Octoparse. It is an easy-to-use tool for web scraping. Every user can collect full search results from DuckDuckGo with it, regardless of coding skills.
You can download and install Octoparse on your device if this is your first time using it. You must create a free account when you launch the app to log in. After successfully logging in, you can follow the steps below to scrape DuckDuckGo search results!
Step 1: Create a new task
Copy and paste the URL of the search result page into the search bar on Octoparse. Then, click “Start” to create a new task. The target page will be loaded in the built-in browser of Octoparse in seconds.
Step 2: Select the wanted data fields
Once the page has finished loading, click “Auto-detect webpage data” in the Tips panel. By doing this, Octoparse will scan the website and give you some data alternatives. Detected data fields will be highlighted on the page for you to check if Octoparse made a correct “guess”. You can also preview data at the bottom and remove any data fields you don’t want.
Step 3: Create and modify the workflow
After selecting all the data fields you need, click “Create workflow”. A workflow will then appear on the right side of the screen. It presents every step of the scraping process. You can read it from top to bottom, and from inside to outside in nested actions. You can preview each step by clicking on each stage and verifying that it functions properly. It is possible to get no data if the workflow does not work well.
Step 4: Run the task
Click “Run” to start the scraper once you’ve double-checked all the details. There are two ways to run the task. When your project is small or you only require a quick run, you can run it on your local device. While large projects are better suited to being transferred to Octoparse’s cloud servers. Just pick one option, and then Octoparse will take care of the rest. After the task has completed running, you can export data into an Excel, CSV, or JSON file as you want.
Build A DuckDuckGo Crawler with Python
Building a crawler with Python is an excellent alternative API solution to scrape DuckDuckGo data if you are familiar with the language. Here is a sample code as a reference for building your DuckDuckGo crawler.
from requests_html import HTMLSession
session = HTMLSession()
response = session.get('https://duckduckgo.com/?q=fus+ro+dah&kl=us-en')
response.html.render()
for result in response.html.find('.links_deep'):
title = result.find('.js-result-title-link', first=True).text
link = result.find('.result__extras__url', first=True).text
snippet = result.find('.js-result-snippet', first=True).text
icon = f"https:{result.find('img.result__icon__img', first=True).attrs['data-src']}"
print(f'{title}\n{link}\n{snippet}\n{icon}\n')
------------------
'''
Urban Dictionary: Fus ro dah
https://www.urbandictionary.com/define.php?term=Fus ro dah
Fus ro dah. Literally means Force, Balance, and Push. The first dragon
shout you learn in The Elder Scrolls V: Skyrim.
In their tongue he is known as Dovahkiin, Dragonborn, Fus ro dah.
https://external-content.duckduckgo.com/ip3/www.urbandictionary.com.ico
Fus Ro Dah - Instant Sound Effect Button | Myinstants
https://www.myinstants.com/instant/fus-ro-dah/
Instant sound effect button of Fus Ro Dah . Fus Ro Dah. From skyrim. 8,072 users favorited this sound button.
https://external-content.duckduckgo.com/ip3/www.myinstants.com.ico
...Wrap-up
DuckDcukGo is a fast-growing platform. Although it offers some official APIs, there are great alternative solutions. We’ve introduced how to scrape search results with Octoparse or build a crawler with Python. Whether you’re good at coding or not, you can pick a method you love in this article to achieve your goal of web scraping.
