3 Easy Ways to Scrape Google Search ResultsFriday, September 9, 2022
Do you want to learn how to scrape Google search results and get the data in Excel or CSV format? If you want to monitor your website's ranking on Google, analyze your competitors, or analyze paid ads on Google, then scraping the search results is the best way to get started.
In this article, we are going to learn about 3 different ways that you can use to extract results from Google Search.
Scrape Google Search Results Without Coding
If you don't want to deal with coding, then Octoparse is a perfect tool that you can use to scrape the Google search results. It is an easy-to-use web scraping tool that can help you extract data from websites. Octoparse makes the process of data extraction from the web more accessible and faster. It can automatically scrape data from any page and save it in an organized format without writing any code.
The free version of Octoparse can meet most of your scraping needs, but if you're looking for some advanced features like cloud extraction, scheduled scraping, IP rotation, preset task templates, and some others, then you can ask for a paid version or the data service. In the next part, we'll take the free version as example, you can watch the video or follow the easy steps.
Steps to Scrape Google Search Results with Octoparse
Step 1: Open Google in Octoparse and enter the search keyword
After you install Octoparse on your PC, we will start by entering the Google URL in the Octoparse application and clicking on Start.
After that, we will click on the search bar in Google and click on the "enter text" button in the Tip Box. Then we enter the search keyword that we want to search for on Google.
Then click on the search button on Google and click “Click Button” on the Tip Box. This will click the Search button and we will be sent to the search result page of Google.
Step 2: Select the data that you need to extract
When we get to the search result page, we click on the data that we need to extract. For example, if we want to extract the search result title, click on the first and second title and it will automatically select all the other titles.
Then click on “Extract Text of Selected Element” and the scraper is ready to get all the titles from the search result.
If you need more information on scraping Google search results with Octoparse and you need to extract other data like the URL of the site, title, meta description, and other details, or you want to paginate to the next pages of Google search results, then you can check out Google Search results scraping guide.
Step 3: Run and export scraped Google search results
Click the "Run" button at the top to start scraping process. Select whether you want to run the scraper on your device or on the cloud. Once the run is complete, you can click on the 'Export Data' button and export the scraped Google search result data in different formats like Excel, HTML, CSV, or JSON.
Scraping Google Search Results With Python
If you have some programming experience and are comfortable with coding, then you can use Python with the Beautiful Soup Library to scrape the search results. To use this method to scrape Google search results, you will first need to install the Python language on your computer. If you need help installing Python on your PC, you can check out this tutorial on installing Python.
Once Python is installed, you will need to install two modules, bs4, and requests. bs4(Beautiful Soup) is a python library used to read XML and HTML data from the web. The request is a module that allows you to send HTTP requests to the website. You can install both modules with the following command in your terminal or command prompt.
pip install bs4
pip install requests
Once the bs4 and requests modules are installed, you can use the following code to scrape the results.
- # Import the beautifulsoup and request libraries of python.
- import requests
- import bs4
- # Make two strings with default google search URL
- # 'https://google.com/search?q=' and
- # our customized search keyword.
- # Concatenate them
- text= "web scraping"
- url = 'https://google.com/search?q=' + text
- # Fetch the URL data using requests.get(url),
- # store it in a variable, request_result.
- request_result=requests.get( url )
- # Creating soup from the fetched request
- soup = bs4.BeautifulSoup(request_result.text,
- # soup.find.all( h3 ) to grab
- # all major headings of our search result,
- heading_object=soup.find_all( 'h3' )
- # Iterate through the object
- # and print it as a string.
- for info in heading_object:
You can then change the value of the text variable in the above code to your desired search keyword.
For example, change text=’web scraping’ to text=’octoparse’.
Then running this python script will give you the output of all the results in the search result.
Is There An API for Google Search Results
The third method to scrape search results is to use search result API tools. API tools are software applications that are created to help you scrape the results from search results in real time. They can handle a huge volume of requests, and the responses can be structured in JSON, HTML, or CSV format.
These API tools can handle proxies, solve captchas, and parse all kinds of rich structured data from Google searches. Then you can get data with all the details like title, link, description, date, and the SERP position.
Some of the most popular search result APIs are:
By using the above 3 ways, we can easily scrape the Google search result page. If you don't want to deal with codes, then using Octoparse to scrape the Google search results is the best way to get the search results data in a structured format. If you want to try Octoparse or if you have any queries about setting up the scraper, you can contact Octoparse support team.