Known as a leading platform for job seekers, employers, and recruiters alike, Glassdoor provides a wealth of information about companies and the labor market. With over 50 million unique monthly visitors and 2.4 million employer clients, it’s a valuable database for market insights. This article will walk you through the process of Glassdoor scraping and utilizing tools like Octoparse for a smooth scraping experience.
What Glassdoor Data is Worth Scraping
When it comes to Glassdoor, there’s a plethora of data available for extraction. A few of the most valuable data points include:
Company reviews and ratings
Company reviews on Glassdoor are generated by current and former employees of these companies. Thus, such information can let you develop trustworthy insights into the company’s work culture.
Salary information
Glassdoor provides detailed information on salary and compensation for various positions within companies. The salary information can help you understand the trends of the labor market and even identify emerging industries.
Interview questions and reviews
Users on Glassdoor have an interest in sharing their interview experiences. You can gain insights into the company’s interview process, the questions asked, and the overall experience of applicants.
Job listings
The job listings on Glassdoor contain detailed information about job openings, including roles, job titles, locations, job descriptions, etc., which is valuable for both job seekers and employers.
Does Glassdoor Have an API
The answer is YES! Glassdoor has an official API but it is now only available to their API partners. If you’re interested in using it, you need to contact them for permission. The Glassdoor API can be used for a variety of purposes.
- Job search and recruitment: You can use the API to retrieve job listings and company information on Glassdoor, then integrate such information into job aggregators;
- Employee engagement and retention: The API can access employee reviews and ratings, which can help you and your company understand job seekers and employees, and eventually identify where to improve;
- Market research: The API can be applied to access company and industry data. With such data, you can develop a deep into the market and analyze the data to identify your competitors;
- Branding and reputation management: Brand images are important for most companies. The API allows businesses to collect company reviews and ratings, then track companies’ reputations and identify areas for improvement.
However, using the Glassdoor API might take a long time to apply for an API key. Writing a script or building a scraper may be more effective for most users.
How to Scrape Glassdoor Data Using Python
Thanks to many free Python libraries, people can now build Glassdoor scrapers with Python at ease. If you’re good at coding, using Python to pull data from Glassdoor can always be a considerable and affordable choice. Taking Glassdoor salary information as an example, below is a sample of code to extract Glassdoor data with a Python library called BeautifulSoup.
import requests
from bs4 import BeautifulSoup
url = 'https://www.glassdoor.com/Salaries/data-scientist-salary-SRCH_KO0,16.htm'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
salaries = soup.find_all('div', class_='salarySnippet')
for salary in salaries:
name = salary.find('h3').text
location = salary.find('span', class_='location').text
salary_range = salary.find('span', class_='value').text
print(f'Job Title: {name}')
print(f'Location: {location}')
print(f'Salary Range: {salary_range}')
print(f'Scraped {len(salaries)} salaries from Glassdoor')Scrape Glassdoor Salary Data with Octoparse
If you’re not good at coding, writing a script to scrape Glassdoor might have a steep learning curve. Well, don’t worry, there is an easy-to-use and no-coding web scraping solution for you. Octoparse is the AI web scraping assistant for anyone regardless of coding skills.
If you don’t have Octoparse on your computer, download Octoparse for free and install it on your device before we begin. When you open the software for the first time, you’ll need an account to log in. You can sign up for a new account or log in directly with your Google or Microsoft account. You can then follow the step-by-step guide below to scrape Glassdoor salary in minutes.
Step 1: Create a new task to scrape Glassdoor salary data
Copy the URL of the page you want to pull salary information from and paste it into the search bar on Octoparse. Next, click the “Start” button to create a task.
Step 2: Auto-detect Glassdoor salary data
After creating a task successfully, the built-in browser will navigate to the target page. Wait for the page to finish loading, then click “Auto-detect webpage data” in the Tips panel. Auto-detection is an advanced feature to scan the page and detect extractable data automatically. Therefore, you don’t need to select the desired data on the page or read HTML files to locate data.
The auto-detection process will take seconds, then it’ll highlight all the detected data in the browser. You can check if your desired data is selected on the page, or check all extractable data in the “Data Preview” panel at the bottom. You can also edit detected data fields and remove unwanted fields or change their names in this panel. For example, we only need job titles and salary information now, so we only keep these two data fields.
Step 3: Create and modify the workflow
Click “Create workflow” after selecting all the wanted data. Then a workflow will show up on your right-hand side. It’s a flow chart that shows how this scraper runs. Before launching the scraper, you can click on each action on the chart to see if it runs as expected.
Step 4: Run the task and export scraped salary data
Click “Run” to launch the scraper after verifying all the details, such as selecting all the needed data fields. Next, select the option to run the task. If you’re working on a small project, running it on your device will be a good choice. But if the project is large and you need up-to-date data from the project, you can run it on Octoparse cloud servers because the cloud platform can work for you around the clock.
When the scraping process is complete, you can export scraped salary data as an Excel, CSV, or JSON file, even to a database like Google Sheets for further use.
Wrap Up
Glassdoor is a gold mine of data. For people who want to gain insight into companies, it allows them to get first-hand information in various aspects. By following this post, we believe you’ve built a Glassdoor salary scraper. Of course, this guide can also be applied to pull data from many other websites. Try Octoparse now and simplify your web scraping process from now on!
