Web scraping Covid data: A Youtube Comment AnalysisMonday, January 25, 2021
By scraping Youtube comments on videos giving covid-19 precautions for Thanksgiving Day, we are looking into how people will plan their big holiday gathering under Covid pandemic in 2020. At last, we gather tips by scraping Twitter top posts and hope we all can gather safely and enjoy our Thanksgiving Day as we always did.
Table of Contents
Covid-19 Latest Update in US
According to the CDC, cases in the US increased 764,680 in the last week, with a total amount of 9,913,553, reaching an appalling number of 10 million. Total deaths mounts to 237,037 and death rate now stands at 2.4%. California, Texas and Florida rank top 3 in the number of Covid cases.
Total Covid Case till Nov. 09 in US
Illinois, Texas and Wisconsin are the three states that have witnessed the greatest jump in the case data over the last 7 days.
Covid Case Increase Nov. 03 - Nov. 09 in US
Will covid-19 really disrupt Thanksgiving gathering
As the grand traditional celebration Thanksgiving Day comes at the end of this month, a number of news agencies warn the public on Youtube about mass gathering and give tips on how to celebrate Thanksgiving safely.
We scraped comments on Youtube, under relative videos published by Global news, CBC News, CBS News and GMA, to see what are people’s attitude towards Thanksgiving under Covid-19. Will they switch to a new style of gathering in consideration of this pandemic?
Using Octoparse Advanced Mode to build the crawler fast and quick
We entered the URL of 4 videos top in view to the crawler and scraped a total 198 comments from viewers and exported it into excel for further studies. The convenient part of web scraping is to get data into this well-structured form for future analysis.
Scraping Youtube Comments within minutes
When we looked into these comments, actually, Youtube viewers split into two different parties on this issue:
- The overwhelming majority of them refuse to take Covid-19 as so severe a pandemic that could change the traditional way of family gathering;
- The rest of them, a rather small group of people, stand that Covid-19 is no joke and would be cautious about big gatherings on Thanksgiving.
The comments even reveal the distrust of the public to US press.
As we categorized people’s reactions to these warnings, 10.7% of them agreed on the severity of the pandemic and stated that they would be precautious, while 61.4% of the comments blamed the overreaction or teasing the press for building up this fraud.
This is rather a simplistic example of sentiment analysis. Octoparse is capable of getting data from all visible web pages. With this raw data, you can do an in-depth analysis using more sophisticated models such as NLP, to get to the bottom of things.
Thanksgiving tips for families and friends
In the same way we scraped Twitter to search for tips on how we can safely enjoy our holiday gathering. We took “Thanksgiving covid safety” as the keyword and got hundreds of posts scraped down to Excel.
Twitter data scraped by Octoparse
We had a closer look at the tips posted by these authors and picked out 4 authoritative sources for our readers to consult with. They have offered comprehensive advice on both food safety and for safe holiday entertaining. If you are taking Covid-19 seriously, this would help guide you how to better protect yourselves as well as your friends and families.
Filter out your wanted data in Excel
1 regionofwaterloo: “When you're cooking your Thanksgiving meal, wash your hands with soap and warm water for 15 seconds before handling food or eating.”
2 The New York Times: “The solution? A scaled-back Thanksgiving — with open windows, fewer people and a big serving of precautions.” (Serve Up Some Extra Precautions at Your Thanksgiving Table This Year)
Advice for safe holiday entertaining
3 CDC.gov: “Thanksgiving is a time when many families travel long distances to celebrate together. Travel increases the chance of getting and spreading the virus that causes COVID-19. Staying home is the best way to protect yourself and others. If you must travel, be informed of the risks involved.”
4 Alberta government: “If you are hosting/travelling...”
Difficulties emerge in our life and work. In this respect, what web scraping can do is to gather relevant data from all over the world and get us well informed in an all-round way before making our own decision. Great potential could be realized if we put web scraping well in use in our business.
Author: Cici Edited by: Cici