Get the data you need
Youâve tried everything else, and you havenât managed to get your hands on the data you want. Youâve found the data on the web, but download option is not available and copy-paste has failed you. Fear not, there may still be a way to get the data out. For example you can:
1. Get data from web-based APIs, such as interfaces provided by online databases and many modern web applications (including Twitter, Facebook and many others). This is a fantastic way to access government or commercial data, as well as data from social media sites.Â
2. Extract data from PDFs. This is very difficult, as PDF is a language for printers and does not retain much information on the structure of the data that is displayed within a document. Extracting information from PDFs is beyond the scope of this book, but there are some tools and tutorials that may help you do it.
3. Screen scrape websites. During screen scraping, youâre extracting structured content from a normal web page with the help of a scraping utility or by writing a small piece of code. While this method is very powerful and can be used in many places, it requires a bit of understanding about how the web works.
With all those great technical options, donât forget the simple options: often it is worth to spend some time searching for a file with machine-readable data or to call the institution which is holding the data you want. Everyone has done this: you go to a website, see an interesting table and try to copy it over to Excel so you can add some numbers up or store it for later. Yet this often does not really work, or the information you want is spread across a large number of web sites. Copying by hand can quickly become very tedious, so it makes sense to use a bit of code to do it.
Web Scraping Advantages
The advantage of scraping is that you can do it with virtually any web siteâââfrom weather forecasts to government spending, even if that site does not have an API for raw data access.
Web scrapers are usually small pieces of code written in a programming language such as Python, Ruby or PHP. Choosing the right language is largely a question of which community you have access to: if there is someone in your newsroom or city already working with one of these languages, then it makes sense to adopt the same language.
While some of the click-and-point scraping tools mentioned before may be helpful to get started, the real complexity involved in scraping a web site is in addressing the right pages and the right elements within these pages to extract the desired information. These tasks arenât about programming, but understanding the structure of the web site and database.
When displaying a website, your browser will almost always make use of two technologies: HTTP is a way for it to communicate with the server and to request specific resource, such as documents, images or videos. HTML is the language in which web sites are composed.
Any HTML page is structured as a hierarchy of boxes (which are defined by HTML âtagsâ). A large box will contain many smaller onesâââfor example a table that has many smaller divisions: rows and cells. There are many types of tags that perform different functionsâââsome produce boxes, others tables, images or links. Tags can also have additional properties (e.g. they can be unique identifiers) and can belong to groups called âclassesâ, which makes it possible to target and capture individual elements within a document. Selecting the appropriate elements this way and extracting their content is the key to writing a scraper.
