Things You Must Know About Data Harvesting & Data Mining

4 min read

Since the phrase “Big Data” went viral, everything related to data sprang up. Web scrapingweb harvestingweb miningdata analysisdata mining, etc. These words have been used interchangeably to make the realm of data even more confusing for many people. A comprehensive understanding of these terms is necessary for respective businesses to be well-informed in the cutthroat marketing industry. 

What is Data Harvesting?

Data harvesting means getting the data and information from the online resource. It is usually interchangeable with web scraping, web crawling, and data extraction. Collecting is an agricultural term which means to gather ripe crops from the fields which involve the act of collection and relocation. Data harvesting is the process of extracting valuable data out of target websites and putting them into your database in a structured format. 

To conduct data harvesting, you need to have an automated crawler to parse the target websites, capture valuable information, extract the data and finally export it into a structured format for further analysis. Data harvesting, therefore, doesn’t involve algorithms, machine learning, nor statistics. Instead, it relies on computer programming like Python, R, Java, to function. Also, data harvesting is more about being accurate. 

There are many data extraction tools and service providers that can conduct web harvesting for you. Octoparse stands out as the best web scraping tool. Whether you are a first-time self-starter or an experienced programmer, it is the best choice to harvest the data from the internet. 

What is Data Mining?

Data mining is often misunderstood as a process to obtain data. There are substantial differences between collecting the data and mining the data even though both of them involve the act of extraction and obtaining. Data mining is the process of discovering fact-based patterns you generate from a large set of data. Rather than just getting the data and making sense of it, data mining is interdisciplinary, which integrates statistics, computer science, and machine learning.

The famous Cambridge Analytica Scandal collected over 60 million Facebook Users information and isolated out those who were uncertain about their votes based on their identity and activities on Facebook. Cambridge Analytica then employed a “Psychographic Microtargeting” tactic to bombard them with inflammatory messages to shift their votes. It is a typical yet harmful application of data mining. Data mining discovers who they are, what they do, and in return, help to achieve the goal. It sounds like magic, yet complicated.

Data mining has Four Key Applications. The first one is the classification. Just as the word implies, data mining is used to put things or people into different categories for further analysis. For example, the bank builds up a model of classification through applications. They gather millions of applications along with each individual’s bank statements, job titles, marital status, school diploma, etc, then use algorithms to calculate and decide which application is riskier than the others. That said, at the moment you fill out the application form, they already know what category you belong to, and what loan applies to you.


Regression is used to predict the trend based on numerical values from the datasets. It is a statistical analysis of the relationship between variables. For example, you can predict how likely the crime is to occur in a specific area based on historical records. 


Cluster is to group data points based on similar traits or values. For example, Amazon groups similar products together based on each item’s description, tags, and functions for customers to identify easier.

Anomaly detection

This process detects abnormal behaviors which are also called outliers. Banks employ this method to detect unusual transactions that don’t fit into your normal transaction activities. 

Association learning

Association learning answers the question of “how does the value of one feature relate to that of another?” For example, in grocery stores, people who buy soda are more likely to buy Pringles together. Market basket analysis is a popular application of association rules. It helps retailers to identify the relationships of consuming products. 

These four applications build the backbone of Data Mining. So to speak, data mining is the core of Big Data. The process of data mining is also conceived as Knowledge Discovery from Data (KDD). It illuminates the concept of data science, which helps study research and knowledge discovery. Data can be structured or unstructured and scattered over the internet. The real power is when each piece is grouped, set apart between categories so we can draw a pattern, predict the trends and detect abnormalities.

Hot posts

Explore topics

Get web automation tips right into your inbox
Subscribe to get Octoparse monthly newsletters about web scraping solutions, product updates, etc.

Get started with Octoparse today


Related Articles

  • avatarAnsel Barrett
    These are 10 essential skills you would need for a data mining project. Just pick one of them and get started!
    August 19, 2022 · 5 min read
  • avatarAnsel Barrett
    You need the right data extraction tool to collect massive amounts of data from the internet before implementing data comparison software and data analysis software.
    December 22, 2021 · 3 min read
  • avatarAnsel Barrett
    In our previous article “Three kinds of analytical modes to extraction data from websites -- XPath, Data Highlighter and Wrapper”, we are trying to solve two problems.
    November 29, 2017 · 3 min read
  • avatarAbigail Jones
    TED (Technology, Entertainment, Design) is a nonprofit global community that has been committed to the spread of creative ideas in the form of short but powerful talks since 1984. The talks cover all kinds of topics from science to business to global issues in more than 100 languages as TED welcomes people from every discipline and culture. TED site rolls out videos all the time and there are more than 2500 videos now. If you are pressed for time, here are 7 must see TED talks about big data that every data science fans should watch for their insights into the future of big data. These 7 talks cover how we make use of big data from machine learning to healthcare to astronomy. These talks make fascinating watching for anyone curious about where big data might take us.
    October 25, 2017 · 3 min read