Blog > Big Data > Post

Enterprise Data Consolidation and Integration

Thursday, March 23, 2017

Big data is no longer strange to people anymore, it can be integrated with any subjects that you can imagine and has become indispensable in all fields. Data is becoming a core property and has greatly influenced how an enterprise runs its business and re-construct its culture and structure. With that said, it could be tough for a company to succeed without participating in the data revolution.

Data integration and consolidation can help enterprises exchange valuable information and make decisions much more efficiently. Enterprises can utilize data integration and consolidation to design their business system leading to a structural and hierarchical disparity on multiple levels, like company roles, product brand and etc. As we know, the traditional method enterprises normally adopt is to collect data manually, then utilize this data to make plans for products development cycle, make marketing decisions and etc. Even though these enterprises have built up BI,  most data come from CRM, ERP, SCM and other management systems that lack deep insight into data analysis and diversified data fields dimensions. 

Knowing the importance of data, if the data relevant to people's characteristic behaviors can flow freely among different enterprises, it would be beneficial to both ends. Enterprises will be able to utilize this data to better predict users' preferences and consequently recommend the best fitted packages based on related data analysis. However there are still unresolved complications such as  privacy , whether such integration of data source is agreed with their users.


Data from multi-sources should be associated in mapping, otherwise the fragmented data can’t be used for a deep learning or further analysis. So far, ID identifier is the only way to identify users. Traditional enterprises identify users using users’ ID via CRM, hence they will need an ID mapping method if it involves big data consolidation among enterprises. The simplest way to deal with data consolidation on an enterprise level is to use ID card or phone number, since each enterprise will probably maintain these personal information. However, a compulsory restriction to limit the unique identifier is that people may refuse to publicize such private info from the security reasons. In Europe, these info which belongs to the typical PII (Personally Identifiable Information) is officially forbidden. As users decline to the use of ID and other private info, the quality of data may be further sacrificed with phone number likely to change over time. Consequently, the telecom service provider, which generally requires authentic names and ID from their users, and tracks for phone number changes, is standing in a much more advantageous position. This data will all be very helpful for enterprises to resolve issues with separated ID's.


Today, Internet device ID is being used more than any other techniques for enterprises consolidation. This type of ID can be captured via Internet or from APP in the local devices with its nature character spanning over time and space. However, it is not perfect still since it is hard to associate Internet device ID with users’ behaviors. Therefore, we still need a proper type of ID to associate all different kinds of data. Verizon once tried to offer a soft ID, like Cookie. Specifically, they insert a HTTP Header into the AD Exchange pointed by its users’ browsing behaviors, and Hash Code these User PII according to visiting time and location. Using this method, AD Exchange User Data Usage flow can receive these IDs. As soon as users’ behaviors gets authenticated within a specific time range, Verizon can reply to these inquisition. These specific ID's are coded according to specific time and service provider and could be applied to online or real-life situations.


 Author: The Octoparse Team

Download Octoparse Today

For more information about Octoparse, please click here.


Author's Picks

Be the Best Junior Management Consultant: Skills You Need to Succeed

Web Scraping|Scrape Booking Reviews

Web Scraping|Scrape Data from Online Accommodation Booking Sites

5 Steps to Collect Big Data

A Must-Have Web Scraper for Data Comparison Software - Octoparse

10 Best Free Tools for Startups - Octoparse

30 Free Web Scraping Software



30 Free Web Scraping Software

Collect Data from Amazon

Top 30 Free Web Scraping Software

- See more at: http://www.octoparse.com/tutorial/pagination-scrape-data-from-websites-with-query-strings-2/#sthash.gDCJJmOQ.dpuf

We use cookies to enhance your browsing experience. Read about how we use cookies and how you can control them by clicking cookie settings. If you continue to use this site, you consent to our use of cookies.
Accept decline