Big Data: Don’t Be a Hoarder

The buzzword floating around companies and their IT departments today is Big Data. The predominant thinking is that there is value in ‘data’ and companies should store as much data as possible. Accordingly, many firms now save almost all their data for data mining purposes. Frankly, this approach is at the least unnecessary and at the most overkill.

According to a survey by the Compliance, Governance and Oversight Council (CGOC), a forum of about 2,300 professionals from business and government agencies, organizations should keep data in the percentages shown below.

DATA RETENTION NEEDS OF BUSINESSES
Data Type	Percentage of All Data
Legal Hold	2-3%
Regulatory Requirements	5-10%
Business Analysis and Insights	25%

Dane Atkinson, CEO of SumAll, a New York City data analytics company, is only interested in data that has immediate value. Atkinson feels companies too often waste a great deal of money warehousing all their data even if they have no idea what insights it might provide. These companies, according to Atkinson, do not know today what inquiries to make of the stored data and hope that in the months or even years to come they will figure it out or somehow have it revealed to them.

Atkinson says: "That's the theory. That's exactly it: 'We don't know smart questions to ask now, so we're going to keep it all so that we can ask them later’.” He also said, "It costs a lot of money. It costs us millions of dollars a year to store our customers' data."

Nevertheless, companies - especially Internet-based businesses, try to keep every byte of data that comes into their possession. These companies may find no value in this data today but believe the data has future value. Companies are becoming data hoarders.

Atkinson suggests that companies cut the data they save by deciding what answers they want from it and store the indexes they need. He claims this will cut data storage needs significantly, from terabytes to only gigabytes.  The data will cost less to store and will be far easier to manage and query.

He believes that currently only the smartest companies have started to cut the amount of data they store. In fact, they may still have all their data secreted away, but they have at least built a new store of data that is more efficient and capable of answering immediate questions. 

One easy way to mine data for insights is to look at expense reports. Mining expense data for insights can highlight trends on travel spend, the cost of customer acquisition, as well as divisions that are running leaner. Be sure to read the new whitepaper, Will the Lack of Big Data Expense Analytics Kill Your Business.