Big Data’s Growing Pains

on June 6th, 2016
career-544952_1280

An article on the Sisense blog asks an interesting question: Do you know how much data your organization will be dealing with in 3 years time? As the article rightly points out, it’s not enough to know the current scope of your data, nor its historical growth rate, since —

The amount of data you have today is only an indication of how much data you will have in the future.

The article goes on to list the ways in which data has been growing, as well as the possibilities that may exist for organizations to use external data sources (e.g. government, weather, or other publicly available datasets) as part of their corporate business intelligence strategy.

In a few years from now, it will be a standard for businesses to think outside the proverbial box and take external data sources into account–simply because the competitive advantages to be had are tremendous.

You can read the full article here.

Indeed, I would tend to agree with this analysis: The fact of the matter is that 10-15 years ago, very few analysts would have accurately predicted the state of the industry as it is today: both in terms the sheer volumes of data we are currently processing and the places where this data is being generated, as well as the things we are actually doing with data — if in those days of yonder the focus was on more accurate reporting and corporate visibility, today the focus has generally shifted to realtime data-driven decision making, predictive analytics, etc.

Accordingly, to understand the amount of data the future has in store for us, it is not enough to look back and assume it will continue growing at the same exponential rate. Technology could provide us with new types of data to measure (just look at all the hype surrounding the IoT, if you dare…), as well as cheaper and faster ways to crunch Big Data, making more of our historical and operational records relevant for analysis. In these types of scenarios we could be looking at much larger amounts of data in a few years time, which begs the question: what would we call this Bigger-than-Big Data? Maybe Humongous Data?

A Disclaimer

Of course, more modest scenarios exist. With all the talk surrounding Big Data, in the real world there are still only very few companies that actually process terabytes of data on a regular basis: so far, most companies are still looking for ways to effectively get a grasp on their ‘small’ data, to visualize it and use it to produce provable value for their business. If this continues to be the case — if Big Data stays mostly a promotional buzzwords used by software vendors to tout their merchandise – then these semi-apocalyptic predictions regarding the growth of data might turn out to be wrong after all.

However, this is definitely not the majority opinion. Hence, organizations looking to stay on top of the ‘data game’ would be wiser to prepare for a future in which their Big Data will only get much, much bigger.