Did the Experts Get Big Data Predictions Right for 2016?

on October 31st, 2016
Data analytics

Brace yourselves …. predictions are coming. Predictions about big data analytics, trends, products, data breaches, and more will dominate the blogs and news posts over the coming weeks. But before you buy the farm and sell the horses based on what the “experts” predict, let’s see if they nailed it with their predictions this year.

1. Experts Said: Big Data Will Become Mainstream in 2016

Through 2015, there were still holdouts. Reports of companies investing huge sums of money and failing to glean an ROI from the expenditure left many wondering if “big data” was the real deal or not. But the experts nailed it. Depending on which survey you put your faith in, about 60 percent of businesses have at least one big data project in place for production purposes. Though perhaps not “mainstream,” adoption rates are high enough to say experts weren’t wrong on this one.

2. Experts Said: Analytical Tools Will Improve This Year

One of the reasons so many companies sat on the fence about big data is that Hadoop and the Hadoop ecosystem wasn’t yet mature. While there are plenty of improvements yet to be made, Hadoop and the big data ecosystem in general have made significant strides this year, particularly in the areas of data security and real-time analytics.

3. Experts Said: Real-Time Analytics Will Rule

As 2016 dawned, experts foretold that the real potential for Hadoop and big data was in real-time analytics, or data streaming, not in batch processing. Survey says: the experts were right. The massive popularity of products like MongoDB, Apache Storm, Spark, and Kafka prove that real-time analytics is where it’s at. As we head into 2017, real-time data streaming as a service is coming into play, with products available from big data powerhouses like Google, Amazon (AWS), Oracle, and others.

4. Experts Said: The Government Will Start Cracking Down on Data Privacy

Oops! Our experts were doing so well. But it was difficult to predict exactly how campaign politics would derail the government’s ability to do anything this year. As legislators debated everything from the legal definition of marriage to the 2nd Amendment and women’s rights — little progress was made in the realm of legislating data privacy. Maybe that’s a good thing. Perhaps before a new administration buckles itself in and cranks the engine, private industry can get on top of regulating themselves so the long arm of the law doesn’t have to. Hey, anything can happen …

5. Experts Said: The Big Data Talent Crunch Will Continue to Hinder Efforts

This one wasn’t hard to get right. Even before the curtain dropped on 2015, about 40 percent of organizations lacked the talent they needed to get it right with big data. The situation hasn’t improved over the course of 2016, with IDC predicting a shortage of about 181,000 big data specialists by 2019, and five times that if the position requires data management and/or interpretive skills and experience.

However, numerous big data educational programs are ramping up across the nation, with some promising graduates waltzing out with fancy degrees from prestigious programs like those at Southern Methodist University, the University of California-Berkeley, the University of San Francisco, Arizona State University, and others. Within the four years it takes to crank out graduates, the supply should begin catching up with demand. As magazines dub data science as the “sexiest job on earth” and salaries soar well northward of $100,000, it shouldn’t be hard to convince students to sign up and study hard.

Ready to see what next year has to offer? You can track the predictions for 2017 and see how well the experts hit their marks when you follow us on Twitter.