Big data analytics is widely heralded as the new do-or-die technology for businesses that want to gain or maintain an edge in our Darwinian economy. But bringing all those mountains of data together from myriad internal and external sources is about to become "uncomfortably cozy," warns Daniel W. Rasmus in a fascinating and provocative post at Fast Company.

The problem is not Big Data itself. The problem, in Rasmus's view, is that it can be used inappropriately and "stretched wantonly until its principles fray." Identifying, consolidating and applying effective governance to data makes good sense, but betting the farm on Big Data's ability to predict the future is over-stepping. Rasmus has some harsh words for the $1 billion Big Data project underway at the Swiss Federal Institute of Technology in Zurich.

"Given the woes of Europe, spending €1-billion on such a project will likely prove to be wasted money...Whenever there is an existential problem facing the world, charlatans appear to dazzle the masses with feats of magic and wonder," he writes. "As Big Data becomes the next great savior of business and humanity, we need to remain skeptical of its promises as well as its applications and aspirations."

Rasmus outline several "existential threats" that he sees to Big Data's success, the first being human overconfidence. People do not recognize their limitations and they end up building faulty assumptions into their models, blind to whether a model is good or otherwise.

The next threat is the need to reinterpret models as new knowledge is gained and assumptions are overturned. "If organizations rely on Big Data to connect far-ranging databases," he writes, "who, it must be asked, will understand enough of the model to challenge its underlying assumptions, and re-craft those assumptions when the world, and the data that reflects it, changes?"

Complexity, stemming from nuanced and obscure models, is a threat in and of itself. If an organization maps out a complex Big Data plan, it has to consider how it will ensure continuity over time, as personnel and models change. They also have to build in an effective feedback loop so that people are alerted when models and theories are proven wrong.

Other threats to the success of Big Data include the absence of a theory, biases, hugely disruptive innovations and inappropriate motives.

"Big Data will no doubt be used to target advertising, reduce fraud, fight crime, find tax evaders, collect child support payments, create better health outcomes, and myri[a]d other activities from the mundane to the ridiculous. And along the way, the software companies and those who invested in Big Data will share their stories," Rasmus writes. "The future of Big Data lies not in the stories of anecdotal triumph that report sophisticated, but limited accomplishments--no, the future of Big Data rather lies in the darkness of context change, complexity, and overconfidence."

For more:
- Daniel W. Rasmus's article at Fast Company

Related Articles:
Five ways to dive into Big Data
Why one plan for big data isn't enough
How to prepare for the 'big data' era