Big data can't supplant instinct

Data alone can lead to efficiency, but probably not genius.

Data is the key to making decisions based on fact rather than feeling, but are we losing something by eliminating feeling from the process all together? When human judgment is overruled by data, the way we value knowledge will be transformed and the subject-area expert will lose supremacy as the statistician gains status, write Viktor Mayer-Schonberger and Kenneth Cukier, authors of "Big Data: A Revolution That Will Transform How We Live, Work, and Think".

Human judgment absent of concrete metrics often falls short of the mark, Mayer- Schonberger and Cukier write at Wired. Professional sports learned this when the Oakland As replaced a time-honored method of relying on instinct to value players to a new method relying on math.

When there isn't enough information, experience and intuition (in the form of the expert) are critical. "In such a world, experience plays a critical role, since it is the long accumulation of latent knowledge--knowledge that one can't transmit easily or learn from a book, or perhaps even be consciously aware of--that enables one to make smarter decisions," they write.

But here's the kicker: While big data is sure to help companies become faster, smarter and more efficient, it won't necessarily make them dazzle. That will still depend on humans. "There is an essential role for people, with all our foibles, misperceptions and mistakes, since these traits walk hand in hand with human creativity, instinct, and genius," the authors write. "The same messy mental processes that lead to our occasional humiliation or wrongheadedness also give rise to successes and stumbling upon our greatness."

For more:
see Mayer-Schonberger and Cukier's post at Wired

Related Articles:
At Etsy, data analytics is more than a craft
Nate Silver's big data genius
The path to data-driven decision-making

Filed Under