The big data phenomenon is spurring a shift in information analysis, leading companies to model complete data sets rather than subsets. Analyzing an entire data set may tend to bring forth outlier results--which historically were disregarded--but those outliers are now viewed as providing valuable insight, writes Brett Sheppard in a post at O'Reilly Radar.

The financial services industry is making use of new tools for analyzing massive data volumes in their entirety for tasks like predicting loan default risk and examining marketing data. Bank of America combines a variety of such technologies for modeling credit risk and forecasting loss, writes Sheppard, executive director of Zettaforce. Before using this risk management system, it took 96 hours to forecast the likelihood of load defaults, and now it takes four hours.

Dealing with big data in a holistic way, rather than sampling, is not necessarily easy. To do it successfully, organizations will likely have to rethink their business processes, reconcile data silos and deploy collaboration and visualization technologies, Sheppard advises.   

Sheppard takes a look at a number of tools available for analyzing big data and examples of businesses using them. It isn't uncommon for companies to use a number of technologies in tandem, including relational database management systems, columnar databases and document-oriented databases.

For more:
- see Brett Sheppard's post at O'Reilly Radar

Related Articles:
Advice: Stop worrying about data quality
Science explores new ways to analyze data
Survey: Database technologies are due for a change