Why one plan for big data isn't enough

Email LinkedIn
Tools

The term Big Data tends to bring to mind notions of analytics, but there are many other IT functions that require strategies for handling the growing tsunami of information. When developing a roadmap for dealing with big data, more than one approach is needed, suggests Steve Duplessie, founder and senior analyst at the Enterprise Strategy Group.

When data is created, it spurs four "lifecycles": transaction processing, reporting and analytics, backup or disaster recovery and application testing and development, Duplessie writes in a post at InformationWeek. The information is replicated in each of these functions, all within the realm of the transaction processing system. Then it is transformed and replicated again for storage in a warehousing system. Further replications are made for analytics, backup and testing/development.

"It doesn't take long to see how one little transaction record can grow 100-fold. Sooner or later, that growth will break the capabilities of conventional IT," he writes. It no longer is possible to process all the data, so subgroups are chosen with the hope that they are representative of the whole. Only they aren't, Duplessie says. The traditional structured database systems are no longer effective because they weren't designed for the massive volume. The analytics system, storage infrastructure and backup system can't keep up.

Effective strategies for big data have to take into account not just the volume, but the variety, complexity and velocity. Many of the technologies being built to deal with these challenges are "simply Band-Aids," Duplessie cautions. But others ultimately will spur fundamental changes in data management. To get ahead of the curve, it is not a moment too late to begin developing multiple plans for dealing with big data.

For more:
- see Steve Duplessie's post at InformationWeek

Related Articles:
How to prepare for the 'big data' era
The emerging world of non-traditional and synthetic data
When big data matters (and when it doesn't)

Filed Under