When big data collides with the cloud

Tools

Cloud-based services and big data projects are two challenging pursuits in their own right, but in some enterprises they are converging. The use of cloud computing to deal with data analytics is on the rise, reports Bob Violino at Baseline magazine.

Geostellar, a company that helps identify the availability of renewable energy resources, processes petabytes of data in a variety of formats. Early on, it gave its customers the processed data on hard drives, and then it used in-house virtualized servers that were hosted. To more efficiently store, process and transfer this data, the Washington, D.C.-based company started using cloud services about 18 months ago, Violino reports.

"All of the data we're processing has to be centralized in our operations center because the various fields are so large, and it's much more efficient in terms of the proximity of dedicated CPUs and disk drives for reading and writing and processing configurations," said CEO David Levine.

Geostellar runs two infrastructures. The first, which is made up of solid-state hard drives and dedicated servers, is used for processing raw data that comes in from a variety of sources in very large files on hard drives. The second, which is based on a virtualized, cloud-based platform, is used for managing the data after it has been integrated and processed. The processed data is then sent to the cloud where it can be distributed to customers, who can then interact with it directly.

Storing big data in the cloud is not without its challenges and therefore not for every business, Violino reports. Security and intellectual property issues, in some cases, require internal data center solutions instead.

For more:
- see Bob Violino's article at Baseline

Related Articles:
Cloud lessons learned the hard way
Study: Cloud isn't as cheap, easy as expected