Data science has become a burgeoning field, as companies increasingly try to glean competitive value from their data and turn the decision-making process into one that's data-driven. DJ Patil, a data scientist at Greylock Partners who used to work at LinkedIn, Skype, eBay and PayPal, takes a very in-depth (and long) look at the role data scientists play, what they contribute to their organizations and where they belong in those organizations, in a post at O'Reilly Radar.

Online retailers have become masters at finding value in their data, Patil notes. Amazon.com (NASDAQ: AMZN) led the way and changed electronic commerce forever by taking an age-old sales technique for suggesting additional purchases and putting it online. Social networks also rely heavily on data science for their success, helping identify connections between people so that newcomers to a network can locate friends and other contacts. 

"Using sophisticated tracking and analysis technologies, [Facebook has] identified the time and number of connections it takes to get a user to long-term engagement. If you connect with a few friends, or add friends slowly, you won't stick around for long. By studying the activity levels that lead to commitment, they have designed the site to decrease the time it takes for new users to connect with the critical number of friends," Patil writes.

Data scientists at Netflix figured out that when customers add more than a particular number of movies to their queue, they are significantly more likely to become long-term customers. PayPal, American Express and other payment services depend on data science for fraud detection, and their data scientists are locked in a race against fraudsters.

Data science falls into several different domains, including product and marketing analysis, risk and security and business intelligence. Product analytics, which uses algorithms to test the effectiveness of new goods, is a relative newcomer to the field, Patil writes. When it comes to detecting and analyzing online crooks, data scientists have the right skill set. They recognize which data must be collected, know how to analyze it quickly, and use the information to mitigate attacks.

The qualities that make a good data scientist include technical expertise, curiosity, storytelling and cleverness, Patil writes. One of the difficulties in finding these professionals is that there aren't a lot of them around yet, he cautions: "Hiring data scientists was such a challenge at every place I've worked that I've adopted two models for building and training new hires. First, hire people with diverse backgrounds who have histories of playing with data to create something novel. Second, take incredibly bright and creative people right out of college and put them through a very robust internship program."

Patil warns against treating data scientists "like any old product group." Instead, you have to "build cross-disciplinary groups with people who are comfortable creating together, who trust each other, and who are willing to help each other be amazing. It's not easy, but if it were easy, it wouldn't be as much fun."

For more:
- see DJ Patil's post at O'Reilly Radar

Related Articles:
The emerging world of non-traditional and synthetic data
When big data matters (and when it doesn't)
How NOAA handles 80 TB of data a day
Science explores new ways to analyze data