In a piercing--yet often quite funny in its sarcasm--column dissecting a recent study on the talents and skills of CIOs, InformationWeek's Bob Evans reveals that sometimes there's a lot less to research than meets the eye. In a scathing analysis, he shows that in this study, the researchers' methods were weak, their conclusions suspect and their recommendations self-serving. 

The study in question was conducted by three researchers at Santa Clara University, and their conclusions were published in a column in the Wall Street Journal under the headline, "Why CIOs are Last among Equals." The researchers determined that CIOs know little about business, interpersonal skills, strategy or leadership.

Evans' smackdown begins with a look at the methods employed in the study. While the researchers said they evaluated 600 people, only 30 of them were CIOs. In other words, the researchers attack an entire profession based on an examination of 30 individuals, or about 5 percent of the study's total subjects. As sampling goes, that is weak, to say the least. What's more, the evaluation was conducted over the course of 11 years. As anyone in the business knows, the CIO of 1999, in many cases, was much different from the CIO of 2010.

"The good news is, the 'research' and all its pointy-headed findings have all the credibility of Tiger Woods because they are based on assessments carried out across 11 years with about 30 CIOs," Evans writes.  "[T]hese three 'experts' on the CIO profession engaged with, on average, one CIO every 134 days spread out across more than a decade."

(Even a cursory glance at Evans' own engagement with CIOs reveals that he is much better positioned to evaluate their strengths and weaknesses than these researchers. Disclosure: I've never met or worked with Evans and I have no connection with his employer.)

It seems to be quite fashionable these days to find fault with CIOs, and the researchers appear to have seized upon the trend. It's a shame they managed to get their ideas into the Wall Street Journal, which may further popularize the erroneous stereotype that CIOs are not positioning themselves as business leaders. 

It's perhaps not surprising that the researchers conclude with a recommendation for CIOs to get some additional education. Hmmmm...I wonder where they could get some of that?

For more:
- see Bob Evans' column at InformationWeek

Related Articles:
Survey: CFOs want to be in charge of CIOs
C-suite should focus on leadership, trust and long-term value
What CFOs want in a CIO
Are you thinking like a CFO?
Get your CFO to buy into IT projects