Analyzing the Analysts

Very cool, very deep story at Information Week about the IT "analysis and research" firms you often see quoted as authoritative sources in assorted media, and how accurate their predictions are - or aren't. A quote from the story: "The leading analyst firms have become so influential that their opinions can help IT chiefs gain senior-management approval for technology investments." Obviously, these firms carry plenty of weight. Should we be scared of their growing power? Or have they become an essential part of the computer and Internet business scene?

Comments

The root of the problem

[jalefkowit]

I've thought that IT analysis firms were a waste of money since the OS/2 debacle back in the early 90s. Go to your local library and get a PC Magazine from 1990 or 1991 -- you'll see dutiful reporting of the latest analysis firm announcement that OS/2 would supplant DOS in 2 years. Even when it was clear to anyone with a brain that Windows was going to beat the tar out of OS/2, the reports persisted.

Why does this happen over and over again? Well, why do people buy Gartner Group reports? To learn that current trends are going to continue? Of course not; people buy these reports so that they don't miss the big changes, the ones that remake the face of their industry -- and which could put them out of business if they don't keep up with the rate of change.

The problem is, the analysis firms make their judgements about what the Next Big Thing is by looking at the market leaders and asking what they are doing. On its face, this makes some sense; after all, who knows more about a market than the market leader? So the analysts work closely with IBM (in the old days) or MS (today), for example, to divine what's going to happen next in the OS marketplace. And the market leader, rationally enough, says "The Next Big Thing is our evolutionary upgrade product, which will add great features and preserve compatibility." So the analysts take demand for the current product, apply a fancy demand curve to it, and predict that the evolutionary upgrade will see exponential growth in demand over the next 3-5 years, and this is The Next Big Thing.

The problem is, revolutionary change never comes from the market leader! It almost always comes from some small, hungry, unknown (or less known) firm that is less worried about cannibalizing current product lines than it is about building the absolute best product. But these are the very firms that fly under Gartner's radar -- because they're not the market leader yet. So Gartner leans on its old sources, which of course are going to predict that current trends will continue to follow a nice steady upward-sloping demand curve. Why would the market leader anticipate revolutionary change? The market leader will do anything to prevent revolutionary change!

So the analysis firms lean on IBM advice on OSes right when Microsoft is about to eat IBM's lunch, they lean on MS sources right when Linux is about to clean MS's clock, and ten years from now they'll all be palling around with people from Red Hat (or whoever) when The NEXT Next Big Thing roars along and steals the spotlight again. And all the clueless PHBs in the world will scratch their heads and wonder how the high-priced analysts could have gotten things so wrong -- while the high-priced analysts scramble, two or three years behind the folks in the trenches, to get to know the geniuses du jour.

Pretty sad. But if you're in the technology business and you can't take the time to really understand the technology you work with, or to muster up some love for the market you serve, you probably deserve whatever nasty fate befalls you by hiring megacorps to do your thinking for you.

-- Jason A. Lefkowitz

Here's The Problem

[mochaone]

A great philosopher of the 1950's once said that too much "analyzation causes paralyzation". Yogi Berra might not have been talking about the IT industry when he uttered those words, as a matter of fact, historians have not been able to discern what he was talking about at all, but I digress; Actually, I have digressed to the point where I have to log out and regather my thougths. Fare thee well.

Most dangerous if management mistrust their staff

[judd]

A former boss had a regular subscription to Gartner Group reports. He would always favour Gartner advice over our own.

This was especially weird, since we were a small New Zealand university, and about as far away from the environment that these reports were written for as one could imagine.

The underlying problem was that the boss (as bosses will) had lost touch with the technology, and was paranoid that his staff were trying to pull one over on him. Rather than taking the high road - building a workforce he could trust - he took the low road of ignoring their advice in favour Gartner's. Result: poor decisions based on advice that was never really relevant to our circumstances.

While we were building cheap effective linux based solutions that actually worked - see, that mistrust had become justified :-( - he would be telling us how an NT only world was "inevitable", and pointing to Gartner to back it up. (Remember how anti-Linux Gartner used to be 2 years ago?)

Analyst reports have their place as preliminary reading before you do your own research, or as a second opinion. If you don't have time to do you own research, and you haven't hired someon whose judgement you can trust, you are not taking things seriously, and deserve whatever you get.

I remember long ago at University learning that there was no subsitute for primary sources. Secondary sources are only ever there to get you up to speed. Treating secondary sources as though they were reliable was always a recipe for disaster.