Slashdot Mirror
Analyzing the Analysts
Very cool, very deep story at Information Week about the IT "analysis and research" firms you often see quoted as authoritative sources in assorted media, and how accurate their predictions are - or aren't. A quote from the story: "The leading analyst firms
have become so
influential that their
opinions can help IT
chiefs gain
senior-management
approval for technology
investments." Obviously, these firms carry plenty of weight. Should we be scared of their growing power? Or have they become an essential part of the computer and Internet business scene?
It's worse than that. Consider the MIS degree. It used to be that companies wanted technical people. Now they want people to manage technical people. Instead of hiring a good sysadmin, they'd rather hire someone to rent a sysadmin.
I suppose I shouldn't complain -- I wouldn't have my job if it weren't for A Big Company deciding that it is too expensive to hire people with technical skills. (Instead, it started a program with two additional layers of management and various contract negotiations and secretaries and the like.)
I wonder, though, if the desire to save quick cash is going to bite some of these companies on the backside when something goes wrong and no one has the foggiest clue how to fix it. Sucks to fire all of the network guys, eh? Sucks to pay me $2000 to show up, type in three commands, and spend the rest of the day at home, wondering how many people you could have hired at $25 an hour to catch these things before they happen.
The moral of the story? If you're renting brains to tell your company how to run its business, your company is in a heap of trouble. I'd short it.
--
QDMerge 0.4!
how to invest, a novice's guide
Presenting a straight detailed truth and presenting material for mass consumption are two entirely different beasts. CNN is good at what they do -- but they must reach a very wide target market and in so doing must make compromises that may make them seem:
a) stupid
b) sweeping stereotypists
c) not in the know
Remember, they have to sugar coat their material and present it in such a way that it is accepted by everyone.
----------
Laugh. The only analyst material I have ever trusted is PCData; and only because the majority of their information is hard numbers.
----------
I've thought that IT analysis firms were a waste of money since the OS/2 debacle back in the early 90s. Go to your local library and get a PC Magazine from 1990 or 1991 -- you'll see dutiful reporting of the latest analysis firm announcement that OS/2 would supplant DOS in 2 years. Even when it was clear to anyone with a brain that Windows was going to beat the tar out of OS/2, the reports persisted.
Why does this happen over and over again? Well, why do people buy Gartner Group reports? To learn that current trends are going to continue? Of course not; people buy these reports so that they don't miss the big changes, the ones that remake the face of their industry -- and which could put them out of business if they don't keep up with the rate of change.
The problem is, the analysis firms make their judgements about what the Next Big Thing is by looking at the market leaders and asking what they are doing. On its face, this makes some sense; after all, who knows more about a market than the market leader? So the analysts work closely with IBM (in the old days) or MS (today), for example, to divine what's going to happen next in the OS marketplace. And the market leader, rationally enough, says "The Next Big Thing is our evolutionary upgrade product, which will add great features and preserve compatibility." So the analysts take demand for the current product, apply a fancy demand curve to it, and predict that the evolutionary upgrade will see exponential growth in demand over the next 3-5 years, and this is The Next Big Thing.
The problem is, revolutionary change never comes from the market leader! It almost always comes from some small, hungry, unknown (or less known) firm that is less worried about cannibalizing current product lines than it is about building the absolute best product. But these are the very firms that fly under Gartner's radar -- because they're not the market leader yet. So Gartner leans on its old sources, which of course are going to predict that current trends will continue to follow a nice steady upward-sloping demand curve. Why would the market leader anticipate revolutionary change? The market leader will do anything to prevent revolutionary change!
So the analysis firms lean on IBM advice on OSes right when Microsoft is about to eat IBM's lunch, they lean on MS sources right when Linux is about to clean MS's clock, and ten years from now they'll all be palling around with people from Red Hat (or whoever) when The NEXT Next Big Thing roars along and steals the spotlight again. And all the clueless PHBs in the world will scratch their heads and wonder how the high-priced analysts could have gotten things so wrong -- while the high-priced analysts scramble, two or three years behind the folks in the trenches, to get to know the geniuses du jour.
Pretty sad. But if you're in the technology business and you can't take the time to really understand the technology you work with, or to muster up some love for the market you serve, you probably deserve whatever nasty fate befalls you by hiring megacorps to do your thinking for you.
-- Jason A. Lefkowitz
Read my blog.
I think in a proprietary software world, all these IT pundits cheering Windows were probably right from a business sense. Microsoft was giving them just what they needed: a "good enough" solution that a predictable amount of dollars and that drone employees could handle, far better than a "quality" solution by a techie who might ask for a raise!
BUT, Microsoft pushed the aesthetic limits of the techies way too far. The Open Source Code Development Model and related business models based on it are a direct result, I believe, of people's frustrations with Microsoft's inferior products, destruction of standards, "unfair" destruction of competition, etc. The best evidence is Judge Jackson's findings of fact. Open Source is for many of us, including the VCs, SGI, etc., is a revolt against the monopoly.
IT analysts have to be business-minded, offering formularic "solutions" based on case histories. They could not have seen the Open Source revolt coming. The business case history wasn't there. Once open source models become an accepted business formula, the IT analysts will be out there peddling them as well.
If all these info-brokers, and sites like Slashdot etc. continue in their way, I think the IT analysts might find themselves hard pressed for a job in the future as opinion, successful practice, etc., starts to permiate from the web itself. Karma!
"... and most people can usually tell the difference"
:).
Seriously. Have you ever dealt with management in any software or hardware companies? Besides those who were originally cs or engr graduates, I've found very few who actually know their stuff in the companies area of expertise. The good ones will establish good relationships with their "people" as well as sampling the industry using a variety of sources. Do they actually know their stuff? Not usually. They rely on trust and good relationships to keep the gears rolling in the right direction. This is why companies want managment with 5-10 years industry experience. That way discression is used wisely.
Analyst firms have their uses. This is especially true for companies ready to jump on the internet bandwagon. If I was going to, say, rework a companies entire distribution network, I could probably go to an analyst (or two) to point me in the right direction. Of course they will have their biases and buyoffs -- but it's better than blindly testing a bunch of companies, especially if I don't know what prices to expect. Like IBM is going to tell me one of their solutions doesn't exactly fit my business model.
As for market trends though, anyone taking what they say for an absolute is incredibly ignorant. It's like asking an economist how the economy will be in a year. They might be able to provide some insight into what direction your company may take -- but you have to question the motives of someone doing this for money. If an analyst told me to get into such and such industry I might wonder how many other people he or she gave the same advice. I might also wonder where his numbers are that are predicting trends, using numbers that could go either way. It's disconcerting to even think about what's going on in the mind of the group of people running these firms. Could they be receiving kickbacks from companies for shoving clients in their direction? Yes of course these firms live on reputation -- but they aren't going to look out for your best given a number of in company politics and motives (especially if they're backed by certain big businesses). In other words, they aren't directly tied to your success.
So take their advice (if you need it), but tread lightly. Remember, analysts can provide insight, but they can't predict the future (no matter how many industry ties they have)
----------
I looked at the article and then thought for a while about what exactly would this imply for anyone who actually picked good sources in the first place. There are plenty of sources good and bad in this world and most people can usually tell the difference. I could just as likely quote Abraham Lincoln or Adolf Hitler for something about what public policy should be towards our fellow men. As far as technology goes we should look for the people who have the best results and use as many results as possible to get a balanced view. I usually go on the web and get as much information as possible to do a report and then cite all these things in my decision. It shows objectivity and evenhandedness
Slashdot social engineering at it's finest
When you see these blown out claims that NT is advancing on the enterprise, and marketshare figures showing Unix is in decline, it's a fair bet that an analyst has invested heavily in Microsoft stock.
These people have no system of ethics or checks and balances like lawyers or sports figures do, they do not dispute each other, nor is there any way for someone to verify their figures or findings. By and large I think they're mostly in bed with Microsoft as much as Ziff Davis is.
The problem with analysts, is that they are (more often than not) wrong. Even the most reputable of them suffer from this: witness Gartner perdicting the victory of OS/2 of Windows, witness Gartner and everybody else predicting the death of the PC and the rise of NC, witness most of them explaining away the Linux phenomenon behind a wall of corporate (== risk averse) logic. The have turned vision into a bean counting excercise ... the most surprising aspect of their existance however is the weight many a clueless person places on their guesses ...
An other thing that ticks me off about them: they all seem to take MSs promises at face value, even though many things they announced never saw the light of day. Yet it is the Linux people who (according to them) have no roadmap.
Fortunately in this informed age that sort of thing is likely to backfire. It's not like 5 years ago when any schmuck could start an Analyst company and make broad (and completely untrue) statements about any given product and ZD would pick it up and print it. These days, any schmuck can start an Analyst company, make broad and completely untrue statements which ZD then picks up and prints and then someone picks it up, posts in here on /. and 150,000 angry slashdotters flame them in E-Mail for the next two months.
Isn't progress great?
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
Yes and no. Using the threat of coercive power isn't always necessary. Use varies as per managment position (middle and upper management isn't in direct line authority of so called underlings); type of work; hygiene factors; their job market; their pay; their work related knowledge et al.
If you're unable to establish good relationships based on respect and friendliness, company influence, expert knowledge, or power to reward -- then obviously the only alternative left is to exact coercive power and strike fear into your employees by threatening discipline or firings.
The problem is that this is very situational. However, remember that the manager may be just as bad for the job as the employees under him. So if you look at your situation and you see other alternatives -- maybe you should change your management style to do your job well.
----------
A great philosopher of the 1950's once said that too much "analyzation causes paralyzation". Yogi Berra might not have been talking about the IT industry when he uttered those words, as a matter of fact, historians have not been able to discern what he was talking about at all, but I digress; Actually, I have digressed to the point where I have to log out and regather my thougths. Fare thee well.
Hates people who have stupid little sigs
The single best source of information available to managers, the one least frequently used, the closest one to them, is their colleagues, coworkers and employees. Frequently, coworkers have knowledge broad, or deep, or both, on subjects outside their immediate job description. These people may have real-world experience having used the products under consideration or worked on a similar project but no one asked them about it. The problem is that artificial barriers such as competition, fear, ignorance, and elitism prevent the exchange of information.
I've harped on this before but it bears repeating: these organizations provide exactly the required service- making one an instant expert in time for the meeting- but nothing more. The wisdom and experience is acquired only by those who actually trudge across a mountain of documentation, suffer a test implementation, and sift through newgroup and listserv archives.
A few months after the project is underway which individual will have a better understanding of the subject? Without question, the latter will be able to explain their decision and the reasoning behind it. The former will fall back on an empty appeal to what amounts to a marketing report. It's good to see that some of the managers cited in the article demonstrate an understanding of the limits of the analysis. It's more worrisome that so many others, not mentioned, blithely accept it as gospel.
Aside, I'm a titled a Business/Systems Analyst but spend most of my time administering and supporting the projects we build rather than turning out white papers (something I'm required to do distressingly often). Am I biased? Yes, probably. Am I off target? Probably not.
There are a number of reasons why anything coming from a market research company should be taken with a pinch of salt.
Firstly, there is the age-old problem of motivation. These guys are in it for the money, and will say whatever it takes to get them the most sales, sponsorship or consultancy fees. At best, you have to question the validity of research coming from a company subject to these pressures. At worst, people are being deliberately misled in the name of the good old bottom line.
All the good things about academic research are completely lacking with these guys. You don't get anything like the level of rigour and peer review that guarantees the quality of most academic work. Market research organisations in general aren't interested in checking someone else's work because there won't be much profit repeating something that's already been done.
There's also the issue of unequal publicity. Some studies get blown out of all proportion to their intrinsic worth. This can happen for a number of reasons, but usually because some big backer who likes the outcome throws the results everywhere in their publicity material for the next year. Either that or someone mentions it on slashdot. What's worse, the tendency is for more controversial or flawed studies to attract the most attention.
Finally, most of the stuff I've seen from market analysis organisations is unscientific twaddle. They don't seem to know the first thing about performing simple regressions. They commit just about every possible statistical sin. Even if they did do some solid groundwork, this gets cut in any published material because the target audience wouldn't have a clue what it meant. This all adds up to pretty meaningless conclusions.
The only reason I read market research at all is that it can give me a real competitive advantage to know exactly what all the bandwagon bozos are thinking.....
AFAIK, market analysts are full of it.
Just from the top of my head, two instances I remember quite well.
The first one took place about the time Gigabit Ethernet specs were close to completion. The e-mail I saw with my own eyes was from one of the highly respect market research group. I din't save the whole thing, unfortunately. But its subject was something like this: Gigabit Ethernet is a horrible mistake. We have to explain all our clients that ATM is the only correct technology above 100Mbps Ethernet... Or something like this, I don't remember exact wording. I don't think I should comment on this.
Another case took place about a couple of years ago. Once again, I was there. Few market analysts were discussing the fugure of portable computing, especially with respect of integrating mobile phones with PDA. The point they have made was
'The future is Windows/CE as a platform'. Someone
asked them 'And what about a PalmTop?' Their answer was something like 'We did an extensive market research and came to the conclusion that PalmTop has no future, and Windows CE is the only viable platform'. At the very moment I remember thinking 'Suckers! Extensive research, my ass. They've simply decided that 'you won't make any mistake with Microsoft'. What a bunch of idiots'.
Grunt. Oink, oink.
OK, I already knew these people existed & were highly paid & used by corporations. I even used some of the free Gartner Group stuff for uni assignments. So what?
I was really hoping to see a little more evaluation. Or at lease some critical focus. Who does watch these watchers? Do they need watching? Are they misusing their powers for Eeevill? It could have been so interesting, and it wasn't.
I feel let down.
No matter how cynical you become, it's never enough to keep up.
A former boss had a regular subscription to Gartner Group reports. He would always favour Gartner advice over our own.
:-( - he would be telling us how an NT only world was "inevitable", and pointing to Gartner to back it up. (Remember how anti-Linux Gartner used to be 2 years ago?)
This was especially weird, since we were a small New Zealand university, and about as far away from the environment that these reports were written for as one could imagine.
The underlying problem was that the boss (as bosses will) had lost touch with the technology, and was paranoid that his staff were trying to pull one over on him. Rather than taking the high road - building a workforce he could trust - he took the low road of ignoring their advice in favour Gartner's. Result: poor decisions based on advice that was never really relevant to our circumstances.
While we were building cheap effective linux based solutions that actually worked - see, that mistrust had become justified
Analyst reports have their place as preliminary reading before you do your own research, or as a second opinion. If you don't have time to do you own research, and you haven't hired someon whose judgement you can trust, you are not taking things seriously, and deserve whatever you get.
I remember long ago at University learning that there was no subsitute for primary sources. Secondary sources are only ever there to get you up to speed. Treating secondary sources as though they were reliable was always a recipe for disaster.