Slashdot Mirror
Are Linux Reviews Fixed?
David Hume writes: "Following up on a Tucows article asserting Linux Reviews Are Bought Rather Than Earned, ZDNET asserts writers fire off glowing reviews for free software and asks Are Linux Reviews Fixed? Is this a real problem? Are reviewers induced to write good reviews by the implied promise of future free software? If so, what do we do about it? Who do we trust? Do we trust Slashdot? :)" I don't think my family even trusts me. Course the only software I've bought in the last 6 months was Diablo2 (Which I beat thank you ;).
For example, my Helix Gnome review was me downloading the distribution over my 56K modem and having a go at it. I had no contact with anyone at Helix Code.
If they want to see irresponsible journalism, perhaps they should read the stories they just wrote.
These kinds of accusations make me ill. I believe I've just read my last ZDNet article, and I don't think I'll be visiting TUCows anytime soon. I urge everyone to write to the respective editors of these "publications" and calmly voice your disgust.
The more you know, the less you understand.
People get flamed here on /. all the time for making an accusation without backing it up. You have to either acknowledge that you have no proof and are likely full of crap, or provide some evidence or at least some passable logic to back up your claim. I would expect better of a news organization... but then this *is* ZDnet. They've never been known for insightful, ethical, or even interesting journalism. Don't know much about Tucows really, but if the article originated with them, then they deserve the flames as much as ZDnet does for reposting it.
It's not enough to bash in heads, you've got to bash in minds. - Captain Hammer
I believe the best reviews are not what a journalist writes, but the testimony of the users. I rarely trust the word of a single person on the quality of a product, but if I read on a mailing list or news site where some number individuals who use the product say they like it, that carries more weight.
--
For every post, there is an equal and opposite re-post.
I don't think it's bribery, so much as a lack of time and a desire not to scare companies away from the linux market.
Take, for example, WPO2000. Corel has done a lot of work to port lots of their software to Linux, and is one of the companies making it possible for more businesses to start using Linux on a desktop. I could see a well-meaning reviewer wishing to downplay problems found so as not to sully Corel's rep before they have a chance to fix things. Because otherwise, they might just abandon the platform entirely.
Also, a lot of the problems in WPO2k appear only for some people under some circumstances. Unless a reviewer happens to try that feature in the right circumstance, everything might work fine. This is a fault of any review: some things you just don't notice until you've used them in production for a while.
"I think it's great if you are willing to promote Linux to your boss. As long as you are aware of the risk you are taking. The risk of getting fired." - Feb 16, 1998
"Is a Linux takeover likely? Give me a break. Of course not." - June 23, 1998
"I personally think Windows NT will be the mainstream operating system within a few years." [...] "My belief: Linux will never go mainstream" - September 9, 1998
"I've always said that Linux could become a serious challenger to Microsoft's Windows NT." - September 28, 1998
So no, of COURSE linux reviews aren't fixed, and how dare you accuse ZDNet of fixing reviews!
That's like the pot calling the dinner plate black. Ziff-Davis has a long history of getting lots of actual money from MS for advertising, and a long history of generating glowing reveiws of every new MS product...
Steven E. Ehrbar
There's nothing special happening with Linux-related software reviews. Damn near all the reviews in the broad-audience print and web publications are superficial, usually on the positive side.
You need only read something by some of the exceptions like the New York Times's Peter Lewis or Byte's Jerry Pournelle to get a sense of proportion.
In the major computer media from IDG, ZD and CMP, reviews of office suites or $600 graphics tools seldom go beyond a checklist of features and a few comments on interface design. How often does it (StarOffice, MS Office) crash? How often does it (Adobe Illustrator 9) mangle files created with it to the point that they can't be opened?
Large-scale databases and high-end application servers sometimes get a proper review, with a realistic test environment, a significant amount of use, and real-world legacy data. But desktop operating systems and applications--even expensive ones, like design software and development tools--just get a review based on firing it up a couple of times and a look through the feature set.
Sure, that UML tool integrates with PVCS, but does it integrate well in an active team environmet? How's the interface? Does it freeze up when there is network latency? Yes, that desktop database can support tables with millikns of rows, but how did it perform on complex queries? And how did its "multiuser" features fare when you try that same query when three other users are making queries?
Linux-related reviews seem no better or worse. Caldera OpenDesktop installs easily? Great. How well preconfigured was Netscape? Did the PPP dialer setup utility require odd gymnastics like firing up a terminal window and running it as root from a command line? Did you have trouble accessing the update and patch download site? How responsive and knowledgable is the phone and email support staff you're paying for? Were tasks like printing a screenshot or installing new hardware easy or difficult?
If there is a general disconnect between the low polish level of many Linux desktop apps and the high praise Linux gets in the resulting review, that probably has something to do with the remarkable stability of Linux itself compared to the major consumer operating systems. No antialiased fonts? Awkward printer setup? But gee! The darn thing ran for four the weeks of the evaluation without a reboot.
It's tough to get a meaningful review of, say, Photoshop, from someone who is a product reviewer by trade, and not a full-time graphic artist. Even if the reviewer was an artist in a past job, s/he is seldom going to give the application the hard workout they would have in the course of real-world use. Instead, we get charts giving rendering times for specific effects, a database-style test suite applied to a creative tool, which should be more about flexibility, interface design and stability.
Enterprise applications are easier to review, since you can draw meaningful conclusions from a rigid, numerically quantifiable test suite. But even here, reviewers grab onto the superficial to make major pronouncements. So-and-so's web-based server management console is lacking? But what if most users of the application use a command line to manage the product anyway, as with many databases and web and application servers?
If the reviews that are out there are telling me Linux is highly reliable when in fact it isn't, then I could stand to loose a great deal more than just some download and learning time.
If you're basing business decisions on reviews you read in glossy software industry magazines, your business is in trouble from a lot more than just potentially unstable software. Please turn over technology purchasing decisions to someone less naive and go back to chasing your secretary.
The only halfway reliable source of information about software is the experience of other users, and preferably not the user who made the purchasing decision. Before you buy mission-critical software, talk to other professionals you respect who have used it.
Proud member of the Weirdo-American community.
You can see over 600 reviews of mine on my Web site.
Having said all that, back to the question: Are Linux reviews fixed? The answer: It depends on the integrity of the individual journalist and publication. For the most part, very few reviews are fixed in the sense that there's pay for a positive evaluation.
When it does happen, it's usually for one of two reasons. The most common reason is that the media outlet is actually an advertising vehicle, and this fact is given up front. One example is "Bunting's Window", the high-tech products show that appears on most in-flight video programs in the U.S. If you look carefully, there's a statement in there about how "promotional consideration" has been paid by companies whose products appear in the show. That promotional consideration is typically in the 5-6 figures. The producers know that most viewers don't know what "promotional consideration" means -- but hey, at least they said it.
The second way that reviews are "fixed" is in media outlets that are less open about their pay-for-play policies. That's considered unethical almost universally in the U.S., but not overseas: For example, the (now-defunct?) Japanese print version of MacWEEK ran positive reviews face-to-face with a full-page ad for the product reviewed. In the U.S., it's often smaller and more cash-strapped publications that tie advertising and gifts to reviews results. These publications deserve their (generally low) reputations.
O.K., so there's little out-and-out fixing. However, reviews often skim over negative points. Again, there are two common justifications. The first reason -- and one that affects "community" publications, such as those in the Mac and Linux worlds -- is that readers really *want* to hear good things about their community, rather than serious evaluations. That's understandable, but not (IMHO) good: Saplings may need protection, but they also need a chance to grow into storm-weathered oaks.
The more pernicious sort of "fix" is when journalists abandon their evaluative senses because of peer pressure. We all know the big example: Microsoft Windows. [Insert here description of bugs and security holes ignored by the media.] Why do they do this? Because... well, because they're human. They may have friends at the company, or have personal (non-financial) reasons for wanting the product to succeed, or whatever -- the fact is, it happens.
However, *almost never* do journalists give good reviews for financial reasons, or for gifts, or for fear that the company will "dry up" if they slam the product. If it's valuable for a company to have its products appear in a publication, they'll be back with the next version, no matter how bad past reviews have been. And if it's not valuable to have reviews in that publication... well, then, why would companies bother with it in the first place?
--Tom Geller, Geller Communications, http://www.tgeller.com
Tom Geller
> this all came from a single Tucows article
Someone on Linux Today suggested that it was a troll on Tucows to suckerpunch ZD. It would be hard to prove such a thing, but it was kind of funny to see the Tucows writer claim with a straight face that he got offers of free copies of Linux if he would gloss over its faults. I suspect that there is more here than meets the eye, and it will be interesting to see what the author says next.
> a more interesting (and feasible) conspiracy theory is mainstream media sites posting FUD to anger specific groups to get more hits.
AKA trolling. Yeah, that's my theory about why JB alternates between praising Linux and dissing it. The alternations keep both sides coming back for more; if he sang the same tune all the time, one side or the other would learn to ignore him.
Just think... all the trolls here may just be doing their homework for Journalism 101.
--
Sheesh, evil *and* a jerk. -- Jade
This kind of thing really annoys me. This was featured on the main ZDnet page for awhile. There were a lot of "talkbacks" flaming the article for its flawed content.
Altavista, who gets its "tech" news from ZDnet, also posted the article on its frontpage.
First of all, this all came from a single Tucows article which had very little evidence other than the one ancedotal incident with Mandrake. The ZDnet article also mentions Red Hat, but with no links to back up their accusations.
Second, and most obviously as others no doubt realize, Linux is free and the danger of this for Linux reviews is MANY times less than commercial software.
But the thing that bugs me most, is the amount of hits Tucows and ZDnet will get from posting this. As I wrote in their talkbacks, a more interesting (and feasible) conspiracy theory is mainstream media sites posting FUD to anger specific groups to get more hits.
It took me a while to figure out what this article is talking about. When the author says "free software" what he means is _not_ the same free as the FSF. He actually means "getting commercial software for free" and not "free software like Linux." The problem with this whole thesis is that I just can't imagine that it's particularly true of Linux writers, because the amount of money you save by getting a free copy of RedHat or Mandrake is pretty trivial, since you can just download them for free off the net.
However, in the world of non-free software, where "review copies" of software can cost hundreds or thousands of dollars, it gets a lot more tricky. I have had plenty of personal experience with people (myself included!) who want to write reviews of product X in order to get a free copy. And that can definitely influence what you write...
Can your IM do this?
.. I used to work fulltime as a test editor for a major networking magazine, and if I mentioned Linux in a negative sentence or wrote a bad review, I got loads and loads of letters with everything from a big 'Fuck you' to 'You're incompetent - get another job!' and the usual 'How much is Microsoft paying you?!'. This is, by the way, a behaviour that one other group of users share with parts of the Linux communitu, and that is large parts of the OS/2 user base... The first 3-4 times, you can handle this but whenever your email is flooded with mean and aggressive stuff, you tend to get fed up by it and either think twice before writing bad stuff about Linux, or not write about it at all.
-- http://z80.org - all opinions, all the time --
Since Linux itself is free, I couldn't figure out exactly what the reviewers were really getting in return for compromising their journalistic ethics... then it hit me!
The FREE CD!
AOL has been doing this for years, sending me FREE CD after FREE CD!!
I just didn't realize that they were expecting me to give them a "glowing review" in return!
Boy, do I have a lot of writing to do!
I spoke to someone who wrote a review of some compilers for a magazine. I read the review long before speaking with him. He told me that the magazine changed the numbers on the score card. The review in the text did not match the numbers on the score card.
At SD94, Gene Wang was presenting to a very nice breakfast for the press people. One woman refered to a press badge as a beer badge.
This type of thing have gone on for quite a while. That is sometimes why you see a review that looks like a rehash of a press release and wonder, if the reviewer actually used it.
Fight Spammers!
ZDNet has no room to talk about biased reviews.
Exhibit A: Review of CyberSitter in PC Magazine a while back. Nothing but glowing praise.
Exhibit B: Recent reviews on the ZDNet site of various graphics programs. The hideously overpriced Photoshop received high marks, while the similarly-featured and much, much cheaper Paint Shop Pro 6 received a scathing review (from ZDNet staff, anyway; the staff gave it 2 out of 5, while readers gave it 4.5. What's wrong with this picture?)
Talk about the pot and the kettle. ZDNet appears to bias its reviews toward whichever company is buying the most ad space.
This is a Chao. A Chao says "Mu."
It isn't like you really need to trust reviews...because most of the time you spend reading reviews your looking for the best investment of your money...and in this case, the software is free, so the only investment is the downloading and learning of the software. And yes, that can be a HUGE investment, but it you would still need to invest that time into commercial software anyway.
Actually downloading and even learning the software is often a tiny proportion of the investment you'll put into software. If I am going to (say) run my business on a Linux computer, then I could be risking everything I have on the reliability of the system. If the reviews that are out there are telling me Linux is highly reliable when in fact it isn't, then I could stand to loose a great deal more than just some download and learning time.
Of course, I'm not saying Linux actually is unreliable, just that the impact of biased reviews can be huge, even for free software.
Sailing over the event horizon
First, I've been writing reviews since 1984, when I purchased my first Compaq computer -- the sewing-machine version that couldn't take a hard drive. Yes, the lure to do reviews was "free" software...but I quickly moved to hardware reviews where you don't get to keep the product at all.
Reviewing platforms, or reviewing software cross-platform, is some of the roughest reviewing possible. Think about reviewing two different operas on two different stages and you get the idea.
The problem with all the reviews I've read that attempt to compare the performance of Linux and NT is that the assumptions behind the test methodology are geared toward either the Linux or NT model. That means you are trying to compare apples and oranges. The two are completely different beasts. Complicating the problem is that much of the performance testing methodology developed for the computing industry is centered around Unix -- very centered around Unix.
The most unbiased performance test suites around are the SPECmark series. I learned the hard way just how Unix-centric the SPECmark series of tests are when I tried to port them to the Macintosh OS -- indeed, I never finished the job. Indeed, I can't even see how to port the SPECmarks to the Windows environment because of the large number of Unix-isms built into the benchmarks. The reason? The benchmarks are actual live real working applications, designed to do a job and not just fiddle bits to eat up resources.
While I agree that the most unbiased reviews come from users, all the vast majority of them can tell you is that "Hey, it worked [didn't work] for me for what I do!" The vast majority of users don't have a clue how to do a structured evaluation of software or hardware...even slash-dot readers. That's why I was able to make a comfortable living for about ten years, writing reviews.
The only review methodology that might make sense is to develop a task, and have two teams configure systems to perform that task. Even then, you will run into variances because the teams may have differing knowledge levels of the systems they are trying to tune for the task. This is the big problem in SpecWeb marks, judging from the reports I've read lately on their sites.
Will there be a fair review? Right now, I think the issue is in doubt.
To the subject of "pay for play" -- in the fifteen years I have been reviewing stuff, I have been offered a number of bribes. None of the companies trying to bribe me ever met my price; hell, they never came close! Other companies have threatened me with lawsuits for what I wrote. None have gone to court, and all but one was settled out of court in my favor. (That one, I admitted that I did the review wrong, and the magazine and I came up with a fix that satisfied everyone.)
Of course I'll eat their food, and of course I'll listen to the PR flacks. That doesn't mean that I'll write a review based on what flacks tell me.
As a founding member of the Internet Press Guild (www.netpress.org) I subscribe to a canon of ethics that require me to write what I experience with products, not what someone tells me to day. That includes editors -- there has been more than one article I've pulled because an editor disagreed with my findings. IT'S MY NAME. Stephen Satchell Satchell Evaluations