Slashdot Mirror
Are Linux Reviews Fixed?
David Hume writes: "Following up on a Tucows article asserting Linux Reviews Are Bought Rather Than Earned, ZDNET asserts writers fire off glowing reviews for free software and asks Are Linux Reviews Fixed? Is this a real problem? Are reviewers induced to write good reviews by the implied promise of future free software? If so, what do we do about it? Who do we trust? Do we trust Slashdot? :)" I don't think my family even trusts me. Course the only software I've bought in the last 6 months was Diablo2 (Which I beat thank you ;).
For example, my Helix Gnome review was me downloading the distribution over my 56K modem and having a go at it. I had no contact with anyone at Helix Code.
If they want to see irresponsible journalism, perhaps they should read the stories they just wrote.
These kinds of accusations make me ill. I believe I've just read my last ZDNet article, and I don't think I'll be visiting TUCows anytime soon. I urge everyone to write to the respective editors of these "publications" and calmly voice your disgust.
The more you know, the less you understand.
People get flamed here on /. all the time for making an accusation without backing it up. You have to either acknowledge that you have no proof and are likely full of crap, or provide some evidence or at least some passable logic to back up your claim. I would expect better of a news organization... but then this *is* ZDnet. They've never been known for insightful, ethical, or even interesting journalism. Don't know much about Tucows really, but if the article originated with them, then they deserve the flames as much as ZDnet does for reposting it.
It's not enough to bash in heads, you've got to bash in minds. - Captain Hammer
I believe the best reviews are not what a journalist writes, but the testimony of the users. I rarely trust the word of a single person on the quality of a product, but if I read on a mailing list or news site where some number individuals who use the product say they like it, that carries more weight.
--
For every post, there is an equal and opposite re-post.
The best, most foolproof way I have found to find sites that are whoring their reviews is to just look for the ones that gave Daikatana a good review.
:)
Those I don't patronize anymore
--
I think there is a world market for maybe five personal web logs.
I don't think it's bribery, so much as a lack of time and a desire not to scare companies away from the linux market.
Take, for example, WPO2000. Corel has done a lot of work to port lots of their software to Linux, and is one of the companies making it possible for more businesses to start using Linux on a desktop. I could see a well-meaning reviewer wishing to downplay problems found so as not to sully Corel's rep before they have a chance to fix things. Because otherwise, they might just abandon the platform entirely.
Also, a lot of the problems in WPO2k appear only for some people under some circumstances. Unless a reviewer happens to try that feature in the right circumstance, everything might work fine. This is a fault of any review: some things you just don't notice until you've used them in production for a while.
Actually, I don't know if this is FUD or not since I'm not sure I even understand what is or isn't... but this article is just plain dumb. Let me get this straight (the article seemed awefully short, so maybe I missed something)... Mandrake, RedHat, and SuSE are being "charged" with giving reviewers FREE software in exchange for better reviews. Huh. Those bastards.
Let's ignore for a second that you can DOWNLOAD all of the software for free. (Somehow I doubt that RedHat sends a Win2000 Server CD with every reviewer's package).
This is something that has gone on for a LONG time in every industry...the SOFTWARE industry isn't the first... Car's...there is a SHINNING example where bribes buy the awards. The companies that hand out car awards get free vacations and trips and other things like that... Of course these aren't for good reviews...its just because the reviewers are "friends" with the car company. uh huh, yeah that's it.
Further more... I think the author of this article might be mis interpreting what is going on. Perhaps reviewers are reluctant to say "oh and by the way, there is this minor issue with an applet you probably won't use" because they say in their review "Mandrake is available for free." Furthermore, it has been my experience that the major distributions are generally pretty well put together (upon release). If reviewers were ignoring HUGE problems, then I can see where this article would have some ground. (Huge problems being something like, the utility to partition the drives won't work with IDE disk drives, or installing Gnome means your mouse doesn't work).
As far as I know, the same kind of reviews happen in the Windows and Macintosh world... So why pick on Free software? Sounds to me like someone needed an article topic.......
---
Again, courtesy of LWN - the 1999 linux timeline.. quotes:
"Betting $5 on a 100-to-1 underdog can be fun. Betting $50,000 would be foolish. Yet some PC users are making similarly outrageous wagers on Linux, the underdog in the operating-system wars." - Jesse Berst, March 02, 1999
That is the last Berst article. There is no need to moderate this post up, unless you happen to *really* dislike Jesse. Then, by all means. :)
"I think it's great if you are willing to promote Linux to your boss. As long as you are aware of the risk you are taking. The risk of getting fired." - Feb 16, 1998
"Is a Linux takeover likely? Give me a break. Of course not." - June 23, 1998
"I personally think Windows NT will be the mainstream operating system within a few years." [...] "My belief: Linux will never go mainstream" - September 9, 1998
"I've always said that Linux could become a serious challenger to Microsoft's Windows NT." - September 28, 1998
So no, of COURSE linux reviews aren't fixed, and how dare you accuse ZDNet of fixing reviews!
That's like the pot calling the dinner plate black. Ziff-Davis has a long history of getting lots of actual money from MS for advertising, and a long history of generating glowing reveiws of every new MS product...
Steven E. Ehrbar
There's nothing special happening with Linux-related software reviews. Damn near all the reviews in the broad-audience print and web publications are superficial, usually on the positive side.
You need only read something by some of the exceptions like the New York Times's Peter Lewis or Byte's Jerry Pournelle to get a sense of proportion.
In the major computer media from IDG, ZD and CMP, reviews of office suites or $600 graphics tools seldom go beyond a checklist of features and a few comments on interface design. How often does it (StarOffice, MS Office) crash? How often does it (Adobe Illustrator 9) mangle files created with it to the point that they can't be opened?
Large-scale databases and high-end application servers sometimes get a proper review, with a realistic test environment, a significant amount of use, and real-world legacy data. But desktop operating systems and applications--even expensive ones, like design software and development tools--just get a review based on firing it up a couple of times and a look through the feature set.
Sure, that UML tool integrates with PVCS, but does it integrate well in an active team environmet? How's the interface? Does it freeze up when there is network latency? Yes, that desktop database can support tables with millikns of rows, but how did it perform on complex queries? And how did its "multiuser" features fare when you try that same query when three other users are making queries?
Linux-related reviews seem no better or worse. Caldera OpenDesktop installs easily? Great. How well preconfigured was Netscape? Did the PPP dialer setup utility require odd gymnastics like firing up a terminal window and running it as root from a command line? Did you have trouble accessing the update and patch download site? How responsive and knowledgable is the phone and email support staff you're paying for? Were tasks like printing a screenshot or installing new hardware easy or difficult?
If there is a general disconnect between the low polish level of many Linux desktop apps and the high praise Linux gets in the resulting review, that probably has something to do with the remarkable stability of Linux itself compared to the major consumer operating systems. No antialiased fonts? Awkward printer setup? But gee! The darn thing ran for four the weeks of the evaluation without a reboot.
It's tough to get a meaningful review of, say, Photoshop, from someone who is a product reviewer by trade, and not a full-time graphic artist. Even if the reviewer was an artist in a past job, s/he is seldom going to give the application the hard workout they would have in the course of real-world use. Instead, we get charts giving rendering times for specific effects, a database-style test suite applied to a creative tool, which should be more about flexibility, interface design and stability.
Enterprise applications are easier to review, since you can draw meaningful conclusions from a rigid, numerically quantifiable test suite. But even here, reviewers grab onto the superficial to make major pronouncements. So-and-so's web-based server management console is lacking? But what if most users of the application use a command line to manage the product anyway, as with many databases and web and application servers?
If the reviews that are out there are telling me Linux is highly reliable when in fact it isn't, then I could stand to loose a great deal more than just some download and learning time.
If you're basing business decisions on reviews you read in glossy software industry magazines, your business is in trouble from a lot more than just potentially unstable software. Please turn over technology purchasing decisions to someone less naive and go back to chasing your secretary.
The only halfway reliable source of information about software is the experience of other users, and preferably not the user who made the purchasing decision. Before you buy mission-critical software, talk to other professionals you respect who have used it.
Proud member of the Weirdo-American community.
You can see over 600 reviews of mine on my Web site.
Having said all that, back to the question: Are Linux reviews fixed? The answer: It depends on the integrity of the individual journalist and publication. For the most part, very few reviews are fixed in the sense that there's pay for a positive evaluation.
When it does happen, it's usually for one of two reasons. The most common reason is that the media outlet is actually an advertising vehicle, and this fact is given up front. One example is "Bunting's Window", the high-tech products show that appears on most in-flight video programs in the U.S. If you look carefully, there's a statement in there about how "promotional consideration" has been paid by companies whose products appear in the show. That promotional consideration is typically in the 5-6 figures. The producers know that most viewers don't know what "promotional consideration" means -- but hey, at least they said it.
The second way that reviews are "fixed" is in media outlets that are less open about their pay-for-play policies. That's considered unethical almost universally in the U.S., but not overseas: For example, the (now-defunct?) Japanese print version of MacWEEK ran positive reviews face-to-face with a full-page ad for the product reviewed. In the U.S., it's often smaller and more cash-strapped publications that tie advertising and gifts to reviews results. These publications deserve their (generally low) reputations.
O.K., so there's little out-and-out fixing. However, reviews often skim over negative points. Again, there are two common justifications. The first reason -- and one that affects "community" publications, such as those in the Mac and Linux worlds -- is that readers really *want* to hear good things about their community, rather than serious evaluations. That's understandable, but not (IMHO) good: Saplings may need protection, but they also need a chance to grow into storm-weathered oaks.
The more pernicious sort of "fix" is when journalists abandon their evaluative senses because of peer pressure. We all know the big example: Microsoft Windows. [Insert here description of bugs and security holes ignored by the media.] Why do they do this? Because... well, because they're human. They may have friends at the company, or have personal (non-financial) reasons for wanting the product to succeed, or whatever -- the fact is, it happens.
However, *almost never* do journalists give good reviews for financial reasons, or for gifts, or for fear that the company will "dry up" if they slam the product. If it's valuable for a company to have its products appear in a publication, they'll be back with the next version, no matter how bad past reviews have been. And if it's not valuable to have reviews in that publication... well, then, why would companies bother with it in the first place?
--Tom Geller, Geller Communications, http://www.tgeller.com
Tom Geller
Actually, Jesse Berst actually said all those things at various times in his editorials... They may be taken a little out of context, but in general accurate...
> this all came from a single Tucows article
Someone on Linux Today suggested that it was a troll on Tucows to suckerpunch ZD. It would be hard to prove such a thing, but it was kind of funny to see the Tucows writer claim with a straight face that he got offers of free copies of Linux if he would gloss over its faults. I suspect that there is more here than meets the eye, and it will be interesting to see what the author says next.
> a more interesting (and feasible) conspiracy theory is mainstream media sites posting FUD to anger specific groups to get more hits.
AKA trolling. Yeah, that's my theory about why JB alternates between praising Linux and dissing it. The alternations keep both sides coming back for more; if he sang the same tune all the time, one side or the other would learn to ignore him.
Just think... all the trolls here may just be doing their homework for Journalism 101.
--
Sheesh, evil *and* a jerk. -- Jade
This kind of thing really annoys me. This was featured on the main ZDnet page for awhile. There were a lot of "talkbacks" flaming the article for its flawed content.
Altavista, who gets its "tech" news from ZDnet, also posted the article on its frontpage.
First of all, this all came from a single Tucows article which had very little evidence other than the one ancedotal incident with Mandrake. The ZDnet article also mentions Red Hat, but with no links to back up their accusations.
Second, and most obviously as others no doubt realize, Linux is free and the danger of this for Linux reviews is MANY times less than commercial software.
But the thing that bugs me most, is the amount of hits Tucows and ZDnet will get from posting this. As I wrote in their talkbacks, a more interesting (and feasible) conspiracy theory is mainstream media sites posting FUD to anger specific groups to get more hits.
I've been writing as a professional journalist about software for 20 years now. Frankly I remember getting into software reviewing just so I could get my hands on lots of software.
But I've never "softened" a review in order to get more software. The idea is ludicrous. If you're a reviewer it's a full time job staving off the avalanche of software that comes your way -- whatever you write about it. No software manufacturer has _ever_ said "You wrote a lousy review so we won't send you any more software". They usually say something like: "OK, but we'll make sure you receive the next update, because you're really going to see an improvement." And you go: "Oh, shit".
What is at issue is the reader's attention span. A lengthy review berating some piece of junk just isn't very interesting, so the tendency is just not to write about it. And I suppose it's true that if I really like a particular product (Mandrake, say), I'm not going to get too hung up about any downside I come across, unless of course it's a show-stopper. I'll spend more time putting across the good points of the product and stand by for dealing with any problemettes from readers when they hit my mailbag.
There may be journalistic technical reasons for downplaying the downside, too. For example, I'm currently gearing up to enthuse in my PCW column about Lars Bernhardsson's very interesting window manager (http://www.fnurt.net/larswm). Yes, it has some problems, but 1) they're of absolutely no interest to anyone who isn't going to try out the software, and 2) they would actually be very hard to explain (many column inches) to non-users of what is a radically different window manager. So I don't expect them to loom nearly as large in the column as they have in my worktime.
el bid
--
el bid
What if you don't know anyone else who uses the product(s) you are considering?
If this includes the people who work in your IS department, then you hire consultants. First question you ask them(before you sign anything): Which OS do you prefer? Favored response: Whichever is best for the situation.
What if you don't have time to fully evaluate every option?
Then you're not doing your job. Get an assistant to foist the lesser half of your work onto so you can concentrate on "fully evaluating every option."
In the real world many business decisions are made on the basis of incomplete information and against tight deadlines.
In the real world there's a difference between acting professional and being professional. Acting professional requires things like nice suits and witty meeting banter, see The Courtier for a more complete list. Being professional has only one requirement: pure fucking competence.
Therefore I would submit that Angry Badger's post is dead-on. If you're making IT decisions based on reading PC Magazine or whatever, there's a real problem. They should have no weight whatsoever in your decisions. If no one you work with knows about the different apps you're considering, you need to go with a third party, one who doesn't push a given package right up front.
You can trust me on this one, my stepfather wrote that magazine article shit when I was in high school, and he's a complete moron.
-jpowers
-jpowers
When looking at everything in the software industry you have to take out the shining reviews and the FUD reviews. In the middle of thoes extremes is where you will find somthing close to the truth.
/. effect:)
But who do you trust? When it comes to Linux I trust my self. Right now I have a server that has been running for 95 days now! It is running Apache, VNC, DNS, FTP, SSH, MySQL, and many other things. Nothing has crashed. In fact when netscape pre6 came out I put a mirror and it withstood the
I trust my self. I trust the fact that my development cycle on Linux is faster than MS. I trust the fact that I can run more software with less hardware than I can with MS because I am doing it.
I trust the fact that Linux and most of the applications for it are more stable than MS crap because it has been running for 3x longer than MS ever has and under a heavier load.
I trust these things because of my experienc. Not because of some company praising Linux or Micro$oft.
If you are in doubt of which is a better product, review them your self. Or like I said discount the stunning reviews and the FUD then you may come to a somwhat truthfull conclusion.
If at first you don't succeed, skydiving is not for you.
It isn't like you really need to trust reviews...because most of the time you spend reading reviews your looking for the best investment of your money...and in this case, the software is free, so the only investment is the downloading and learning of the software. And yes, that can be a HUGE investment, but it you would still need to invest that time into commercial software anyway.
One of the reasons for their success is their strict refusal of all offers of free products (nor do they solicit). They obtain their products as a regular consumer would.
Their procedures are briefly documented on their Web page.
Competent reviewers who wish to remain objective should adhere to these guidelines.
WWJD -- What Would Jimi Do?
I am quite civilized, and I should be brought a beer immediately. -- Bruce Sterling
It took me a while to figure out what this article is talking about. When the author says "free software" what he means is _not_ the same free as the FSF. He actually means "getting commercial software for free" and not "free software like Linux." The problem with this whole thesis is that I just can't imagine that it's particularly true of Linux writers, because the amount of money you save by getting a free copy of RedHat or Mandrake is pretty trivial, since you can just download them for free off the net.
However, in the world of non-free software, where "review copies" of software can cost hundreds or thousands of dollars, it gets a lot more tricky. I have had plenty of personal experience with people (myself included!) who want to write reviews of product X in order to get a free copy. And that can definitely influence what you write...
Can your IM do this?
Remember that a large percentage of reviews are bought -- larger than most people think. The way to get an unbiased review is to check out smaller sites, like HappyPuppy.Com for game reviews, to take only one example. The only time reviews can really be biased is with larger sites like ZDNet, whose reviews are read by a sufficiently large number of people that vendors have a strong interest in swinging them in their favor.
So when all is told, I suspect that Linux reviews are just as biased as nay others in the industry. Now that Linux qualifies as being in the "industry", this will happen more often until it is at a norm with the rest of technology.
P.S. Sorry if the spelling and formatting are bad -- IE (stuck with it) has decided to all of a sudden not refresh the screen, making it impossible to read wat I'm typing.
Friends don't let friends misuse the subjunctive.
.. I used to work fulltime as a test editor for a major networking magazine, and if I mentioned Linux in a negative sentence or wrote a bad review, I got loads and loads of letters with everything from a big 'Fuck you' to 'You're incompetent - get another job!' and the usual 'How much is Microsoft paying you?!'. This is, by the way, a behaviour that one other group of users share with parts of the Linux communitu, and that is large parts of the OS/2 user base... The first 3-4 times, you can handle this but whenever your email is flooded with mean and aggressive stuff, you tend to get fed up by it and either think twice before writing bad stuff about Linux, or not write about it at all.
-- http://z80.org - all opinions, all the time --
Since Linux itself is free, I couldn't figure out exactly what the reviewers were really getting in return for compromising their journalistic ethics... then it hit me!
The FREE CD!
AOL has been doing this for years, sending me FREE CD after FREE CD!!
I just didn't realize that they were expecting me to give them a "glowing review" in return!
Boy, do I have a lot of writing to do!
I spoke to someone who wrote a review of some compilers for a magazine. I read the review long before speaking with him. He told me that the magazine changed the numbers on the score card. The review in the text did not match the numbers on the score card.
At SD94, Gene Wang was presenting to a very nice breakfast for the press people. One woman refered to a press badge as a beer badge.
This type of thing have gone on for quite a while. That is sometimes why you see a review that looks like a rehash of a press release and wonder, if the reviewer actually used it.
Fight Spammers!
Based on the distribution of advertisements and actual content, how large do you think the advertising department is compared to that of the writing staff?
Truth is, for most magazines (both online and offline), the actual content is kind of an afterthought, so they're hesitant to put too much effort and money into them. So the writers either:
- Rewrite press releases to make it seem as if they've tested the software/equipment/cars themselves. This costs far less time and hence money than actually testing something.
or- The writers get a computer with software/piece of equipment/car that's been thoroughly prepared by the manufacturer to ensure a smooth ride and hence an easy review. You didn't really expect that journalists who gloat over W2K/Linux ever had to install it themselves, now did you?
or- You've actually stumbled across the one magazine that does in fact test the software/equipment/cars they're sent, but in this case you'll notice as the reviews are always much less positive than those in other magazines.
I only recall, for example, one review of Corel Linux that stated that it was easier to install for newbies than most distributions, but still a long way off from being completely userfriendly (user meaning: my grandma). Game magazines tend to be an exception, as they're run by people who love to play games, which is why there were almost no positive reviews of Daikatana, for example (even though I've seen one magazine that claimed that John Romero was "setting new standards of excellence" hahaha).This is not news, this is stating the obvious and adding a conspiracy twist to it.
News and bla for computer musicians: http://lomechanik.net/
ZDNet has no room to talk about biased reviews.
Exhibit A: Review of CyberSitter in PC Magazine a while back. Nothing but glowing praise.
Exhibit B: Recent reviews on the ZDNet site of various graphics programs. The hideously overpriced Photoshop received high marks, while the similarly-featured and much, much cheaper Paint Shop Pro 6 received a scathing review (from ZDNet staff, anyway; the staff gave it 2 out of 5, while readers gave it 4.5. What's wrong with this picture?)
Talk about the pot and the kettle. ZDNet appears to bias its reviews toward whichever company is buying the most ad space.
This is a Chao. A Chao says "Mu."
I recall that when I bought and started using NetObjects Fusion 5.0 that I immediately had problems with it. The software was buggy in key areas, crashed itself, crashed the system, etc. I went on the NOF user groups and NetObjects was being dragged over the coals by a dozen or more people who accused it of rushing out a horribly flawed product. Then I read a Computer Shopper review of NOF 5.0. Nada on the problems. Either the reviewer only used the most superficial features of the product for about 10 minutes, or he just decided not to mention that it had major difficulties. Maybe most product reviews are untrustworthy. The only place I can recall reading truely critical remarks is Maximum PC. Those guys don't seem to pull any punches.
First, I've been writing reviews since 1984, when I purchased my first Compaq computer -- the sewing-machine version that couldn't take a hard drive. Yes, the lure to do reviews was "free" software...but I quickly moved to hardware reviews where you don't get to keep the product at all.
Reviewing platforms, or reviewing software cross-platform, is some of the roughest reviewing possible. Think about reviewing two different operas on two different stages and you get the idea.
The problem with all the reviews I've read that attempt to compare the performance of Linux and NT is that the assumptions behind the test methodology are geared toward either the Linux or NT model. That means you are trying to compare apples and oranges. The two are completely different beasts. Complicating the problem is that much of the performance testing methodology developed for the computing industry is centered around Unix -- very centered around Unix.
The most unbiased performance test suites around are the SPECmark series. I learned the hard way just how Unix-centric the SPECmark series of tests are when I tried to port them to the Macintosh OS -- indeed, I never finished the job. Indeed, I can't even see how to port the SPECmarks to the Windows environment because of the large number of Unix-isms built into the benchmarks. The reason? The benchmarks are actual live real working applications, designed to do a job and not just fiddle bits to eat up resources.
While I agree that the most unbiased reviews come from users, all the vast majority of them can tell you is that "Hey, it worked [didn't work] for me for what I do!" The vast majority of users don't have a clue how to do a structured evaluation of software or hardware...even slash-dot readers. That's why I was able to make a comfortable living for about ten years, writing reviews.
The only review methodology that might make sense is to develop a task, and have two teams configure systems to perform that task. Even then, you will run into variances because the teams may have differing knowledge levels of the systems they are trying to tune for the task. This is the big problem in SpecWeb marks, judging from the reports I've read lately on their sites.
Will there be a fair review? Right now, I think the issue is in doubt.
To the subject of "pay for play" -- in the fifteen years I have been reviewing stuff, I have been offered a number of bribes. None of the companies trying to bribe me ever met my price; hell, they never came close! Other companies have threatened me with lawsuits for what I wrote. None have gone to court, and all but one was settled out of court in my favor. (That one, I admitted that I did the review wrong, and the magazine and I came up with a fix that satisfied everyone.)
Of course I'll eat their food, and of course I'll listen to the PR flacks. That doesn't mean that I'll write a review based on what flacks tell me.
As a founding member of the Internet Press Guild (www.netpress.org) I subscribe to a canon of ethics that require me to write what I experience with products, not what someone tells me to day. That includes editors -- there has been more than one article I've pulled because an editor disagreed with my findings. IT'S MY NAME. Stephen Satchell Satchell Evaluations
I used to work for a hardware manufacturer that shall remain nameless. Whenever we would release a new box, we would not only give them to the reviewers, but were told who the reviewers were and how to contact them. Then if/when a problem arose (not with my stuff, of course) they were on the phone with the dude doing damage control. Some of these guys had exposure clear up to the exec-vp level on some products. Talk about service! You try getting that level of service when you buy the junk.
This has always been the case. Its not that they give good reviews for want of free software/hardware/etc. - they know they'll get that because they have circulation. Its that they give less than perfect reviews because they are not totally objective. And then there's the advertising revenue to consider, etc.
The only objective reviews are those published by those who care more about helping their fellow /.ers than the one trying to sell the junk.
Long live the free press!
"Stop whining!" - Arnold, as Mr. Kimble