Slashdot Mirror
Where Do You Go for Worthwhile Product Reviews?
An anonymous reader asks: "What's the deal with reviews and product comparisons? My boss wants independent comparative reviews of proxy and web servers to use to make/justify his decision. We all know that what the vendors write about their own (and competitive) products, so I tried searching for 3rd party reviews. I can find heaps of articles on the web telling us how great IIS is or how good Microsoft's Proxy server is, but nothing showing a back-to-back comparison of Squid vs. Sun Java Proxy vs. Microsoft Proxy, and the same for Apache and IIS. What's happening here? Where can I find an honest back-to-back product comparison?"
I was a student at the time. I was interested in race relations, so what better way to get the 'inside scoop' on the whole deal than to see the KKK in action. Know your enemy, and all that. After the rally I sat down with the Grand Cyclops and asked him point blank, 'Why can't I find any good information regarding the superiority of any given race over any other given race?'
Your question reminded me of that, for some reason.
quit and work for a real company with employees who don't believe published trade media "review" propaganda.
Everyone's use will be different... For production use, I've rarely found independent reviews that test what I want tested, in the conditions I want, doing the same things I'm looking to do.
For your example case, I'd personally test each product in-house, drawing up conditions and test plans ahead of time. If you're planning a significant deployment, vendors will generally supply product for you to evaluate. Sometimes if you ask nicely, too.
Just my two cents... And yes, I get that it may not be feasible. Its labor and time-intensive. But in-house testing and evaluation almost always beats 3rd party reviews, in my book.
Lie about it, and recommend whichever vendor gives you the most kickbacks.
Slashdot - where whining about luck is the new way to make the world you want.
Choice magazine does unbiased, in-depth reviews, comparisons and evaluations, although from what I've seen so far their software reviews are more consumer oriented.
Search in google the product you want to review then add the following phrase ", problems"
:)
I'm sure you would get all the bad side, then weigh which one of the products are the lesser evil
sample query: iis, problems
There's just too many product types out there to expect any site to track the feedback of, well, the entire market of stuff that's out there. For stuff I've looked for recently, garden equipment and robotic vacuums (ends up there's a bit more than just Roomba out there), I've found specialist forums and even commercial ads to be useful in tracking down details to search further on.
As far as generalist sites - I've found the eclectic community over at Slickdeals.net to be fairly useful in getting a quick grip on what to look for - but forum-goers there are intentionally against bad-mouthing products (thread-crapping), so you have to take a large variety of recommendations there with much due skepticism. Great place for leads though.
Then, of course, there's the Resellerratings-style sites. Once you've scoped product details, it's quite important to get feedback on who you're buying from. Again - due skepticism in all regards will help you in various ways, but large negatives or fake praise for rarely-rated stores can be an important part of an investigation for a large purchase.
If it's not a big purchase though, I'm usually comfortable just hitting Froogle, Amazon, or NewEgg and being done with it.
Ryan Fenton
I've given up searching "$PRODUCT review".
If you're lucky, a magazine will have a comparative review and will have taken roughly equal amounts of ad revenue from each of the competing vendors. Useful search terms include "shootout" and "versus".
Anecdotal evidence from the tech community can be a heuristic if you're wondering about general bugginess and hassle factor. If you need real benchmarks, the only ones that mean a thing are those you run yourself.
Are you running a mixed shop or a single-vendor one? Don't underestimate the pain of interoperability and equipment management hassles if you've never experienced them.
Work as hard as you can to pin down what you need: good scaling on SMP machines? Easy management? Particular features? Good local talent pool for running/fixing it? Low purchase price? Support contracts? The more questions like that you answer, the clearer the choice will be and the easier the web searching will be. "Apache scale SMP OR cluster" is likely to get more informative results than "Apache IIS comparison".
If you are worried about security, then abandon all hope of useful information from the press, concentrate more on lockdown and scheduling updates then on the choice of product (but never install IIS 5), and keep an eye on the news.
Cultivate sysadmins in other places who have environments about your size and with similar needs.
Can't he trust you to decide what fits your needs?
IIS and Apache are _very_ different, for example, and you can't choose between them based on product reviews.
There might not be a huge market for reviews of proxy servers, which means it doesn't sell ad space which means it doesn't get done there. And, even if there were I'm not sure there's a Consumer Reports level of integrity in any of the IT periodicals that you might be looking in, they all take ad money.
Is this important to your business? Then why not hire someone who's done this before to talk to you about how the different products work and how they might apply to your specific situation? That's why consultants exist - you usually don't need someone with this skillset in your business, but today you do, so you hire him in for a day or so and you learn alot of specialized information really quickly.
If it is an important business decision and you're limiting yourself to magazines or serendipitous website reviews you're probably not considering the costs of the project as a whole.
Oh, and for hardware reviews NewEgg is pretty good. Amazon is not so bad for published software. These are obviously simple cases.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
> vendors will generally supply product for you to evaluate
It's getting harder and harder to find one that will. Even though we bought 90 750 GByte drives from Newegg, the wouldn't even let us borrow a selection of controllers to try with the drives. We were buying either 45 2 port or 23 4 port controllers along with a $30,000 hard drive order, but they still wouldn't do it. I had heard those drives were finiky with controllers, and that was correct since we found only a single controller that would work even though we bought one of each that Newegg carries that was less than $100 each. We now have to RMA each of the controllers that wouldn't work and eat the 15% restocking fee.
Aside: the big new Seagates are very picky about controllers. The Promise SATA300 TX4 was the only one that would work with them. Unfortunately it's $60. Nice drives, but I thought at first we had received a bad batch of 90 drives.
If you go to a related forum you can often post asking about a couple products and often get a reply from someone who used both. Forums can be pretty useful and users can have great expertise. I suspect some manufacturers even let forums do problem solving for them. I got much better advice for a new ASUS mobo from a user in a photography forum then I could get from the ASUS forum. If you go to a forum for webmasters you can get all sorts of advice on servers. I don't trust a lot of published reviews in magazines that take advertising. Recently I was shopping for an LCD monitor and reviews often use wrong specs and don't seem to understand the product very well (they won't even mention if a panel is S-IPS or S-PVA etc). Certain brands seem to get a lot of wiggle room and a look at the ads on the bottom of the page usually shows why. One good site for head-to-head comparisons for monitors is http://www.lesnumeriques.com/duels.php?ty=6&ma1=52 &mo1=149&p1=1606&ma2=36&mo2=105&p2=1041&ph=6 .
I know monitors are not what you asked about but I still think forums are best bet. You may be lucky not many reviews exist because I find its a good way to get hung out to fry.
Erm, 750GB drives from Newegg are SOHO equipment. I'm sure, if you as a company asked IBM, HP, or whichever your server supplier is for a evaluation pack of one of their SAN appliances, that would've happened in a rather short space of time.
We usually do that for our regular customers (i work for an IBM BP).
I usually type in "product manufacturer's name" .com and read the opposing arguments. Whichever spouts the most useless, redundant statistics about their product ("This printer has USB AND plug and play support! PLUS it can print 8.5x11 and is composed of protons, neutrons AND electrons!") or the most bullshit ("MADE FOR GAMERS!") obviously is trying to overcompensate for a lack of serious ware.
Ginga no Rekshiya Mata Each page.
Reviews aren't for you.
If you're in a deciding position (i.E. CIO/CTO), and your boss doesn't trust your decision, it isn't going to be easy anyway.
But if your real question is "how should i decide?", then here's the answer:
Evaluate! It's to only way to be sure the product meets your needs completely. Yes, it costs alot of money and time. The alternative is to guess a product.
E.g. if you're already using a Windows Environment with Active Directory (and like it), going for Exchange as a Groupware isn't a stupid idea. Or for ISA as a Proxy/PF solution.
You could start with www.measurement-factory.com/.
I can only wish it was that simple. But other than keeping quitting and moving to fresh small startups (which probably don't exactly pay a lot for an admin), it's not really practicable.
And even as a startup, if your work isn't purely developping your own product, you end up doing stuff for various clients. Which have their own ideas set in stone, based on reading some IT-for-retards ragazine or on a golf round with the nice salesman from MS/IBM/whatever. And we all know that you can't trust those techies with their techno-babble speak, whereas a salesman would never tell a lie
An as soon as the company grows past a certain size, and doubly so for companies whose primary product aren't programs or IT services, well, my favourite metaphor is: clue is heavier than air. The higher you go up the hierarchy pyramid, the thinner it gets. If clue were oxygen, you'd see higher level managers blue in the face like they're Smurfs.
To their defense, it's not their job to know the finer points and differences between web servers, but then it also shouldn't be their job to take such low level decisions. So you have a bunch of people taking decisions about stuff that they knew nothing about, and it wasn't their job to know anything about. What really makes it worse is having several layers of shielding against the effects of bad decisions. He made some "strategic decision" to go all-IIS, and can claim credit for any positive results (even coincidental or immaginary), but it's _your_ fault if something goes wrong with it or it takes too long to port your application to it. And whenever such shielding is in place, out goes the incentive to get any real clue or to refrain from taking bad decisions.
But, to get back on topic, you'll find very few large companies where such shielding from responsibility isn't in place. So you're limiting your employment oportunities drastically if you only accept jobs from the few who aren't led by people who don't take their IT info from ads and salesmen.
Probably a more realistic thing to do is realize that, in the end, few things matter _that_ horribly much. Some people have a penchant for blowing minor differences out of proportion, and make mountains out of molehills. There _are_ product issues that matter, and there _are_ awfully bad management decisions, but there's also a lot of stuff which really isn't as critical as the "either something is perfect or it's complete crap" gang makes it sound. If some proxy is 5% faster than another, pfft, it doesn't even start to matter. You'll want plenty of margin for when you get slashdotted anyway, but 99% of the time it'll be _way_ under-used. Having 5% or even 10% less unused capacity isn't the end of the world.
And once you do a realistic assessment of how bad it really is, a lot of things aren't _that_ horrible after all. So management picked a less than optimal proxy. Who cares? Compared to some other decisions I've seen various managers take, this doesn't even start to matter. If you're going to quit a job solely because of something like that, methinks you need to rethink your standards. And maybe look for an OCPD support group in your area.
A polar bear is a cartesian bear after a coordinate transform.
Where can I find an honest back-to-back product comparison?"
In your test lab, stupid. I havent yet found a vendor that would deny an evaluation of their product.
Going for what single product review will recommend is silly. I rather go for infomration on quality in many reviews, collect thoughts that are very similar, discard that are wildly different, to get understand what the product is *about*, in terms of use and see wether I like it or not. My preference is with manufacturers that don't boost specs, on monitors, like Samsung does:
g ital/ls19mewsfxaa.asp
m JLK/viewproduct/Samsung_SyncMaster_931BW/
2 E16824116381
http://www.samsung.com/ca/products/monitor/lcd_di
vs:
http://www.hardware.info/en-US/productdb/bGNkbJiX
But I go for hidden gems like:
http://www.newegg.com/Product/Product.asp?Item=N8
I find the best source of information about a product is the manufacturer's website. Or just ask a salesperson in a distributor's showroom. They'd never give me bad advice, would they? You're not going to rely on some random "third party" on the internet, are you?
... and then they built the supercollider.
I have 3 SATA Seagate 500GB 7200.10s hanging of a Sil3124 with no problems, Silicon Image provide specks for the Linux driver (which I always like to support) and they're supposed to play nicely with FIS based port multipliers so in theory this controller can have 60 drives hung off it.
In order to save our freedom it was necessary to destroy it.
Some things I learned include...
1. The only job a real programmer will take must involve Ruby on Rails
2. Never buy a MS product
3. Filesharing music is "fair use"
4. Programmers should not create closed source programs EVER.
5. Linux sucks, BSD sucks, MacOS sucks, and Windows sucks. (I am posting from an IBM/360)
6. The only safe browser is Lynx
What's the best car to buy? Although some brands have quality issues that rule them out, ultimately the question "which car is best?" depends on who will be driving it.
I'm a former cabdriver, so I'd be happiest in a Dodge Charger police package; my ex-wife hates big cars, so her Saturn is perfect for her.
Similarly, I'm a Windows geek/MCP. I'm better at installing, configuring and running M$ products, so IIS would be best for an environment I had to design and support.
Others who read this would be far better off (and happier) running *nix, so a non-M$ solution would best meet their needs.
Choose the one you want, then find facts to support your preference... they're out there somewhere.
I get all my product reviews from the front page of Slashdot. I know they're particularly trustworthy when they're in the form of an article about the product, rather than a "proper" review. It helps when I can google certain key phrases from the article and find identical matches in other articles about the product. I also pay close attention to the square and rectangular colored boxes that announce products on the different pages of Slashdot. They really get my attention when they flash or make sounds.
This guy's the limit!
These won't be cheap, but you said business, not personal, so give these a try.
The Burton Group http://www.burtongroup.com/
Forrester Research http://www.forester.com/
Clearly, we need people to write reviews of reviews and post them.
Thanks to slashdot, I know that the PS3 is total crap, and that the Wii is the best thing since sliced bread. I also know that Vista is nothing compared to Mac OS X. vi is better than emacs.
perception is reality
That's what happen when you don't carefully read other comments before posting :)
perception is reality
The more complex the software, the less useful the review will be. Unless it's something major like Office 2007 which will get a 25 page special, with almost anything else journalist paid the same amount whether it's twenty hours work or 30mins and in most cases, it's only just about worth it for a 30min one.
In some ways, those that write for free, the so called vanity writers, are producing better stuff because they'll quite happily produce a ten page review with dozens of images for pretty much anything whereas a 'proper' journalist will only have time to do a single page if they are to keep the food in their families mouths.
As someone else suggested, the best bet is to google on the product and 'problem'.
I want a list of atrocities done in your name - Recoil
RTFM
Before I buy any piece of electical equipment or software, I download the manuals first and then compare these.
Advertisements can juggle around with specs and features and make all sorts of claims which they don't need to keep, or atleast can be interpreted to fit the actual lack of features.
Manuals have much less room for this and will typically expose problems with a product, since manuals have to help the user get around these problems. They're also invaluable in determining whether a product will be user-friendly and whether the features claimed do what you need them for.
There's nothing that beats evaluating it yourself, since even manuals don't mention every single fact you might want to know. For instance, one deciding factor when I bought my TV was the speed by which I could change channels (which can vary a lot!); manuals and reviews typically don't mention this, so I tried out the remotes in the store.
Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
Sometimes its too expensive to run in-house tests. I work with a variety of small community groups that use IT, and also advise friends who are students and low income. I can't turn round to these people and tell them -
:-)
"Buy a laptop (/digital camera/access point/ etc) randomly and if it doesn't come up with the goods throw it in the garbage can out the back of your property and buy another one and test that, continue until you find the right one"
- I'm afraid that's just not a solution for people/organisations on limited income. Useful advice for good reviews to read is really useful.
p.s. can I have all the brand new kit that you throw out for not being up to standard after you've tested it?
It really depends on the type of product. For computer and related stuff I use CNet and PCWorld and also go to the forums that seem to cover the product or category. For PDA stuff I go to PDAPhonehome.com and read reviews/post questions. Fr software, I usually go toe the appropriate support board and read the posts to learn about bugs, vendor responsiveness, etc.
I just found this site yesterday: http://www.hardwareranking.com/ while looking for reviews of Syncmaster 244T
It seems to pull in reviews from many different sources.
Newegg has great prices and good service for a distributor. They're a big warehouse, don't expect anything beyond basic order information. You want low prices, great. You want technical support, forget it. Go with someone else and pay more for that support.
In the end it was worth it, however, since the Printronix company never sent us the $12k bill and we got the printer for free. I still wish I had asked PHB right after I ruled out IBM.
For IT-related stuff, it's Google or your favorite industry-specific newsgroups.
For general "gadget" related items, I typically check out The Gadgeteer first, then Google.
For digital cameras, it's Steve's Digicams all the way, then Google.
For cars, it's AutoTrend or Consumer Reports Autos, then Google.
For general household stuff, it's Consumer Reports, then Google.
And in pretty much every case, I check Google.
My mom always said, "Jim, you're 1 in a million." Given the current population, there are 7000 of me. God help us all!
I work in the embedded industry, and I've found that it's impossible to find non-biased product reviews on cpu nodes and switches for the associated architectures (ATCA, cPCI, ETX, uTCA, etc..). Everything is a press release from the manufacturing company.
I had an idea to create a site that would rely on submitted product reviews by engineers who had actually used each product. I kind of gave up on it, because I figured it might be too niche of a market to actually generate the traffic needed to drive the content.
I guess here is as good a place as any to ask what you guys think...
I use http://www.anandtech.com/ for the best reviews on the internet! I also just google the item/product.
Ad eundum quo nemo ante iit!
Your boss is giving you a "make work" project; any (non-government, non-union) IT employee ought to be find such reviews (biased, of course to his/her particular leaning) in about 15 minutes. Is it possible you've done something to cause your boss to no longer trust your judgement?
I don't know of any general product evaluation sites so I don't think you'll find a standard research approach. In gathing and organizing information from a multitude of diverse sources, I would do the following:
* get a spreadsheet...
* define what you want. This should become a long list of function points.
* evaluate the importance by weighting the important of each function point (eg. 1 to 10)
* get your stakeholders to review and approve your list if they haven't already when providing the importance score
* start looking at potential solutions. For each function point, input a percent of coverage provided by that solution. Footnote your source for future reference.
* group related function points for clarity so you can evaluate a products weaknesses at a higher level
* Scores will be higher for packages providing better function point coverage * start communicating your findings via charts, graphs, etc.
For extra credit, you can correlate your function points with gross margin (i.e., to what extent and to what level of impact a given function point will have on gross margin). Get with the accounting or operations folks.
That's my approach. It's not cool, but I haven't found anything better.
Nowadays with blogging as a profession, as that a lot of the postings are rants and opinions of people who have never actually bought or used the item (case in point - PS3 or Wii).
"Enjoy what you're doing! If it becomes drudgery, you're doing it wrong!" - Jim Butterfield
'Why can't I find any good information regarding the superiority of any given race over any other given race?'
Because you've never spent an afternoon with La Griffe du Lion.
Be afraid, people, be very, very afraid.
If I'm thinking about purchasing an Acme Widget, I google something like acme widget sucks .
Or if it's a technical product, like the Acme Flux Capacitor, I might google acme flux capacitor teh sux0r .
Because a bad review is worth its weight in gold.
http://www.dpreview.com/
or
http://www.fredmiranda.com/reviews/
I've found over the years that reading about the issues folks are having with Product X in one of the technical newsgroups is often quite worthwhile. It doesn't always cover enterprise stuff, but that isn't normally my area of interest anyway...
Mainframe/UNIX Bit Twiddler and long time Windows/Linux Hobbyist.
The Theorem Theorem: If If, Then Then.
Here are some of my tips to avoid products that suck.
1. If the product is available on Amazon.com, check out its reviews. Also note that sometimes slightly different/older versions of the same product have more reviews. It takes some time to sift through the sycophants and astroturf but it's a great source.
2. Search for negatives. Try google searches on "*product* sucks" or "*product* problems" and other permutations to find peoples' complaints about a product or its company.
3. Look for refurbs.... if you see a lot of refurbished versions of your product in the marketplace, this is a bad sign usually.
4. eBay... search completed auctions to see what the going value and interest is in the product. Also eBay auctions tend to have the most comprehensive array of specs on these products, often more informative than the manufacturer's web site.
5. Avoid all the large web sites with the bogus reviews and meaningless content. If you search on "*product* reviews" you're guaranteed to get a bunch of shill web sites that are worthless.
Comment removed based on user account deletion
I cannot and will not rely on a single source.
The only thing new in this world is the history that you don't know.[Harry Truman]
They're NEVER wrong!
For things like monitors / TVs, screw reviews. Go evaluate a lot of different models in person. Then you can see just how much the pixel size and contrast compare without having to wade through tons of numbers that may not even have much bearing on the actual picture quality.
For computers, I read the reviews at somewhere like Tom's Hardware. But not the hardware reviews, the software benchmarks. Then I take a look at what they're using for the test setups. Buy up those parts, slap them together, and you have a rather good system. You might need to look up older software benchmarks, though, if you're not trying to build a higher-end system.
You can also, to a lesser extent, go by brand reputation to a degree: e.g. Sony sucks hard, Maxtor sucks, but maybe Crucial has been pretty good to you, etc. Of course, you may find that one particular product in their lineup is the black sheep, so that's by no means perfect.
To a lesser extent, you can also search for "$PRODUCT sucks" and compare the hit counts, assuming that higher means worse. But that's highly unscientific, and if it ever becomes widespread, you can expect them to start manipulating the results via spam pages, etc.
Just make up your own numbers and write a fake review on your website that says the manufacturer giving you the largest kickback is best. Show that review to your boss for validation - it's not like he has time to check. Sheesh. The things you have to teach kids these days...
That is all.
Search Phrase: linux sucks 1,630,000 hits Well then. Good enough for me! Tonight the linux server in my closet gets wiped and Solaris goes on. Unless... Search Phrase: Solaris sucks 602,000 hits Damn. Linux sucks 2.7 times worse than Solaris, but apparently Solaris still sucks.
Now there's a guy who knows what's what.
And now, a PSA from David Lynch.