Slashdot Mirror
Does Software Need a Siskel and Ebert?
theodp writes "Over at Scripting News, Dave Winer laments the lack of serious software reviews in the NY Times. That wasn't always the case, recalls Dave. 'When they started doing software reviews in the early '80s it was with the usual Times flair,' says Winer. 'But somewhere along the line they stopped taking tech seriously. It's as if they would only review Saturday morning television shows. How could television like The Sopranos or Breaking Bad take root in the culture if there was no criticism that discussed it? Yet that's where we are today with software.' So, does software need a Siskel and Ebert (or A.O. Scott and Manohla Dargis for you highfalutin NYT readers!)?"
Software is everywhere.. systems are too disimilar. The fact that Mac OS != Windows alone without including Linux means this task is Herculean. We do have people who review software more seriously but their in more specilized formats. If you want to do something more open and with a wider target audience like S&E then I don't see how it could work with Software
Just another second banana
Most reviews are shills ... companies have whole departments dedicated to getting bloggers to post sham reviews ...
The moment a critic gives a FOSS package a bad review, hundreds of "advocates" come out of the woodwork and assault the author with everything from crude obscenities to accusations of trolling and shilling.
No. Siskel and Ebert rocked don't get me wrong, but we have a thing now called the Internet and google which can pretty well give you any info you want on most software out there. Anything in a newspaper or magazine is going to be influenced by $$$ anyway while the Internet is typically pretty damn raw
There are literally hundreds of review sites. Some of them are even name recognizable. Gamespot, Metacritic, IGN, etc.
Before you say that they are limited to a certain category, when was the last time Siskel and Ebert reviewed an Anime that wasn't produced by Studio Ghibli?
He declared games to be even more creatively bankrupt than movies, and came up with the Boulder Pledge. ("Under no circumstances will I ever purchase anything offered to me as the result of an unsolicited e-mail message. Nor will I forward chainletters, petitions, mass mailings, or virus warnings to large numbers of others. This is my contribution to the survival of the online community.")
The really funny thing to me is that computer games are pretty much the only sector of software with something even approaching a regular review/rating system, and they have long acknowledged that their "Roger Ebert" is either not writing reviews or hasn't been born yet. For other software you have to rely on advertisements disguised as reviews in PCMag et al.
Perhaps Statler and Waldorf...
I was thinking more along the lines of "The Computer Chronicles"... if you are old enough to remember that every Saturday.
Yeah but most of those "Review" sites are not really neutral - I don't trust them.
CNET is pretty good with the Editor's and User's ratings.
Other than that, I think most review sites are just advertising in disguise.
All kinds of places review Games and Apps.
If your a professional there are long articles written about each upgrade for your tool suite.
There are lots of long articles about each change to Facebook, Yahoo, and Google.
So what is left? I don't use that much software outside of work and play. Tax software?
Are there even any TV movie critics left now? It sucks because I have to root around online to learn about quality indies now. It used to be I just had to watch Siskel and Ebert once a week.
The cow says "Moo." The dog says "Woof." The Timothy says "Thanks, valued customer. We appreciate your input."
In the 1980s I did not go to the NYT for software reviews, I went to Byte and other serious magazines for that information.
Time Bomber the Book coming soon.
If you're talking about games or entertainment (iTunes, etc.) then a duo like Siskel and Ebert would be very helpful. When it comes to operating systems and productivity applications, I would prefer an approach more like Consumer Reports.
[Sir Garlon] is the marvellest knight that is now living, for he destroyeth many good knights, for he goeth invisible.
I knew that the future of reliable reviews at the NYT was over when David Pogue gave MS Word 6.0 for the Mac a good review. It's almost universally seen as one of the worst software upgrades in history.
I emailed him and told him I was disappointed.
With upgrade cycles within months, why review something that gets added features within a year.
In the old days, you made an investment with s/w products, cause the refactor/version cycles were in years. Now it's in months--for cloud apps, maybe weeks.
http://www.youtube.com/watch?v=CpoGIV_XMD8
If it turns out anything like movie reviews. Every software will be named 'software of the year!' "must have!"
The entire review thing has turned to shit really. Marketers got to everyone with the greed.
.
Now the software package has become the app, and is priced very cheaply.
The resulting high-volume, low-cost business model produces an audience for the app that gives ad hoc reviews via social media and other word of mouth communications.
OpenStep has morphed into Mac OS X
FutureWave SmartSketch became Flash
Altsys Virtuoso became FreeHand and has since been buried by Adobe
I desperately need a replacement for the latter, and it looks like the only way I'll get a decent vector drawing program in the future is to virtualize OpenStep and run Altsys Virtuoso in that.
Sphinx of black quartz, judge my vow.
Reviewing software is not even remotely similar to reviewing movies. A review can watch a movie (multiple times, even) and write a review that covers the breadth of the experience within a nominal work week. Would anyone trust a review of any serious software from a novice with 40 hours of experience with same? I'm sure that commentary on icon design and font selection would be insightful, but that's about it.
Software, games anyway, turned into pointless, stupid, doomed clones of past things that sold.
We are currently laboring under a continuos stream of MOBA clones and "Action MMOs", which abandon depth in exchange for console-style button mashing.
They roll out, and off a cliff, rocketting into the ground like Wile E. Coyote, and their investors lose millions, and wonder why.
(-1: Post disagrees with my already-settled worldview) is not a valid mod option.
Software reviews in a newspaper? Even online, it's not going to reach people.
The era of the professional critic is over. His monopoly on his soapbox passed. The problem of sorting shills from genuine isn't new nor exclusive in the digital era.
It's kind of ironic Ebert was mentioned when it comes to software reviews. What Siskel and Ebert did for decades was give their opinion on works of art created in the film medium. Of course they would take into consideration the technical achievements of the film (cinematography, timing, etc), but even if a film was implemented perfectly, they would still give it a bad review if it wasn't entertaining or worthy artistically as a whole. Obviously the whole thing is quite subjective.
Ebert famously stated that "Video games can never be art", and although many disagree with him on that, he does make a point why a "Siskel and Ebert" kind of reviewing doesn't really work when it comes to software. By what criteria is software to be evaluated? Certainly the artistic side (and do utilities and the like even have an artistic side, especially if they're using the platform's standard widgets and GUI elements?) can't be a major factor, unless we're talking about games. Obviously then that depends on what you even mean by "software". Are you talking about applications? Apps designed to streamline access to a web-based service (Facebook, Twitter, etc)? Games?
For example, when evaluating the official Facebook app, do you simply compare how well it works and many features it contains compared to the web version, or do you also get into issues about Facebook more generically (like privacy, etc). Do you get into details like how many user interactions are required to accomplish certain tasks, start up times, and other technical aspects that can be objectively and directly measured?
Another problem is just pure volume. How many applications are produced in a year compared to films? So how do you decide what applications to go to the trouble of reviewing? Stick to the top 100 lists by popularity? Obviously that method would suck in a number of ways. It is easily in the realm of human possibility to watch all feature-length movies that will be shown at the average movie theater, but when it comes to apps, it's simply impossible to even try them all.
When it comes to software found in app stores, it seems to me that the simple 5 star reviews by users is working pretty well. Apps quickly accumulate 1 star reviews if they are greedy (very little free content, or it costs a fortune to unlock things individually when it they should just sell the app outright, etc), buggy (people will quickly butcher an app if it is unstable), are just simply crap. Of course the 5 star reviews can be manipulated by shills, but that can't erase the 1 star reviews. That's why it is very helpful when app stores show individual counts (how many 5 star reviews, 4 star, etc). If you see a lot of 1 star reviews and the app still has a 4+ on average, then that is a warning flag and a quick perusal of individual reviews will reveal what's going on.
Better known as 318230.
Software is not like movies, it is like cars. Stick with the well established analogies, people.
Seriously, software needs to be reviewed the way cars are... both for performance and functionality, and for aesthetics.
Ideology: A tool used primarily to avoid the bother of thinking.
You look at games and and they more or less have it figured out. While a company may have shills with good SEO skills they have a much harder time defeating review site like Metacritic that calculate a weighted group think. The problem is money. There's money in game reviews. There's money in large enterprise software analysis. It's pretty sparse when it comes to productivity and utility software.
I thought the Penny Arcade guys were the Siskel and Ebert of software reviews.
It's absolutely impossible to do real software reviews of many software products without risking getting sued. This is due to the industry using NDA's for software that prohibit unapproved reviews. NDA's are why on release day you will all of a sudden see a plethora of reviews on release day. Reviews off of sites like Amazon are largely worthless due to the sheer number of shills and the most popular reviewers getting large quantities of merchandise for free.
One merely needs to look at what happens with video games to know why. If you work for a video game magazine and give a scathing negative review you won't get selected to review the next product from that publisher. After a while you end up being unemployable as video game reviews have to be ready for release day. It doesn't take long to realize you have to carefully write about a game without pissing off the publishers. The net result is that pretty much every game review web site effectively becomes a shill for the publishers as they can't afford to miss out on day zero releases.
Take your favorite site and select all their reviews and put them on a bell curve. Most (average) software should fall somewhere in the middle of their scale. In practice you will find many sites will give average reviews of a 7 or 8 on a 10 point scale. An honest site will fit the bell curve, a dishonest site will quickly be exposed by the bell curve distribution being shifted towards better scores. These problems are why some sites make claims about refusing to sign NDA's, they are showing that they have more integrity to give honest reviews.
This can even extend through to things like operating systems where many beta or rtm releases have excluded the right to review the product without approval in exchange for getting an early release. One simply needs to review the history of Operating System releases to see the effect of reviewers that are afraid to piss up companies. Look back at Windows Me, Vista, Mac OS's before 10 and so on and you can find a plethora of initial approving reviews (ZD Net in particular comes to mind).
The problem gets even worse with actual commercial software. Read your fine print from Oracle or any other commercial product and you will almost certainly find the license prohibits benchmarking and other similar activities that could be used for a review - especially for trial versions. In addition to license issues the cost for commercial software makes it unfeasible to purchase.
Trying to review enterprise class software becomes even more unfeasible as you can't simply install it. In order to properly set it up you need a consultant who knows the product fairly well and that is cost prohibitive for a company that isn't even going to use it. Since enterprise software tends to include language in the EULA that prohibits unapproved reviews no consultant, who naturally depends on having a good relationship with the publisher, is going to help you if you might say critical things about it.
So how do you get a real review of a product that your considering investing a lot of money in? Go to a conference or users group for the software, find an admin who's been using it and take them out to a nice dinner for an off the record review of how the product actually works.
Part of the problem is that a critic can sit down for two hours watch a movie and write a meaningful review. This is not possible when it comes to software.
Let me use a real life example: I was an early proponent of Java since my first few interactions with it in 1994 were positive. Only when I was is deep in the bowels of the beast did I start to see the problems: flawed parameter passing model, the "everything has to be an object" religion (which ironically is violated by built in data types), the "you must write a preamble bigger than COBOL's to have a well designed piece of code", the horrible graphics library that if first shipped with, etc.
After that I realized that maybe moving to Java is not such a good idea after all. I think the popularity of C#/Haskell/Scala/C++11/Python are a result of this realization.
I really miss PC Mag. It was great in its time.
This seems like a bizarre lament: We have plenty of software reviews, for various flavors of software, mostly located on the parts of the internet where people who care would find them(since 'Medal of Warfare 3: Gorepocalypse Now' and 'Oracle Enterprise Resource Dominance Solution 11' are somewhat less similar than a bad summer action movie and an occupational safety training video, they aren't reviewed by the same people or in the same places). Who is the audience for the 'NYT Software Review'? What are their perceived interests? What do they not know that they should? Why do they need to get their reviews from a dead tree rather than the internet?
Also, what 'category' would they be shooting for? Movie reviews (implicitly or explicitly) exist largely in the context of assuming that movies are some combination of entertainment and art/culture(exactly what the mix is depends on how highbrow the review is supposed to be). Software, though, does just about everything. What aspects of it is the NYT supposed to care about?
Instead of Siskel & Ebert maybe we need more of a Mystery Science Theater 3000 for software...
Mike Krahoulik and Jerry Holkins, better known as Gabe and Tycho.
I didn't realize the NYT was still being published. Long gone are the days when the NYT drove the news cycle. When has anyone ever gone to the NYT for serious tech info anyway?
Software needs an Edward Deming
This posting is provided 'AS IS' without warranty of any kind, implied or otherwise.
NYT readers couldn't care less about actual software advice. They just want their latest tv series fix. Interesting.
And how many people are scouring the Grey Lady for their technology information? How many people are opening up the print-version of the Grey Lady these days? What business sense would that make for the Grey Lady? The answer to all of those questions is near-zero.
"Your Software Sucks!"
How could television like The Sopranos or Breaking Bad take root in the culture if there was no criticism that discussed it?
I have never, ever, not even once, watched a TV show because of something a critic wrote about it. I started Breaking Bad because all of my friends were raving about it and A Game of Thrones because it was swamping my Twitter feed. In other words, those shows "took root in the culture" because they were good, not because some smarmy ass at the Times blessed them with their indulgence.
I like Dave Winer but have a hard time accepting that he wrote those words. It sounds like something a TV critic would have written to justify their employment.
Dewey, what part of this looks like authorities should be involved?
Guys like Neil Rubenking at PCMag cause more worry among Product Managers and PR than a thousand forum posters. People with microphones are louder than people without, and this sort of thing is no exception.
With the exception of entertainment and the rare 'culturally relevant' application, the vast majority of software is primarily a tool to get its job done, rather than an item of artistic merit in its own right. The New York Times reviews are — for the most part — cultural reviews; they're not the appropriate venue for most software reviews.
With that said, there are those exceptions where one can speak about the artistic or cultural merits of a piece of software, and my strong impression is that the Times has never really stopped speaking about those. The difference between the '80s and today is that at that point, there was so much less understood and so much more that was new in the world of software that a lot of what came out was of cultural relevance and worth talking about on those merits.
...is the only software critic I need.
I would claim software needs Statler and Waldorf instead...
Reviews made sense back when the copy of the software you purchased was the copy you were going to use for at least three years. Now we expect our software to auto update, and perhaps even have a major version change at least yearly. There's no point in having a review, if the item being reviewed has a high chance of being changed by the time the consumers are going to get their hands on it.
No, he's commenting on the bias of the NYT. That should be completely obvious from his comment.
I guess I shouldn't be surprised; most conservatives are so hopelessly gullible and idiotic they actually believe Onion articles are real.
After that I realized that maybe moving to Java is not such a good idea after all. I think the popularity of C#/Haskell/Scala/C++11/Python are a result of this realization.
Yeah, good call there, Alomex. Java's been a real failure and programming languages like Haskell have taken off like a rocket.
<rolls eyes>
Right, because majority opinion has always been a reliable way to judge the quality of something, e.g. bell bottom pants, mullets, Justin Bieber, George W. Bush.
How could television like The Sopranos or Breaking Bad take root in the culture if there was no criticism that discussed it?
I know you want to feel like your job has had some kind of impact on the world but, especially with regard to entertainment media, the quality of the work itself is what allows it to "take root in the culture." Critiques and critics are entirely unnecessary unless the whole audience is full of sheep incapable of thinking even a single thought for themselves. In fact, critics most likely hurt things more than they help, giving undue praise to inadequate things and likewise entirely too much hate to things which only slightly fly under the "good" line. For example, Siskel and Ebert would probably tear a lot of bad movies apart, but the fact still remains that bad movies can still be entertaining and enjoyable.
What would be more useful than something like that would be a group or panel that investigates new software applications to determine which ones are direct copies of other existing applications.
No, it needs a Ralph Nader. Start making software companies legally liable for the proper operation and security of their products.
Time for the software business to grow up and start getting professional.
I guess I shouldn't be surprised; most conservatives are so hopelessly gullible and idiotic
your bias is showing...
Remember-the majority *didn't* vote for GW Bush.
They did the second time around: 50.7% of them.
So I am a developer from Romania (we invented the Artifical Blood, remember?) putting up an OSS project on berlios.de or sourceforge fro the entire wrold to see, use and improve etc but I need give some sort of liverage to some liberal arts, tech illiterate (or at least software illiterate) writer pandering to US centric masses? No thanks. It is not like NYT is good enough to be a news paper but some how should have a say over stuff they have no clue about.
True, most really good games won't give you much of an idea in the first two hours. However many of them can at least tell you "wow, I want to keep playing!" versus "this seems kind of buggy and the dialogue is simplistic."
Don't worry, I'm also biased against many liberals, especially the ones who are fans of Obama.
I like Dave Winer but have a hard time accepting that he wrote those words. It sounds like something a TV critic would have written to justify their employment.
Serious question: why do you like Dave Winer? I don't think you've been paying much attention if you have a hard time accepting he wrote that; he's been saying equally dumb things on a huge variety of topics for ages and ages. He's got his head shoved way up his own ass, too -- thinks he was God's gift to programming and/or the Internet and/or humanity. There are very few people out there as aggressively clueless and egotistical as Winer.
But conservatives are hopelessly gullible and idiotic. It doesn't take a genius to see that. It's an uncomfortable truth about your side.
This doesn't mean no liberals are gullible and idiotic. There are plenty there too. But right now the conservative movement in the US is defining itself by its collective gullibility, idiocy, and mendaciousness. Seriously, we're supposed to think that a government shutdown and threatening debt default is reasoned, rational resistance to something which was signed into law a few years ago and has survived a Constitutional challenge that went all the way to the Supreme Court? We're supposed to believe that the resulting damage to the country can be blamed on the side which didn't initiate the shutdown, using perverse domestic-abuser logic? ("Why did you make me so angry I hit you again?! You need to stop doing the things that make me hit you!")
You have to be gullible, idiotic, or a willing tool of shameless liars and demagogues to believe in that crap. You want to make it so that poor people go back to getting denied insurance coverage? Then win more elections so you have enough power in our representative government to overturn the laws you hate, write new ones, etc. That's how the Framers you conservatives profess to respect so much intended our system to work. Until you can collectively start acknowledging that this is how you should seek to accomplish your goals, sorry, you're collectively a bunch of gullible idiots. (And a few liars who are fooling the gullible idiots.)
(hint: lots of us who are this angry at you used to be on your side. Then you went fucking nuts and caused us to wake up.)
I'm surprised no one has mentioned Walter Mossberg. His reviews are influential.
Let Notch from Mojang review code. He always loves poking holes in others code since his code is too holy to be insulted.
Still waiting on that follow-up post from the HardOCP realtime demo interview with that Unlimited Detail tech. "impossible, scam", pfft.
Oh Notch, you kidder you.