Rating System for Open Source Software

prostoalex writes "Carnegie Mellon University, Intel and SpikeSource are launching a rating system for open source software, New York Times says. OpenBRR 'is being proposed as a new standard model for rating open source software. It is intended to enable the entire community (enterprise adopters and developers) to rate software in an open and standardized way.'"

Comments

Oh No!

[AAeyers]

This could be hurtful! Everyone should be a winner!

Think of the children!

FUD from the NYT

[Catamaran]

The author of the article talks as though evaluating software objectively were a problem unique to adopters of Open Source:

Free software, despite the price, can be confusing and costly for corporations to use. A few freely distributed programs, like the Linux operating system and the Apache Web server, have become well known, but most are still unproved.

A more simple and accurate statement would be, "Software can be confusing and costly".

Re:FUD from the NYT

[1000101]

"A more simple and accurate statement would be, "Software can be confusing and costly"."

The entire article is about open source software, not all software. His statement is valid.

Re:FUD from the NYT

[timeOday]

Hah! The main difference is that with OSS, at least you can evaluate the software if you choose to spend the time to do so, without paying a big fee and agreeing to a ridiculous license. Furthermore, with commercial software there is more incentive to lie and exaggerate the software capabilities.

I don't blame the NYT, but I'd love to hear the rationalization for limiting the rating system to OSS. I know I'd love to rate a few of the commercial applications I've used. Some shareware sites do have ratings systems of a sort, but it's a sticky issue all around. Even comparing two cpu's that implement the exact same instruction set is something we can't all agree on!

About freakin' time.

[generic-man]

If you execute a specific elisp file at a key time, emacs displays a very graphic mini-game involving Richard Stallman. As a responsible parent, I want to make sure that this sort of thing isn't seen by my children when I'm not watching them.

I applaud this rating system and wish it well.

Re:About freakin' time.

[Infernal Device]

If you execute a specific elisp file at a key time, emacs displays a very graphic mini-game involving Richard Stallman.

I would pay money not to see that.

My eyes! The burning!

Good idea...

[msmercenary]

I can't count how many times I've googled for some OSS to do a specific task, and found what I wanted only after installing and uninstalling four programs that were buggy, slow, didn't have the features I wanted, or simply wouldn't build/install.

On the flip side, there has always been an inherent and objective rating system for the quality of non-free software -- At what price will enough people purchase it to make it worth producing?

Moderating....

[fembots]

The rating system has 11 categories, including Normal, Offtopic, Flamebait, Troll, Redundant, Insightful, Interesting, Informative, Funny, Overrated and Underrated.

Each category is to be rated -1 to 5.

There will also be filtering tools so a potential corporate user can specify its most important considerations.

As good as CMM?

[orthogonal]

Carnegie-Mellon these are the guys who love to quantify the unquantifiable.

Didn't they also give us the "Capability Maturity Model"? I've seen organizations race to get to CMM-3 or CMM-4, and it's all been a joke.

A bunch of highly paid consultants tell everyone a new way to count beans ("under CMM, we group the beans starting from the right, not the left....").

Promises are made about code auditing, but once the CMM level has been awarded (usually by highly paid consultants who just happen to work with the highly paid consults who "mentored" the company's CMM training), all tat's actually done is that the people doing the real work of writing software are regularly distracted by a clown with a check-list and a clipboard.

Carnegie-Mellon continues to have a fetish for quantifying and for creating check-lists, and middle management continues to have a fetish for anything that allows them to quantify (even spuriously), because it takes the risk and bother out of their jobs.

Middle Manager: "The WordPerfect Project only got a 3 on the Carnegie Mellon software score, but the Clippy Project got a 5! So, it's perfectly safe for me to decide that to disband the WordPerfect Project and devote its resources to the Clippy Project. (And if it turns out later that was a bad decision, they can't fire me, because I relied on hard numbers generated by a known process!"

1 to 10

[b17bmbr]

1: absolutely horshit. stuff i wouldn't use if paid a million dolars.

10: barely usable, requires constant tweaking, stuck at version 0.9.3, crashes occasionally, and requires three new libraries each upgrade which break other applications.

Re:The problem with rating software...

[broller]

If a user can't figure out how to do X, then X might as well not even be implemented... it amounts to the same thing.

I absolutely agree if we change it to "If all users can't figure out how to do X" but saying "a user" leaves no room for a learning curve. I think it's completely valid to expect that some software will be written that is not necessarily meant for the novice user.

I can see how you might feel that way..

[msimm]

But the truth is propriety software is quite well reviewed (there's an entire industry who makes it their business to review and recommend commercial software, usually somewhat usefully).

Reading reviews of you're favorite Windows Antivirus software or researching an enterprise class database package will turn up a wealth of infomation (of course you still need to dig into it and make the final decision, but some things simply can't be helped:).

OSS software is comparably a total mess, with only certain major projects (and not surpisingly usually projects with some sort commercial support, i.e. apache, mysql, sendmail, etc, but the water gets pretty muddy quickly).

And aside from all those mainly concrete (maybe to you and I anyway) worries there are other concerns when reviewing OSS software for deployment in a business/production environment: support, boss appeal; someone has to sign off even if the software is free, that the software is mature/will meet or exceed your needs and that (if you decide to leave the company) its reasonably well supported (so someone that comes in and doesn't know the particular software has a reasonably good chance of configuring and maintaining it).

Those crazy business people.

Re:I can see how you might feel that way..

[Kludge]

But the truth is propriety software is quite well reviewed

The truth is that some of it is and some of it isn't. And a good number of the "reviews" are done by hacks who are getting paid by the developers for it.

Reading reviews of you're favorite Windows Antivirus software or researching an enterprise class database package will turn up a wealth of infomation ... OSS software is comparably a total mess, with only certain major projects

To the contrary. I can type the name of almost any OSS project into Google and get lots of posts by people who have actually used the software. A lack of hits indicates software that's probably not mature.

Or I can just search the software that I have installed on my computer. Suse has no problem picking out thousands of great software packages for me to use.

... support, ... supported ... maintaining

Support is way over rated. I have never ever gotten good support from any company that produced anything. The only company worth what I've seen them paid was a little 4-5 person Linux consulting/computing biz we contracted. One of the reasons they do so well is that they have the ability to grab and utilize whatever code does the job.

Re:I can see how you might feel that way..

[Tim C]

Support is way over rated. I have never ever gotten good support from any company that produced anything.

There are two aspects to support; actually getting support, so that your problems are fixed quickly and efficiently is one of them. The other, and often just as important to some people, is being able to point the finger at someone else.

It's arse-covering - being able to say "Yes, there's a problem, yes, I recommended the software, but it's not our fault, the support firm is dicking us about - we've paid them, they should fix it!" is a lot better than "Yes, there's a problem, yes, I recommended the software, we're doing our best - sorry"

Generally speaking, managers and clients don't want to hear that in the event of a problem there's a whole bunch of people on the internet who are bound to be able to help. They want to know that someone with the appropriate expertise is going to be dedicated to fixing the problem ASAP. Whether or not that's actually what happens is often beside the point. (Generally if it doesn't, it just means that next time you'll get support from someone else instead)

A vignette for illustration...

[kimanaw]

SCENE: The year is 2007. Staff architect Erst Wile Programmer enters the office of low level IT manager P.H.B. Risq'averse.

EW: Boss, I think I've found a great little open source perl script to solve our database reporting issue...

PHB (turns to PC and begins typing): That was "p-e-a-r-l", right ? Sorry, SpikeSource(TM) doesn't report a BRR for it...

EW confused look: Er, no, "p-e-r-l"...

PHB looks anxious, types some more: OK...Perl is OK, but whats the module ?

EW: "Super::Califragilistic"

PHB typing furiously: OK, its listed, but the BRR is only 11.23065. Sorry, our required min BRR is 27.83409.

EW: Wha...?

PHB: BRR. You know, number of downloads, numer of reported errors, number of reporting users, that sort of thing.

EW: But its only been out about 5 months, and its only really relevant to this particular problem we've got...

PHB: Look, E-dub, we have to follow practices and procedures. If we don't, CEO's go to jail, and the insurance company drops us like a bad case of clap. And one requirement is, "Open source software must be a minimum BRR of 27.83409".

EW: But what about...

PHB looks concerned and sympathetic: Look, E-dub, I'd love to help ya, but frankly, I'm not even certain you're allowed to download this software; I'd hate to have to report you to Network admin, so why don't we just pretend this conversation never happened ?

Thats the road BRR leads us down.

I'd love to believe that the BRR was(a) a useful metric that would (b) be used intelligently, but 2.5 decades of experience leads me to believe otherwise.

Furthermore, we've given them the damn source! How about doing something actually useful, like running an automated metric on it (e.g., McCabe testing), or maybe just looking at it ? Apparently, BigBiz isn't satisfied with finding money in the street anymore, they expect someone to pick it up for them, too.