MySQL & Open Source Code Quality

dozek writes "Perhaps another rung for the Open Source model of software development, eWeek reports that an independent study of the MySQL source code found it to be "in fact six times better than that of comparable commercial, proprietary code." You can read the eWeek write-up or the actual research paper (reg. required)."

Comments

All that's missing ...

[JSkills]

All that's missing - to go along with the defects per lines of code comparision - is a comparison of features and performance benchmarking to other commercially built database products. Now that would be the complete comparison.

As strong proponent of MySQL, I'd be very curious to see how it stacks up in those regards.

Stanford Checker

[eddy]

Anyone know how this one is faring? Will it ever be released? It's based on GCC, right? How many students can it pass between until it's "distribution"?

The reason I'm asking is because I saw that one member of the team has jumped over to a company called Coverity where one can read:

Originally developed by a team of researchers in the Computer Systems Lab at Stanford University, Coverity's patent-pending source code analysis technology successfully detected over 2000 bugs in Linux including hundreds of security holes.

I just think it'd be horrible if they used the GPL'ed GCC to develop their methods (having access to a full portable compiler onto which to do research and development is hardly a "small thing"), and then lock these same methods away from the community.

I'm grateful for their work on checking linux, but really... this smells bad, IMHO.

(If you don't know what I'm taking about, don't assume it's off-topic, okay? The Standford Checker is a related topic to the Reasoning analysis of MySQL, and I'm not sure we'll ever have a _better_ fitting topic to discuss this)

Re:Stanford Checker

[Error27]

I wrote a similar tool to the Stanford Checker called smatch.

I post the bugs and stuff that it finds on kbugs.org. The most recent kernel that I've posted is 2.6.0-test11.

One thing that I was working on a couple weeks ago was invalid uses of spinlocks. Here are my results from that. I found quite a few places that don't unlock their spinlocks on error paths etc.

As John Carmack put it...

[rafael_es_son]

The main difference between open and *MOST* closed code is the fact that the early release of closed code means mucho mas money to corporate pigs and dogs, thus, proper requirements analysis, design, coding and testing are usually pummeled in the name of happy-go-lucky capitalism. "It will be ready when it is ready." -Carmack "I love America!" -Murphy

Re:Duh!

[I8TheWorm]

I've used mySQL, Oracle, MS SQL, DB2, and MSDE. I'm not sure I get your comment about MS SQL server. Like any other RDBMS, a little performance tuning goes a long way. As a matter of fact, until Oracle's release of 10g, MS SQL beat all commercial offerings in the TPC benchmarks.

MS has a buggy os and an awful model for business practice, but I think MS SQL server is a fairly nice offering. It's too bad it only runs on Windows servers though.

Must have been baaad commercial code then..

[jordan]

Because there are portions of the MySQL code that are just painful to look at.

Take for instance the part that takes as input the key index size and calculates internal buffer sizes. The option's size is an unsigned long long, but they cast it to an unsigned long all over the place, do in-place bitshifting on the cast (and cause it to wrap -- try specifying 4G for your key index sometime and you'll get 0), and the quality of code in that case is just painfully horrible to look at or even figure out what it's doing.

I could only shudder to think what the quality of the commercial product looked like, in comparison. Hell, I'll have nightmares if I consider the quality of MySQL++ as a comparison..

--jordan

Re:Debatable scale

[Zathrus]

Faults per 1000 lines of code may seem like a valid scale, but I think it is indicatory at best, not proof.

It's actually a really miserable scale because of your 3rd point. If they ran the code bases through something like cindent and standardized the code formatting and removed all comments and whitespace then it's a somewhat more valid comparison. I didn't look at the actual research paper -- maybe they did. Odds are, your other two points are valid though.

Additionally, they only say that the commercial code is "comparable". What does that mean (again, maybe answered in the paper)? Do they have roughly the same features? Are the query optimizers of roughly the same quality? Do they support the same platforms? I can't think of a major commercial database that doesn't exceed MySQL in all of these areas (ok, excepting SQL Server which fails on the 3rd only). Maybe it was a minor player in commercial databases. Dunno.

These are the kinds of points that are raised when someone bashes OSS. There's no reason that they shouldn't be raised when the inverse is true as well. MySQL has progressed nicely and is worthy of consideration for light to moderate database loads now, I don't question that. All I'm saying is don't take things at face value.

So, small victory, but the race goes on.

The nice thing is that this is small and succinct -- it's suitable for showing to upper level management. That's a big win IMHO -- because normally the text bites they read are biased against free/open software.

Re:If you would RTFA...

[Dun Malg]

they quantified it by dividing verified defects by lines of code.

Problem with that is that it assumes the same "code density". Granted, it's probably not going to differ by a factor of six, but remember the old question about programmer productivity:
who's more productive: the coder who solves a given problem with 100 lines of code written in one hour, or the coder who solves it with 10 lines in two hours?

I mean, simple stuff like doing this:

bool function(int i);
main(void)
{
int i;
if(function(++i))
//blah blah blah
}

...instead of:

bool function(int i);
main(void)
{
int i;
bool foo;
foo = false;
i++;
foo = function(i);
if(foo)
//blah blah blah

}

...will give you a threefold difference in line count (specifically counting lines in the main() function). Throw in an identical line using malloc in each, both forgetting to free it later, and you've got a "bug density" of .33 for the former, and .14 for the latter. Heck, you could have two un-freed malloc's in the latter an it'd still only be at .25! I'm not saying the study is wrong-- I'd rather have the code out where I can see it, no matter WHAT the "bug density"-- I'm just saying that I wouldn't take any statistic that is derived using "lines of code" as a variable as a serious, hard number.