MySQL & Open Source Code Quality

dozek writes "Perhaps another rung for the Open Source model of software development, eWeek reports that an independent study of the MySQL source code found it to be "in fact six times better than that of comparable commercial, proprietary code." You can read the eWeek write-up or the actual research paper (reg. required)."

Comments

Six times better?

[pyite]

Six times better? I didn't know it was possible to quantify code quality in that matter. Interesting.

Just wait...

[cableshaft]

...until I release my MySQL source code to the open source community. Then that 6x multiplier will drop down to 2x.

Yeah, it's really that bad. Gets the job done, though. Hell to maintain. Probably would've helped if I documented any of it.

Maybe I should read that Code Complete book I keep meaning to read sometime.

Don't rest on your laurels.

[grub]

Perhaps another rung for the Open Source model of software development

Uhh... no.

It's is a glowing report for this particular open source project but that brush shouldn't be used to paint all open source. That will just lull open source developers into a false sense of euphoric contentment. Code quality didn't get this far by having a fixed target, that target should be a carrot on a stick that will never quite be reached.

Measurements

[Stiletto]

Undoubtedly()
{
when();
you = measure(quality);
in.defects();
per->lines_of(code, anyone);
can = write(good, solid, code);
}

Re:Measurements

[Walterk]

Post:2: warning: return-type defaults to `int'
Post:2: In function `Undoubtedly':
Post:3: warning: implicit declaration of function `when'
Post:4: `you' undeclared (first use in this function)
Post:4: (Each undeclared identifier is reported only once
Post:4: for each function it appears in.)
Post:4: warning: implicit declaration of function `measure'
Post:4: `quality' undeclared (first use in this function)
Post:5: `in' undeclared (first use in this function)
Post:6: `per' undeclared (first use in this function)
Post:6: `code' undeclared (first use in this function)
Post:6: `anyone' undeclared (first use in this function)
Post:7: `can' undeclared (first use in this function)
Post:7: warning: implicit declaration of function `write' Post:7: `good' undeclared (first use in this function)
Post:7: `solid' undeclared (first use in this function)
Post:8: warning: control reaches end of non-void function

If you would RTFA...

[Theatetus]

...they quantified it by dividing verified defects by lines of code. MySQL had 0.09 bugs/KLOC while the "commercial" defect density was 0.53 bugs/KLOC. (Their use of the term "commercial" confused me since MySQL is, after all, a "commercial" project, just an open-source one.)

Re:If you would RTFA...

[pyite]

"Defect" is also a difficult term to define. Some errors are much worse than others. It's not all about numbers, folks. Don't get me wrong, I'm not saying that MySQL isn't a great product. I just get skeptical when I hear things talked about in terms of "better" and "best."

Re:If you would RTFA...

[SonicBurst]

Not only is it hard to define defect (and it is very obvious that some defects are worse than others), but this code review sounds like it only spots "grammatical" or style errors in the code. It doesn't sound like it could find a defect in an algorithm implementation or logic. To me, these are where the true defects are, in the logic/reasoning breakdowns.

Re:If you would RTFA...

[rembem]

0.09 vs 0.53 bugs/KLOC can also mean mysql has six times the amount of code per line, compared to an average "commercial" program. Those numbers should be divided by a code-density-factor.

Re:If you would RTFA...

[B'Trey]

I'm not sure what you mean by "grammatical" or style errors. If you're talking about syntax errors, those should prevent the code from compiling. I'm not aware of how coding style can be an error (unless you're programming in Python).

The specific errors in MySQL were dereferencing null pointers, failure to deallocate memory (memory leaks), and use an uninitialized variable. These aren't the only bugs that such an analysis can find; they're the ones that were found in MySQL. And they're definitely errors in logic.

Certainly, there are bugs that such an analysis can't find. If you define PI as 3.15, your calculations are going to be off. If you create a function to determine the circumference of a circle as 2 * PI * Diameter, you've got a bug. I suspect that those are the types of errors in logic that you were referring to, and you're right that they will not be caught by a code analysis. However, that doesn't mean that comparing the frequence of the errors that CAN be caught between two programs is an invalid act. From my experience, programmers who make fewer of the former errors also make fewer of the latter. Analyzing catchable errors is a good metric for the frequency of errors in a given source tree, even if all errors aren't caught.

Re:If you would RTFA...

[Tassach]

No defects != good software.

A flawless implementation of a crap algorithm is still crap. I don't care if your bubble-sort routine has no memory leaks or buffer overruns; it still scales O(N^2). Likewise, a so-called "database" which does not implement key features like transactions and stored procedures is fundamentally flawed even if there are zero coding errors.

MySQL may be well-written, but it's still a piece of crap by the standards of any professional DBA.

Re:If you would RTFA...

[Dun Malg]

they quantified it by dividing verified defects by lines of code.

Problem with that is that it assumes the same "code density". Granted, it's probably not going to differ by a factor of six, but remember the old question about programmer productivity:
who's more productive: the coder who solves a given problem with 100 lines of code written in one hour, or the coder who solves it with 10 lines in two hours?

I mean, simple stuff like doing this:

bool function(int i);
main(void)
{
int i;
if(function(++i))
//blah blah blah
}

...instead of:

bool function(int i);
main(void)
{
int i;
bool foo;
foo = false;
i++;
foo = function(i);
if(foo)
//blah blah blah

}

...will give you a threefold difference in line count (specifically counting lines in the main() function). Throw in an identical line using malloc in each, both forgetting to free it later, and you've got a "bug density" of .33 for the former, and .14 for the latter. Heck, you could have two un-freed malloc's in the latter an it'd still only be at .25! I'm not saying the study is wrong-- I'd rather have the code out where I can see it, no matter WHAT the "bug density"-- I'm just saying that I wouldn't take any statistic that is derived using "lines of code" as a variable as a serious, hard number.

Re:If you would RTFA...

[Sxooter]

Sorry, but until MySQL has a mode where ALL tables are transaction safe, or at least throws an error when you try to create a fk reference to a non-transaction safe table, it's transactions are too prone to data loss due to human error.

It's a good data store, but the guys programming it have to "get it" that transactions can't be optional in certain types of databases, and neither can constraints, or fk enforcement.

MySQL has a tendency of failing to do what you thought it did, and failing to report an error so you know. This is a legacy left over from being a SQL interpreter over ISAM files. It makes MySQL a great choice for content management, but a dangerous choice for transactional systems.

Re:If you would RTFA...

[scrytch]

If only it were MySQL just lacking features that would, after much mudslinging at the ideas themselves, be grudgingly retrofitted into a new table type. MySQL's brokenness goes deeper than that.

MySQL's attitude toward data integrity can be summed up as "if the constraint can't be satisfied, do it half-assed anyway". I find myself having to write application code to manage data integrity with MySQL, something I can take for granted with a real database.

Re:"6 times better"

[dcordeiro]

I agree with you that you can't simply measure quality but...
If you just RTFA, you'll see that is not "6 times better" but "6 times less bugs found then the average on commercial products"

The only thing wrong in the article:
They should replace the term "commercial" with "closed source", because Mysql is also a commercial product and what makes it different is the open source model.

All that's missing ...

[JSkills]

All that's missing - to go along with the defects per lines of code comparision - is a comparison of features and performance benchmarking to other commercially built database products. Now that would be the complete comparison.

As strong proponent of MySQL, I'd be very curious to see how it stacks up in those regards.

Stanford Checker

[eddy]

Anyone know how this one is faring? Will it ever be released? It's based on GCC, right? How many students can it pass between until it's "distribution"?

The reason I'm asking is because I saw that one member of the team has jumped over to a company called Coverity where one can read:

Originally developed by a team of researchers in the Computer Systems Lab at Stanford University, Coverity's patent-pending source code analysis technology successfully detected over 2000 bugs in Linux including hundreds of security holes.

I just think it'd be horrible if they used the GPL'ed GCC to develop their methods (having access to a full portable compiler onto which to do research and development is hardly a "small thing"), and then lock these same methods away from the community.

I'm grateful for their work on checking linux, but really... this smells bad, IMHO.

(If you don't know what I'm taking about, don't assume it's off-topic, okay? The Standford Checker is a related topic to the Reasoning analysis of MySQL, and I'm not sure we'll ever have a _better_ fitting topic to discuss this)

Re:Stanford Checker

[Error27]

I wrote a similar tool to the Stanford Checker called smatch.

I post the bugs and stuff that it finds on kbugs.org. The most recent kernel that I've posted is 2.6.0-test11.

One thing that I was working on a couple weeks ago was invalid uses of spinlocks. Here are my results from that. I found quite a few places that don't unlock their spinlocks on error paths etc.

Debatable scale

[Basje]

I do believe that Open Source is better than proprietory. Faults per 1000 lines of code may seem like a valid scale, but I think it is indicatory at best, not proof.

* It does not take into account the design of the software. This is often as important as the actual quality of the code.
* It does not take into account the kind of errors. This is related to the first, but a buffer overflow that allows root access is worse than a failed instruction.
* It does not even take the length of lines into account. Shortening the lines could lower the number, without actually changing anything.

So, small victory, but the race goes on.

Re:Debatable scale

[ebuck]

Good points, and I agree.

Also if "lines of code" are going to be part of any code comparisions, then a standard should be propsed that does (at a minimum) the following:

1. Formats the code consistently. We don't want one project to have more lines of code (and therefore less bug density) because they put a brace or parenthesis on a separate line while others do not.

2. Strip the comments. Someone could decrease bug density by heavy, heavy commenting. Comments are a vital part of coding (and more usually is better), but they have no impact on the bugginess of the code.

3. Format conditionals, blocks, and function calls consistently, or better yet, ditch the line counting and count bugs per (function call, assignment operation, operation).

Lines are easy to count, but they hold so little meaning in determing code quality.

Re:Debatable scale

[Zathrus]

Faults per 1000 lines of code may seem like a valid scale, but I think it is indicatory at best, not proof.

It's actually a really miserable scale because of your 3rd point. If they ran the code bases through something like cindent and standardized the code formatting and removed all comments and whitespace then it's a somewhat more valid comparison. I didn't look at the actual research paper -- maybe they did. Odds are, your other two points are valid though.

Additionally, they only say that the commercial code is "comparable". What does that mean (again, maybe answered in the paper)? Do they have roughly the same features? Are the query optimizers of roughly the same quality? Do they support the same platforms? I can't think of a major commercial database that doesn't exceed MySQL in all of these areas (ok, excepting SQL Server which fails on the 3rd only). Maybe it was a minor player in commercial databases. Dunno.

These are the kinds of points that are raised when someone bashes OSS. There's no reason that they shouldn't be raised when the inverse is true as well. MySQL has progressed nicely and is worthy of consideration for light to moderate database loads now, I don't question that. All I'm saying is don't take things at face value.

So, small victory, but the race goes on.

The nice thing is that this is small and succinct -- it's suitable for showing to upper level management. That's a big win IMHO -- because normally the text bites they read are biased against free/open software.

6 times better?

[kjba]

I don't see how you can make the statement that MySQL is 6 times better than the proprietary code from the facts that the defect densities are 0.09 and 0.54 per 1000 lines respectively.

This just looks like some quasi-scientific statement, trying to express things as a number that really don't fit such a representation. For example, as the number of defects decreases, it becomes increasingly more difficult to find the ones that are left. And is code that contains no bugs at all infinitely much better than code that contains a single bug which hardly ever occurs?

As John Carmack put it...

[rafael_es_son]

The main difference between open and *MOST* closed code is the fact that the early release of closed code means mucho mas money to corporate pigs and dogs, thus, proper requirements analysis, design, coding and testing are usually pummeled in the name of happy-go-lucky capitalism. "It will be ready when it is ready." -Carmack "I love America!" -Murphy

Re:Lines of Code?

[nojomofo]

I'm under the impression that most "bugs" in software (certainly most bugs in my code) aren't bugs like these in the article (null dereferences, uninitialized variables, etc), but they're algorithm bugs. As in, there's a subtle interplay between different parts of complicated algorithms that can be easy for programmers to miss. Those types of bugs are going to be much harder to find, and certainly not going to be found in analysis such as this one.

Re:New unit ?

[pragma_x]

Since we're measuring Defects per 1000 lines, perhaps calling them "Gates" or "Ballmers" might be more appropriate.

Re:Duh!

[I8TheWorm]

I've used mySQL, Oracle, MS SQL, DB2, and MSDE. I'm not sure I get your comment about MS SQL server. Like any other RDBMS, a little performance tuning goes a long way. As a matter of fact, until Oracle's release of 10g, MS SQL beat all commercial offerings in the TPC benchmarks.

MS has a buggy os and an awful model for business practice, but I think MS SQL server is a fairly nice offering. It's too bad it only runs on Windows servers though.

Must have been baaad commercial code then..

[jordan]

Because there are portions of the MySQL code that are just painful to look at.

Take for instance the part that takes as input the key index size and calculates internal buffer sizes. The option's size is an unsigned long long, but they cast it to an unsigned long all over the place, do in-place bitshifting on the cast (and cause it to wrap -- try specifying 4G for your key index sometime and you'll get 0), and the quality of code in that case is just painfully horrible to look at or even figure out what it's doing.

I could only shudder to think what the quality of the commercial product looked like, in comparison. Hell, I'll have nightmares if I consider the quality of MySQL++ as a comparison..

--jordan

Re:Duh!

[James Thompson]

Need a particular reason? Take your pick. http://sql-info.de/mysql/gotchas.html

Re:Duh!

[pyite]

Up until recently, MySQL had no transaction or atomic operation support. As such, you need to write application code to trap problems. Whereas with Oracle, when you run an atomic operation, you know without certainty whether the query failed in its entirety. I also believe stored procedure support is somewhat lacking in MySQL (however, there is that new Java function support). The MySQL 3 tree does not enforce constraints which is something most essential for data integrity. MySQL does not have subrow locking, whereas enterprise databases do. Once again, MySQL is great. I use it. However, it is not enterprise.

Re:Slashdot is a bad commercial for MySQL.

[pegr]

"but the Slashdot database regularly becomes confused, such as posting a comment to the wrong story"

That's not the db... around here, we call them "trolls"...

;)

FUD

[Kenneth Stephen]

This is proof positive that the marketing engine has started churning in the Linux / Open Source arena. The quoted statistics are meaningless. Here are is a short list of things (in no particular order) that are wrong with this "study" (who paid for it anyway?):

Lines of code is meaningless as a reliable measure of anything. The most this number can be used for is for assessing the high level complexity (i.e. simple, non-trivial, or hard) of an application / code construct. It is absolutely pointless to compare two different applications against each other by lines of code. This means that you can say that one is non-trivial and the other is complex or you can say that both are complex, but there is no valid way of determining (by using this particular metric) that one application is more complex than the other. I believe this is the fundamental flaw in this "study".

The study igores capabilities. If application A has feature a, b, and c, and application B has features a, b, c, d, e, f, g, h , is it even meaningful to compare the number of defects detected between applications A and B? And no - normalizing it by lines of code is not valid (see previous point).

Testing methodology : from the defects quoted in the article, it appears as if they "study" did white box testing on MySQL. This is hardly complete. While null pointer dereferences are certainly terrible, I would be also very very concerned about bugs pertaining to SQL capabilites, data integrity, performance, etc. If I go out and do a comparison of RDBMS's for a client, my report wouldnt be complete at all without covering these areas. How come the "study" doesnt mention any of these things?

Lets face it : this is a paid propaganda article by the marketing machinery. Much like Microsoft has done in the past.

It is embarassing to show bad code.

[jhines]

It is really embarassing to have bad code with your name on it, released to the public.

Not only that, but there is a small percentage of coders when presented with an ugly solution to a problem, will pretty it up, just "because". And it is a good way to get known in the OSS world.

Unlike the corporate world, working but ugly code is hidden deeper and deeper, and people go out of their way to avoid it.

Re:MySQL vs. Oracle

[IANAAC]

If you're using any of Oracle's standard feature set, you'll have a tough time converting everything over. Oracle is much, much more SQL standards compliant (what's with MySQL's backticks anyway?). If your applications use stored procedures, triggers, primary and foreign keys, transaction-based recovery/redo, you're looking at a complete rewrite of your apps. Regardless of what database you choose to use, you're looking at at least a partial rewrite, but why complicate matters more than you need to?

Sorry, but my opinion is pretty strong on this. Going from anything Oracle to MySQL is NOT trivial.

Re:Don't generalize!

[SuperBanana]

This "proves" that MySQL is better than commercial offerings. Good.

No it doesn't. It "proves" that on average, by line, MySQL has fewer errors in code. It says nothing of the severity of the errors in either package.

Furthermore- MySQL is not even close to being equal in feature set to almost any commercial DB; replication/backup sucks, it's not ACID compliant, it had no transaction support until recently, no stored procedures, no triggers.

How on earth could you possibly compare it to almost any commercial SQL DB which has all these...and say MySQL is better?

A lot of people knew that.

No, every two bit web designer thinks its the greatest thing since sliced bread, since they think a select w/group+sort is an advanced query. Every professional DBA I've met refuses to work with MySQL and/or hates it, and they can go on for an hour about why. When are you people going to realize that PostgreSQL is so much better than MySQL, save some incredibly risky performance options?

MySQL is awesome! But let's be careful about this story, okay? It's the over-generalization that gives OSS/Linux advocates a bad name ("The Gimp is equivalent to Photoshop!").

But you just said "This proves that MySQL is better than commercial offerings!"

Re:MySQL vs. Oracle

[kpharmer]

Porting between dbms products depends primarily on two issues:
1. usage of vendor extensions
2. usage of standard relational functionality

Generally speaking, if you've minimized #1 in your application you can easily port between Oracle, DB2, SQL Server, Sybase, Postgesql, etc: sure, you could hit some issues with jdbc drivers, and may need to port a few idioms (partitioning for example), but it shouldn't be a killer. But going from any of the above list to mysql isn't suggested: you'll get hung up on #2 (it doesn't support standard SQL or DDL)

Realistically, if I wanted to go to a less expensive product than oracle I'd look down this list:
- db2 (1/3 to 1/2 oracle cost)
- sybase (cheaper than oracle, but dwindling market share)
- firebird (very low cost)
- postgresql (free)
All of the above are mature relational databases that you could port oracle applications from.

But you mentioned 'mission critical'. At this point I'd be very cautious about either postgesql or mysql in a mission-critical role. How important is it to you that you can recover 100% of your data in the event of a database crash? I'd put my money (and career) on db2 or oracle delivering that kind of quality over mysql...

Kinds of errors -- it's Reasoning, Inc. again

[Anonymous Brave Guy]

Not only is it hard to define defect (and it is very obvious that some defects are worse than others), but this code review sounds like it only spots "grammatical" or style errors in the code.

It does indeed sound a bit like that, and with good reason. If you notice, the "indepedent review" was carried out by Reasoning, Inc., and we've heard of them before in these parts.

For the benefit of those who haven't seen this trollfest^H^H^H^H^H^H^H^H^Hstory in its previous incarnations, Reasoning's services spot what some people call "systematic" errors, things like NULL pointer dereferencing or the use of uninitialised variables. As many people note every time this subject comes up, any smart development team will use a tool like Lint to check their code anyway, as a required step before check-in and/or as a regular, automated check of the entire codebase, and so any smart development team should find all such errors immediately. IOWs, it's grossly unfair to compare open and closed source "code quality" on this basis. Any project that has errors like this in it at all isn't serious about quality, and it shouldn't take an external study to point this out.

Serious code quality is not dictated by how many mechanical errors there are that slip through because of weaknesses in the implementation language. Rather, it is indicated by how many "genuine" logic errors -- cases where the output differs unintentionally from the specifications -- there are. Of course, no automated process can identify those, but to get a meaningful comparison of code quality, you'd need to investigate that aspect, rather than kindergarten mistakes.

There are other objections to their principal metric as well. For starters, source code layout is not normally significant in C, C++ or Java, so any metric based on line count is going to be flawed at best. But the big objection is that they're talking about childish mistakes, and comparing supposedly world class software based on childish mistakes isn't helpful (except to dispel the myth that some big name products have sensible development processes).