Slashdot Mirror
Another Look At Mozilla's BugFix Rate
An anonymous reader writes "Washingtonpost.com's Security Fix blog has published the results of a look back at three years worth of critical patches from Mozilla, and found that Mozilla typically ships updates for critical flaws in about three weeks, though in more than a third of the cases it pushed out a fix in ten days or less. The data comes just a few weeks after The Post published data from a similar study that found Microsoft averaged 130+ days to fix critical flaws. Slashdot also covered that study in a previous post."
No problem, they just made the shovels REALLY HEAVY, so they only had to make a few of them.
Software metrics are very slippery things.
I'd love if Firefox didn't take up 256 megs of ram with 5 tabs open. Is that something we can get fixed soon? That'd be great. All I want is for Firefox to take less memory than Azureus. I only have adblock and bugmenot, so it's not extensions causing the problem.
Not a Twitter sockpuppet... but I wish I was.
I'm not a statistician, but the average is sometimes a poor way to describe data. It's often useful to look at modes, standard deviations, and so on.
For example, the standard deviation for 2005 had Microsoft with a 80.87 stdev and Firefox with a 97.5 stdev.
Firefox had one flaw that took 674 days to fix, nearly twice the max of Microsoft's 357 days. Does that make up for such a larger average? Dunno. I suppose you could look at the issue and decide for yourself.
Averages are important, but it's not always the single most important thing to consider.
FTA: In recognition that 2004 was most likely the first year in which a significant share of the company's new user base was coming from Windows users, Security Fix based each of "date patch issued" date for 2004 and 2005 on the release date of the next product update that incorporated the fix for that critical security vulnerability -- not the date on which a fix was available to developers. For 2003 critical Mozilla flaws, Security Fix relied on the times listed in the "date fixed" field for each critical flaw listed on the "Older Vulnerabilities in Mozilla Products" page.
:)
So if you cut the days-to-fix time up by year, for 2004, the avg is 65 days. In 2003 they used the "fixed" date in the bug DB. For 2005, its 37 days, and for 2004/2005 combined, its 42 days.
The 2004/2005 # is the interesting one, because that measures how long until the patch actually makes it into a shipping build. The availability date of source-code patches is irrelevant because most organizations are not equipped to deal with them; this is especially the case with firefox!
None of this is an excuse, however. As an MS employee, the data on our speed of patch delivery is pretty upsetting - the numbers are much worse than I would have expected. They're so bad that I am suspicious and wonder if there isn't some deeper story somewhere, but in any case, I'd like to think the maximum time anyone would have to wait would be ~1 month (flaw reported on the wednesday after "patch tuesday"), but according to this data we're not hitting that at all. I can't speak for the IE or the MSRC teams but those numbers really appear to suck. Sorry about that, everyone
My opinions are my own, and do not necessarily represent those of my employer.
It was well-known in the late 1960s that it was not a good idea to use unbounded copying. Indeed, that is before C, let alone strcpy(), was first implemented.
.NET languages. Their use of a virtual machine often brings in more problems than it solves, in addition to using poor garbage collection policies and schemes.
Memory-related exploits should never be an issue. A proper garbage collection system, like that offered by most implementations of LISP, Smalltalk and OCaml, for instance, eliminates memory leaks. It lets the developer focus on writing solid code, without having to worry about mundane issues, without affecting performance to any extent.
And no, don't use a GC'ed language like Java or the various
Cyric Zndovzny at your service.
However, other bugs simply aren't fixed. For about 3 years many, many people have reported the CPU hogging bug which is unique to Firefox and Mozilla browsers. For a small example of the reports of problems see Firefox is the most unstable program in common use.
Now the problems are beginning to be reported in technical magazines, newsletters, bloggers, and even the mainstream media.
Under the conditions mentioned in the bug reports, I'm not able to make the CPU hogging bug fail; it is always there. I've tried Linux, Windows XP SP2, and Windows 98 SE. I've tried Intel and Via chipset motherboards. For about 3 years, in all versions, the CPU and memory hogging bug has always been there. Firefox version 1.5.0.1 is worse than Firefox version 1.5, and those versions are worse than earlier ones. This is with a clean profile and no extensions except DOM Inspector, which is a menu choice on the installation program.
In 3 years, I've never had any evidence that any Firefox or Mozilla developer has reproduced the conditions that cause the problem.
The problem with Firefox and Mozilla developers not fixing difficult bugs seems to be a social one, not primarily a technical one. The developers keep asking for the problem to be made easier, but it appears to me that there is already plenty of evidence that would allow further investigation.
Perhaps the developers do not understand that there is a class of bugs that can only be found using the methods of scientific research. Many people like programming, but only people who accept the biggest challenges truly have programming in their hearts and minds:
Three biggest challenges of programming
Here are programming's three biggest challenges. Coding is relatively easy. It is these challenges which separate a true professional from an average programmer:
Instead there are excuses:
Mozilla Top 12 Excuses
Top 12 things Firefox and Mozilla developers say about those who report difficult bugs, collected during the last 3 years:
Then I'll flat out say it - the code base is sloppy.
You've got a mixed up mash of reference counted objects (XPCOM) interacting directly with garbage-collected objects (JavaScript) and this causes no end of memory leaks. Mozilla is designed from the ground-up to leak memory, since it mixes two different memory allocation schemes in its core.
Of course, that's not the end of it. There are a TON of support libraries that don't even use XPCOM *OR* JavaScript. So now we've got JavaScript GCed objects, reference-counted XPCOM objects, objects that use a custom malloc/free (so you better remember to call the Finish function!), C++ libraries that use new/delete, and - wow.
It's just a huge mishmash of memory techniques that's practically impossible to unravel. What does the PR hashtable implementation use to free memory when it's deleted? DOES it even free memory? I have no idea, because the documentation is completely messed up. I sure hope it deletes its contents, because my code doesn't even try.
Then there are the assertion errors. Assertion errors in released code?! WTF?! Starting up a fresh Firefox build with debugging enabled brings up several assertion errors. For added fun, closing windows also causes assertion errors. WHY are there still assertion errors in release code?! Especially 1.5 code?!
Anyone who's taken more than 5 minutes trying to deal with Mozilla from the C++ side can tell you that the system is a massive poorly documented mess. C, C++, JavaScript, all merged together to create a huge cludge of a browser. You peak under the hood, and you become amazed that the thing starts up, let alone runs. I suppose I'm glad that Firefox only leaks around 100MB/day, and hangs for a good 30 seconds after closing a window - because I wouldn't expect it to even run after looking at the code base.
If you want constructive critism, here are things I haven't been able to figure out after a month of playing with the code:
How are you supposed to use the "new" embedded string library?
How are you supposed to use the "old" string library?
How do you know which one you're using in the first place?
How do you create "thread-safe" XPCOM objects?
Does the PR hashtable implementation attempt to clean up its contents when it's destroyed? If so, which memory allocation scheme are you supposed to use for it?
Links to the MDC articles on string handling and hashtables will not be appreciated, because I already looked over those.
You are talking about things that _you_ want, and I can even mostly agree. However it has been proven repeatedly that only a very small minority are willing to put security/quality before other design trade offs ... things might well get better in the next 10 years due to python/C#/etc. not having as significant downsides. But even so we are going to be stuck with C based programs for a long time, and there are still very few people who want to pay to do even the minimal fixes.
Again, I'm not saying I wouldn't like to do that trade off, indeed I wrote my own secure Web server (with a monetary guarantee you won't be owned) because I didn't like apache-httpd ... but there just isn't the general buying power to bring secure software out of the niche.
ustr: Managed string API with ave. 44% overhead over strdup(), for 0-20B