Another Look At Mozilla's BugFix Rate
An anonymous reader writes "Washingtonpost.com's Security Fix blog has published the results of a look back at three years worth of critical patches from Mozilla, and found that Mozilla typically ships updates for critical flaws in about three weeks, though in more than a third of the cases it pushed out a fix in ten days or less. The data comes just a few weeks after The Post published data from a similar study that found Microsoft averaged 130+ days to fix critical flaws. Slashdot also covered that study in a previous post."
Maybe it's just imagination, but I thought I've been reading these kind of stories on Slashdot since the dawn of time.
Beware: In C++, your friends can see your privates!
While it's important to fix bugs quickly and correctly, perhaps the Mozilla project should take some initiative within the open source community to work on preventing security issues in the first place. They could partner with the OpenBSD project on such an initiative, for instance.
Together they could come up with a development system that focuses on writing solid code. Such a methodology won't prevent all security bugs by any means, but it could go a long way towards vastly increasing the quality of the Mozilla project's software.
Cyric Zndovzny at your service.
Ok, I guess three weeks can be counted in hours, but that's a LOT of hours.
If you need web hosting, you could do worse than here
Funny. IMHO, the speed of the browser peaked a long time ago (0.8 IIRC), and now it's just getting progressively slower over time.
They might be fixing critical security bugs, but they certainly don't seem to be fixing memleaks and such.
"Mod, mod, mod...and another troll bites the dust."
Has anyone collected similar data regarding Opera, OmniWeb, Safari, and other alternative browsers?
If so, how do they compare to Mozilla and Internet Explorer in not only the time it takes to address security problems, but also in terms of the number of incidents per release?
Cyric Zndovzny at your service.
In fairness, everything that I've read about MS's patch cycle indicates that it is a pretty huge undertaking. Joel from http://joelonsoftware.com/ is always going on about have every single code fix/feature addition has to go through a whole bunch of people (several testers, documentation team, etc) before it can be released. If anything maybe Microsoft is a bit too thorough with their patches, in some ways at least.
No problem, they just made the shovels REALLY HEAVY, so they only had to make a few of them.
Software metrics are very slippery things.
To be fair, Microsoft's flaws are alot more serious, so it's only logical they will take longer to fix.
<laugh\>
Big ones, small ones, some as big as yer 'ead!
Give 'em a twist, a flick o' the wrist...
While I hope that this will encourage the average person to download Mozilla, I think that the person who might be swaded by this is already infected and doesn't know/understand what is to be gained.
I'd love if Firefox didn't take up 256 megs of ram with 5 tabs open. Is that something we can get fixed soon? That'd be great. All I want is for Firefox to take less memory than Azureus. I only have adblock and bugmenot, so it's not extensions causing the problem.
Not a Twitter sockpuppet... but I wish I was.
Because they chose to weld IE to the OS, they have more difficulty with patching (and the vulnerabilities become OS vulnerabilities).
If they had maintained a rigid distinction between OS & apps, they wouldn't have those problems.
This was predicted back when MS first "integrated" their browser.
Acomparison between Mozilla's time to patch and MS's isn't necessarily apt. After all, since their browser can't be separated from the OS, they clearly must regression test the entire Windows platform with every IE patch.
That bug has existed for several years now and has not been fixed. Hardly offtopic...
What we're seeing these days is an exodus from Firefox towards Opera, Konqueror, Safari, and other alternative browsers.
The Firefox 1.5 release didn't go nearly as well as it should have. Touted as being a breakthrough release, numerous people upgraded only to find that it was buggy, consumed far too many system resources, or just plain didn't work. Thus many people moved towards the other browsers that are available.
I used to recommend Firefox to my relatives, non-technical friends, and others. But I won't do it any more. Firefox has started to get a bad reputation, and I won't let their reputation affect my reputation. Thus I recommend Konqueror most times, but for people who can't switch to Linux or BSD I often suggest the use of Opera. Opera has shown for years now that they can write solid, secure, portable, performant browser. Thus they get recommendations from me.
Cyric Zndovzny at your service.
There were cases when there was a security bug known for years but only got fixed after public disclosure. If you look at that way they are not better than Microsoft at all.
Never learn by your mistakes, if you do you may never dare to try again
I'm not a statistician, but the average is sometimes a poor way to describe data. It's often useful to look at modes, standard deviations, and so on.
For example, the standard deviation for 2005 had Microsoft with a 80.87 stdev and Firefox with a 97.5 stdev.
Firefox had one flaw that took 674 days to fix, nearly twice the max of Microsoft's 357 days. Does that make up for such a larger average? Dunno. I suppose you could look at the issue and decide for yourself.
Averages are important, but it's not always the single most important thing to consider.
FTA: In recognition that 2004 was most likely the first year in which a significant share of the company's new user base was coming from Windows users, Security Fix based each of "date patch issued" date for 2004 and 2005 on the release date of the next product update that incorporated the fix for that critical security vulnerability -- not the date on which a fix was available to developers. For 2003 critical Mozilla flaws, Security Fix relied on the times listed in the "date fixed" field for each critical flaw listed on the "Older Vulnerabilities in Mozilla Products" page.
:)
So if you cut the days-to-fix time up by year, for 2004, the avg is 65 days. In 2003 they used the "fixed" date in the bug DB. For 2005, its 37 days, and for 2004/2005 combined, its 42 days.
The 2004/2005 # is the interesting one, because that measures how long until the patch actually makes it into a shipping build. The availability date of source-code patches is irrelevant because most organizations are not equipped to deal with them; this is especially the case with firefox!
None of this is an excuse, however. As an MS employee, the data on our speed of patch delivery is pretty upsetting - the numbers are much worse than I would have expected. They're so bad that I am suspicious and wonder if there isn't some deeper story somewhere, but in any case, I'd like to think the maximum time anyone would have to wait would be ~1 month (flaw reported on the wednesday after "patch tuesday"), but according to this data we're not hitting that at all. I can't speak for the IE or the MSRC teams but those numbers really appear to suck. Sorry about that, everyone
My opinions are my own, and do not necessarily represent those of my employer.
Skimming through the previous Slashdot story, it looks like the Microsoft vulnerabilities covered both the OS and IE, not just IE. Mozzilla, afaik, only does the browsing and mail programs.
Granted, that's no small task, but it still isn't on the level of fixing an O.S., in my opinion. It's like comparing apples and pumpkins.
It would be better to compare Windows patch release time with Linux patch release time, which I believe has been done before (and then covered on Slashdot- Linux probably had the shorter time.)
Regardless, how much does market share factor into this? With Linux, if a patch breaks a program, most people can just shrug it off and rewrite the program to work with the patch. So mass testing isn't as big of an issue. With Windows, if a patch breaks a program, a user doesn't have a lot they can do except to sit there and weep until Company X releases their own patch or next version.
Something I've really wondered about, and would really like an answer to: Why the hell is it so hard not to include bugs and exploits? I used to think it was just poor management, but now you have open source projects with thousands of eyes looking at every line of code. How is it that you can't write code that prevents these exploits? It's nice that you can patch it after the fact, but from what I remember in taking computer science, if you follow some simple procedures, the code is robust. What's the problem? I am dying to know.
Rank my idea: http://www.sinceslicedbread.com/node/531
Firefox 1.5 introduced a keyboard error which drives me crazy -- keyboard navigation drops out while editing messages (I think due to activity on other tabs), and I also lose the apostrophe key and other things (both it and the forward slash bring up the Find toolbar even when in an edit box like this one).
I tried to search for a way to report the problem, and found the Firefox bug reporting page to be a fricking maze.
Mainframe/UNIX Bit Twiddler and long time Windows/Linux Hobbyist.
The Theorem Theorem: If If, Then Then.
Remember, there are many, many user out there who have systems with only 512 MB of RAM, if not far less. They will quickly notice their system performance tank when they start using Firefox, especially if along with other heavy software such as Microsoft Office and Photoshop.
Even if they don't care about the numerical values in question, they still do care about getting work done. And when Firefox (or any other piece of software) is directly responsible for such slowdowns, it will be remembered.
Recall, excessive memory consumption is one of the most difficult reputations for a piece of software to break. People will come to associate Mozilla and Firefox with slow, bloated software. That won't bode well for future adoption of their products by the masses.
Cyric Zndovzny at your service.
The amount of time it takes the average user to install the updated software, both figures pale into insignificance....
It's the difference between people who write code with the intention of benefitting the community, and people who write code because they're paid to.
I kill harmless processes for sport
I used to think it was just poor management, but now you have open source projects with thousands of eyes looking at every line of code.
IMHO, I believe that the reason why is because most of the developers are looking "at the edges" - where new functionality is being added. For example, how many of those developers are looking at the JPEG decompress routine? Turns out that wound up being important exploit-wise recently. And there it sat for years, unnoticed.
from what I remember in taking computer science, if you follow some simple procedures, the code is robust.
Well, robust doesn't just come from simple procedures. It's also design and style. You can't come up with excellent procedures and guarantee good software. You have to design well, communicate well, and implement ideas correctly. A lot is also owed to experience - sometimes, the only way to find out you've screwed up is after the fact. A good example is strcpy(). We know unbounded copy is a bad idea now, but how many years went by before we did?
Weaselmancer
rediculous.
After upgrading to 1.5, I too have been using Opera more and more as of late. I've had the keyboard nav periodically not work. Trouble getting certain plugins to work and a noticeable increase in memory usage.
I'm doing with Opera what I did with Firefox (then Phoenix just prior to the name change to Firebird) when IE was pissing me off.
I downloaded Phoenix and would use it when I thought of it or when IE did something specific to piss me off. As time went on Mozilla's browser became the default browser and that has remained for some time.
Now with these nagging problems in Firefox, Opera is seeing a lot more light of day with Firefox use decreasing. I suspect that Opera will end up being my default browser when all is said and done if the trend continues unless Mozilla pulls it together.
As always, that's just my $.02 and YMMV.
Slashdot has only existed since 1997.
My amazing wife - Artist, Author, Philosopher - Laurie M
I'm well aware that Mozilla supports the Boehm garbage collector, in addition to various other memory allocators. The fact remains that none work as well as the garbage collectors offered by most production-quality LISP and Smalltalk implementations. Then again, that's partially because of C++ being so fundamentally different from other languages, to the extent where it isn't an easy task to write a solid garbage collector for it.
Cyric Zndovzny at your service.
Current useage for me with 4 tabs open is 76 mb, I don't know if this is a lot my Tuneup Utilities 2006 is green and I'm experiencing no system lag. In the days of 1 gig of ram do we really need to worry about a little memory leak, its like you put Norton on it slows down your system but it protects (I use AVG) but is security the price we pay for a less memory use browser. Remember leaks could also be down to the extensions you use that may have memory leaks themselves.
When even a normal user finds that Firefox has consumed 400+ MB of their 512 MB of RAM,
;)
Buy more RAM!
"I am the king of the Romans, and am superior to rules of grammar!"
-Sigismund, Holy Roman Emperor (1368-1437)
This is the view of an outsider to the project; I'm speaking strictly as a user here. It appears that the in particular Firefox and Thunderbird were brought along to certain level of functionality and usability and now are floundering. Critical security problems are being addressed but very old usability bugs stick around for years. It's the small things that tend to drive a person over the edge. That last bit of polish that makes the difference between a 'commercial' application and an open source one. I had high hopes that Firefox 1.5 would raise the bar even more but thinking back, I can't think of what was significant about the 1.5 release now. I think it was fixing the automatic updates, which was pretty critical.
Author of Enyo: Up and Running from O'Reilly Media
However, other bugs simply aren't fixed. For about 3 years many, many people have reported the CPU hogging bug which is unique to Firefox and Mozilla browsers. For a small example of the reports of problems see Firefox is the most unstable program in common use.
Now the problems are beginning to be reported in technical magazines, newsletters, bloggers, and even the mainstream media.
Under the conditions mentioned in the bug reports, I'm not able to make the CPU hogging bug fail; it is always there. I've tried Linux, Windows XP SP2, and Windows 98 SE. I've tried Intel and Via chipset motherboards. For about 3 years, in all versions, the CPU and memory hogging bug has always been there. Firefox version 1.5.0.1 is worse than Firefox version 1.5, and those versions are worse than earlier ones. This is with a clean profile and no extensions except DOM Inspector, which is a menu choice on the installation program.
In 3 years, I've never had any evidence that any Firefox or Mozilla developer has reproduced the conditions that cause the problem.
The problem with Firefox and Mozilla developers not fixing difficult bugs seems to be a social one, not primarily a technical one. The developers keep asking for the problem to be made easier, but it appears to me that there is already plenty of evidence that would allow further investigation.
Perhaps the developers do not understand that there is a class of bugs that can only be found using the methods of scientific research. Many people like programming, but only people who accept the biggest challenges truly have programming in their hearts and minds:
Three biggest challenges of programming
Here are programming's three biggest challenges. Coding is relatively easy. It is these challenges which separate a true professional from an average programmer:
Instead there are excuses:
Mozilla Top 12 Excuses
Top 12 things Firefox and Mozilla developers say about those who report difficult bugs, collected during the last 3 years:
"Today's post marks the second in what I hope will be a series of similar analyses. " He's got a link to a meta-study from CMU in that article too.
Best Slashdot Co
There's an analysis up at CMU (linked to in the WaPo article) which shows that closed source vendors as a group are slower at patching than open source ones.
Best Slashdot Co
It's important to remember that the enemy in TFA is Microsoft, not Firefox. It doesn't matter that Firefox has long-running problems that people have complained about for years. Firefox is Good. Microsoft is Bad. Remember that next time you want to complain about Firefox chewing a couple hundred megabytes of memory.
Firefox's leniant users
either your computer or what you're doing is the problem. Sorry. It doesn't matter if everything else works fine either. It is not Firefox.
I got firefox 1.0.7 here on gentoo gnu/linux with 5 tabs open using 21 MB shared or 50.4 MB RSS memory. Extensions: Adblock v5 d2 nightly 39, web developer 0.9.3, BugMeNot 0.6.2, adblick filterset G updater 0.2.4, proxybutton 0.2.1 and cookie button 0.8.4.
Remember, these values are WITH shared libraries that can be and are used by many other applications. So no, I don't think firefox is using too much memory.
What's eating memory is probably flash (which I don't have) or some other plugin.
Anyone with a bit more education in stats would have noticed the Extreme Outlier (674), the data looks much different when not counting item: Average about: 23 days. (NOT 37!)
Your comparing fixes to a web browser to that of an operating system!
Give me a friggin break!
I mean, Mozilla makes a web browser and an email client, period. Regardless of what any other project they have on the go, that is their bread and butter. I would expect a company with a singular focus to be able to fix bugs in their TWO major products quickly. On the other hand Microsoft makes an OS, an infinitely larger code base and more complicated set of code to fix in addition to many many many other products. Even if its just an I.E. vulnerability, Microsoft still has to focus on ensuring OS system components are not affected because of the integration of I.E. in Windows. Microsoft has billions of clients, and while firefox is a hot product now, Mozilla doesn't have to ensure that 95% of PC's are not going to be adversely affected by a quickly rushed security patch.
It would be more appropriate to compare bug fixes between Apple and Microsoft, or Sun and Microsoft, (not really fair between RedHat and Microsoft because RedHat is a one hit wonder as well).
I can't stand double standards and people jumping on the bandwagon every time Microsoft is mentioned negatively in an article. If Mozilla had the depth of innovation and breadth of products Microsoft maintains, and they still fixed critical flaws in 3 weeks, then my hats off too them. But to say a company making one product fixes bugs 10 times faster then a company with a more complicated set of products and larger codebase is ridiculous, period.
I am not fan of Microsoft, but give me a break here. If Microsoft was a person, it would be criminal the kind of bias, slander and double standards imposed on them by every self righteous narrow minded geek out there.
I haven't thought of anything clever to put here, but then again most of you haven't either.
Who cares how much work it takes for them to "turn around" a patch. What matters in the end is when the hole is fixed. And Mozilla seems to fix them faster. End of story.
How about the irony of seeing this article with a nice big "get the facts" advertisement from M$?
Nice work slashdot. This is journalistic ethics at its peak.
I know it is randomized and not everyone gets the same add but it is so hypocritical as to make me want to vommit.
RTFA: "I must insert a strong caveat here, however. The 37-day average is skewed mightily by a flaw found in various Mozilla products that potentially allowed malicious Web sites to trick users into accepting security dialog boxes -- a flaw which Mozilla took 674 days to fully remedy. This was a vulnerability that apparently existed in all browsers. (Microsoft got around to fixing an identical flaw -- which it labeled a "moderate" security risk -- in December.)
According to Chris Hofmann, Mozilla's director of engineering, the fix was delayed in part by speculation that it could cause the browser to constantly pop up annoying alert dialog boxes. But Hofmann noted that the early beta releases of Firefox in March 2004 closed off the problem as originally defined by the guy who discovered the flaw (Jesse Ruderman, who was since hired on as a full-time researcher at Mozilla).
With that flaw left out of the data, Mozilla took an average of 23 days to develop and incorporate fixes. And even this lower average does not give a clear picture of Mozilla's typical response time. In the past three years, Mozilla produced roughly one-third of its critical security updates within less than 10 days of being notified of a potential problem"
Your skill in oversimplifying is clearly underrated. And your adept ability to equate apples and steak should not go unmentioned.
Move out of your parents house and get a job.
As a user, Firefox works fine for me, I have never had any "critical" problems or seen any "bugs" in it. And I would not trade my Firefox for any other browser
Personally I would have thought that this was more a development model and documentation issue than a market share issue. One of the major reasons that third party software breaks when Microsoft changes its own software is that it's so often unclear about its API's. Programmers have to rely on half-documented API's, and on brittle work-arounds for badly documented Microsoft bugs rather than robust and clear interfaces.
Try writing a non-trivial Outlook addin, for instance, without having to cope with a range of Outlook API bugs and strange ways of acting. The unofficial way of getting around these is to use undocumented hacks that end up being completely unofficial and quite flakey.
Market share seems to be one reason that Microsoft needs to test so many individual software packages with its changes, simply because it can cause such huge problems for people every time they break. If it'd provided stable, robust and well documented API's in the first place, though, I don't think that other people's software breaking would be nearly as much of a problem.
Some examples:
-Gonz
No one forced Microsoft to integrate the browser into the OS. That was their mistake.
!#@%*)anks for hanging up the phone, dear.
Judging by replies, perhaps my intended sarcasm wasn't apparent...