Slashdot Mirror
Another Look At Mozilla's BugFix Rate
An anonymous reader writes "Washingtonpost.com's Security Fix blog has published the results of a look back at three years worth of critical patches from Mozilla, and found that Mozilla typically ships updates for critical flaws in about three weeks, though in more than a third of the cases it pushed out a fix in ten days or less. The data comes just a few weeks after The Post published data from a similar study that found Microsoft averaged 130+ days to fix critical flaws. Slashdot also covered that study in a previous post."
Maybe it's just imagination, but I thought I've been reading these kind of stories on Slashdot since the dawn of time.
Beware: In C++, your friends can see your privates!
While it's important to fix bugs quickly and correctly, perhaps the Mozilla project should take some initiative within the open source community to work on preventing security issues in the first place. They could partner with the OpenBSD project on such an initiative, for instance.
Together they could come up with a development system that focuses on writing solid code. Such a methodology won't prevent all security bugs by any means, but it could go a long way towards vastly increasing the quality of the Mozilla project's software.
Cyric Zndovzny at your service.
Ok, I guess three weeks can be counted in hours, but that's a LOT of hours.
If you need web hosting, you could do worse than here
Funny. IMHO, the speed of the browser peaked a long time ago (0.8 IIRC), and now it's just getting progressively slower over time.
They might be fixing critical security bugs, but they certainly don't seem to be fixing memleaks and such.
"Mod, mod, mod...and another troll bites the dust."
Has anyone collected similar data regarding Opera, OmniWeb, Safari, and other alternative browsers?
If so, how do they compare to Mozilla and Internet Explorer in not only the time it takes to address security problems, but also in terms of the number of incidents per release?
Cyric Zndovzny at your service.
In fairness, everything that I've read about MS's patch cycle indicates that it is a pretty huge undertaking. Joel from http://joelonsoftware.com/ is always going on about have every single code fix/feature addition has to go through a whole bunch of people (several testers, documentation team, etc) before it can be released. If anything maybe Microsoft is a bit too thorough with their patches, in some ways at least.
No problem, they just made the shovels REALLY HEAVY, so they only had to make a few of them.
Software metrics are very slippery things.
To be fair, Microsoft's flaws are alot more serious, so it's only logical they will take longer to fix.
<laugh\>
Big ones, small ones, some as big as yer 'ead!
Give 'em a twist, a flick o' the wrist...
I'd love if Firefox didn't take up 256 megs of ram with 5 tabs open. Is that something we can get fixed soon? That'd be great. All I want is for Firefox to take less memory than Azureus. I only have adblock and bugmenot, so it's not extensions causing the problem.
Not a Twitter sockpuppet... but I wish I was.
Because they chose to weld IE to the OS, they have more difficulty with patching (and the vulnerabilities become OS vulnerabilities).
If they had maintained a rigid distinction between OS & apps, they wouldn't have those problems.
This was predicted back when MS first "integrated" their browser.
That bug has existed for several years now and has not been fixed. Hardly offtopic...
I'm not a statistician, but the average is sometimes a poor way to describe data. It's often useful to look at modes, standard deviations, and so on.
For example, the standard deviation for 2005 had Microsoft with a 80.87 stdev and Firefox with a 97.5 stdev.
Firefox had one flaw that took 674 days to fix, nearly twice the max of Microsoft's 357 days. Does that make up for such a larger average? Dunno. I suppose you could look at the issue and decide for yourself.
Averages are important, but it's not always the single most important thing to consider.
FTA: In recognition that 2004 was most likely the first year in which a significant share of the company's new user base was coming from Windows users, Security Fix based each of "date patch issued" date for 2004 and 2005 on the release date of the next product update that incorporated the fix for that critical security vulnerability -- not the date on which a fix was available to developers. For 2003 critical Mozilla flaws, Security Fix relied on the times listed in the "date fixed" field for each critical flaw listed on the "Older Vulnerabilities in Mozilla Products" page.
:)
So if you cut the days-to-fix time up by year, for 2004, the avg is 65 days. In 2003 they used the "fixed" date in the bug DB. For 2005, its 37 days, and for 2004/2005 combined, its 42 days.
The 2004/2005 # is the interesting one, because that measures how long until the patch actually makes it into a shipping build. The availability date of source-code patches is irrelevant because most organizations are not equipped to deal with them; this is especially the case with firefox!
None of this is an excuse, however. As an MS employee, the data on our speed of patch delivery is pretty upsetting - the numbers are much worse than I would have expected. They're so bad that I am suspicious and wonder if there isn't some deeper story somewhere, but in any case, I'd like to think the maximum time anyone would have to wait would be ~1 month (flaw reported on the wednesday after "patch tuesday"), but according to this data we're not hitting that at all. I can't speak for the IE or the MSRC teams but those numbers really appear to suck. Sorry about that, everyone
My opinions are my own, and do not necessarily represent those of my employer.
Skimming through the previous Slashdot story, it looks like the Microsoft vulnerabilities covered both the OS and IE, not just IE. Mozzilla, afaik, only does the browsing and mail programs.
Granted, that's no small task, but it still isn't on the level of fixing an O.S., in my opinion. It's like comparing apples and pumpkins.
It would be better to compare Windows patch release time with Linux patch release time, which I believe has been done before (and then covered on Slashdot- Linux probably had the shorter time.)
Regardless, how much does market share factor into this? With Linux, if a patch breaks a program, most people can just shrug it off and rewrite the program to work with the patch. So mass testing isn't as big of an issue. With Windows, if a patch breaks a program, a user doesn't have a lot they can do except to sit there and weep until Company X releases their own patch or next version.
I used to think it was just poor management, but now you have open source projects with thousands of eyes looking at every line of code.
IMHO, I believe that the reason why is because most of the developers are looking "at the edges" - where new functionality is being added. For example, how many of those developers are looking at the JPEG decompress routine? Turns out that wound up being important exploit-wise recently. And there it sat for years, unnoticed.
from what I remember in taking computer science, if you follow some simple procedures, the code is robust.
Well, robust doesn't just come from simple procedures. It's also design and style. You can't come up with excellent procedures and guarantee good software. You have to design well, communicate well, and implement ideas correctly. A lot is also owed to experience - sometimes, the only way to find out you've screwed up is after the fact. A good example is strcpy(). We know unbounded copy is a bad idea now, but how many years went by before we did?
Weaselmancer
rediculous.
Slashdot has only existed since 1997.
My amazing wife - Artist, Author, Philosopher - Laurie M
I can imagine it's probably easy to think of a bug as some kind of syntax mistake or typo in the program where, were you to look at the offending line, it would be obvious. That may be true in some rare cases, but with complex programs like FireFox a bug or security hole can be caused by an unusual reaction or side-effect that happens only when two components interact under some specific, unusual (often unexpected) circumstance. There may not be one single line with a mistake - it may encompass various libraries or even the architecture of the application. When a program grows beyond a certain size, the possible interactions grow exponentially and it isn't realistic to simply test every situation by hand.
Also, the code-base is constantly evolving. A new feature or a fix in one place may break something else. If there are 1,000 developers working on the code, then it is likely a large code-base and developers each work on their own tiny part. There is probably not an individual developer who knows all of the code.
Writing solid code is very tough, especially so as the organization grows. The coders do need to follow best practices as you mention, but you need a lot more than that. A good QA procedure and getting people to follow the rules in a large organization is really a difficult task. My hat is off to companies that do a good job.
TODO: come up with a clever sig
However, other bugs simply aren't fixed. For about 3 years many, many people have reported the CPU hogging bug which is unique to Firefox and Mozilla browsers. For a small example of the reports of problems see Firefox is the most unstable program in common use.
Now the problems are beginning to be reported in technical magazines, newsletters, bloggers, and even the mainstream media.
Under the conditions mentioned in the bug reports, I'm not able to make the CPU hogging bug fail; it is always there. I've tried Linux, Windows XP SP2, and Windows 98 SE. I've tried Intel and Via chipset motherboards. For about 3 years, in all versions, the CPU and memory hogging bug has always been there. Firefox version 1.5.0.1 is worse than Firefox version 1.5, and those versions are worse than earlier ones. This is with a clean profile and no extensions except DOM Inspector, which is a menu choice on the installation program.
In 3 years, I've never had any evidence that any Firefox or Mozilla developer has reproduced the conditions that cause the problem.
The problem with Firefox and Mozilla developers not fixing difficult bugs seems to be a social one, not primarily a technical one. The developers keep asking for the problem to be made easier, but it appears to me that there is already plenty of evidence that would allow further investigation.
Perhaps the developers do not understand that there is a class of bugs that can only be found using the methods of scientific research. Many people like programming, but only people who accept the biggest challenges truly have programming in their hearts and minds:
Three biggest challenges of programming
Here are programming's three biggest challenges. Coding is relatively easy. It is these challenges which separate a true professional from an average programmer:
Instead there are excuses:
Mozilla Top 12 Excuses
Top 12 things Firefox and Mozilla developers say about those who report difficult bugs, collected during the last 3 years:
I understand this, but I don't think it changes my point. That kind of complexity can be solved with asking "what are all possible inputs" and "what are all possible outputs"?
What you're talking about is one of the oldest pieces of programming advice ever: never trust input. One of the most common human cognitive failings is to focus on intended outcomes (in this case processing valid data) and not focusing on unintended outcomes (having to handle invalid data). Probably 90% of the bugs in the world would go away if people took this very old advice.
But it isn't that simple.
Software is the product of social institutions. The culture of these institutions, the structures of rewards, the character of the people they attract are a huge influence. In a nutshell: broken organizations produce broken software. You may know that taking an extra week to design your input routines is the right thing to do, but it may not be the answer that is acceptable to the person you work for, and the aggregate sum of doing all the things that are right to do is nearly certain to be unacceptable at some level. In that sense, many bugs are like pollution: you take a problem you have right now, and make somebody else pay for it later. Often that person is your future self.
Secondly, you can't really consider "every possible input". A mere stream of eight bytes has 2^64 possible values, but is not enough to contain the smallest possible HTTP transaction.
Naturally, what you do is break inputs up into categories or sets. But this process itself is fraught with error. There was an academic vogue a few years back for "provably correct software". The idea was to create a detailed specification, then use a mathematical proof to show that a given program implemented that specification. The reason it didn't take the world of professional programmers by storm is that the proof itself, indeed the specification itself, are nearly complex as the program and susceptible to their own errors.
Which leads us to the last item: your specification. IF your specification doesn't meet the real needs of your user, now and forever more, it will be perceived by the user as a bug.
That said, it's really our dsyfunctional development organizations that are the biggest source of bugs. When we build the perfect society, or at least a perfect piece of a society, then we'll be able to eliminate the largest source of bugs.
Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.