Slashdot Mirror
Debugging in OSS Always Faster
dex@ruunat writes "Damien Challet and Yann Le Du of the University of Oxford studied a model of software bug dynamics, which resulted in a paper on cond-mat this morning. In this paper they study the difference in evolution of number of bugs in open and closed source projects. They conclude: 'When the program is written from scratch, the first phase of development is characterized by a fast decline of the number of bugs, followed by a slow phase where most bugs have been fixed, hence, are hard to find'. Another, perhaps surprising conclusion is that debugging in open source projects is always faster than in closed source projects."
what software package do you know of that is bug free? (besides solitare;) )
Another, perhaps surprising conclusion is that debugging in open source projects is always faster than in closed source projects."
Why, may I ask, is this suprising?
As with all statistics, you can make them say whatever you want...
Maybe most OSS projects are easier to debug because of lack of features, or smaller scope, etc.
What percentage of OSS projects, on say sourceforge.net, have a version number 1.0? (and are "widely" used). The first one that comes to my mind is MythTV and/or FreeVo. I can't speak of freevo, but mythtv (while being impressive) is still very bug ridden and has been out for over a year.
Another factor is the user group, of course. With OSS I imagine the kernel gets more bugs submitted by users than mythtv, just because the users aren't so much code hackers... they just want to use it.
The one rule in the software engineering is that there are no rules.
no comment
The paper's conculsion seemed to be that debugging open source projects is faster because you don't have a version problem where customers report bugs in code that has already been modified for the next version.
I don't buy it. Many open source projects (ACE/TAO, Mozilla) for instance have large customer bases using non-current versions, and presumably finding bugs. Sure, if you only want to fix the bug in the released version, its faster, but it's not like closed source vendors don't have the source code to their previous release to debug with.
Sure debugging is faster if you always make everyone upgrade to the latest version before filing a bug report. Good luck getting mass acceptance with that.
Not always. A more likely explanation is the 'many eyes' that can review the code.
You are being MICROattacked, from various angles, in a SOFT manner.
I'd imagine that one of the most difficult parts of debugging for a large OSS project is dealing with the deluge of bug reports.
That's a prime example of survival of the fittest, and the true formula for the best software. Competition is the key, and not just inter-OS competition like Outlook->Evolution, but things like different ftpd variants, different crons, syslogs, etc - Gentoo Linux has most openly encouraged this variety from what I can see..
-Digital Extremist
"Another, perhaps surprising conclusion is that debugging in open source projects is always faster than in closed source projects."
.... is to release it to the outside world)
...
...
I thought that this was the one pro OSS argument that was the least argued; release early and often (since the only way to simulate all the different conditions your software will encounter in the outside world
This is also based on the fact (?) that the hardest part about squashing bugs is finding them
Is this actually controversial? (seems pretty much common sense to me, but I do not do programming as a profession, have I been misled?)
PS
Not to take anything away, it's one thing to have something be so "obvious" as to be "common sense", and quite another to have some data to back it up
I would think the opposite -- that there are more (talented?) eyes looking at OS projects than CS projects. Many times closed projects have several developers, but only one ever sees a particular module at any given time. At my current company, we have 30 or so developers, but the modules I write are owned by me and usually only seen by me. Peer review sessions can alleviate this, but those are generally short and cover major functionality. With OSS, you have an untold number of eyes viewing a project that can help catch problems in a more timely manner.
The theory that your best work will be done when the most eyes are watching can also apply. I think we (developers) are all guilty of shoving some nasty code in a project at some time or another under the notion that no one else will ever see it. When the whole world can look at your work, sometimes those attitudes change.
This was a paper written for a class on statistics. It was not a rigorous study. Their findings are based on a lot of assumptions. They have a very small sample set--they only test their model on Linux, fetchmail, and Mozilla. Many people, including myself, consider these the cream of the crop so far as OSS goes.
Before you praise it, I urge you to actually read the paper. Don't be intimidated by it--FUD is FUD, even if it's mixed with a heavy does of greek letters and charts.
I do a lot of coding working mostly with open source products, and sometimes closed source. When I get some bug come up in an open source product, I actually go digging into their code sometimes to figure out what went wrong. If it's closed source I can dig down through my code, but once I hit their code, it's a brick wall.
Frankly this is why I try to stick to open source software when I do development work. Hell of a lot easier to figure out how something works when you've got code and direct access to the developers via a mailing list.
This sig has been temporarily disconnected or is no longer in service
Something else I can think of is that your testers have access to the code and can give you more precise bug reports than if they were doing traditional black box testing. Better bug reports and information means less time investigating.
"People that quote themselves in their signatures bother me" - athakur999
There are a number of different possiblities here, given the different dynamics of each group.
1. In general, there are more users of closed source software, so bugs may be discovered at a faster rate. With limited development resources, the greater number of bugs take longer to fix.
2. Users of open source software tend to be more programmer-minded. They find bugs and fix them themselves, since they have access to the source code. Everyone fixing their own bugs leads to faster debugging times.
3. In larger companies, development and test typically are two distinct groups. There is an inherent lag in this that leads to slower response time in removing bugs. Open source software is developed and tested by the same core, ususally small group. Since the same people develop and test, they are more likely to find the bugs in their own code and fix them quicker.
Just some observations, and I am sure there are other reasons. I'm sure the results of the study are a combination of many factors.
-- Fighting mediocrity one bad post at a time.
all the users use the modified code at time t + 1 and report bugs exclusively on the updated code.This assumes that all the users update their software at every release.
This is completely bogus. Not every user is going to update immediately. In fact, the larger the company is the less likely this is due to their desire to qualify new software. This means as Open Source gets more popular with big companies, the more bogus this assumption is.
The paper also mentions nothing about QA efforts or beta sites on closed source, which are typically on the "immediately updated" products.
I'm not going to argue whether oss is better/worse than closed, but I heavily doubt this paper proves anything other than if you make lots of assumptions you can prove whatever you want.
How come stories like these always seem like they get posted because of zealotry?
No sig for you!!
Your "untold number of eyes" is nearly indistinguishable from "0" unless your open source project is widely used. Sure, this may hold for the Linux kernel, or Apache, or even Mozilla, but what about all of the open source projects on SourceForge?
Sounds to me like you're working with an inadequate number of testers, or at least an inadequate unit testing plan for developers. Testers are invaluable because that's all they do -- test code. This frees up the developer to be able to actually write code while not having to sacrifice quality for lack of testing. Sure, it's more expensive to hire both testers and developers, but I'd bet that of your 30 or so developers, you could fire 10 of them and hire 15 testers for the same cost, and still have enough man power for development while increasing your code quality from rigorous testing. (Note: This isn't saying that developers can or should write bad code so long as they have testers. Developers should still aspire to writing quality code, so that the testers can focus on the really heinous parts, and not on trivial bugs or logic errors. But at least with testers and good unit tests, you're less likely to let slip a silly bug.)
That may be true in theory, but it doesn't pan out in practice. In practice, most open source projects will only be seen (code-wise) by the author(s). If you're lucky, you might have two or three active users that will submit bugs or patches, but often that's not the case.
We have egos at work too. I am not supposed to TOUCH anything my coworkers do, it is THEIR code and THEIRS to command, hands off! Even when I sneak a peek and find obvious bugs, I am not allowed to fix them because it violates territoriality.
This sort of code hording is impossible in OSS, and it's a shame. It's well known in literary circles that the more eyes that grace a page, editorially, the more likely you'll be to have a coherent and gramatically correct work. Code is no different IMO.
Of course, code hording does decrease the inevitable argument over implementation that plague many software houses...and for which OSS is uniquely famous.
Hey freaks: now you're ju
All generalizations are false.
25% Funny, 25% Insightful, 25% Informative, 25% Troll
That is a general insight into bug testing and perfectly valid, except your statement is a bit flawed.
If the testers have some general coding and OS knowledge, they can help determine the fix before sending off an issue. It doesn't really matter whether the code is publicly available or not. Testers for non-open source projects can have access to the code as well. The code isn't usually kept in a vault with armed guards protecting it, you know. It is more a matter of having knowledgable and studious testers.
Successful OSS projects must be well documented in order to survive. Naming variables in an intuitive manner and providing insightful comments isn't about improving your annual review scores, it is about ensureing that others can and will read your code.
Companies like Microsoft need to introduce policies to create the same effect. Code reviews and extreme programming are good examples. They often degenerate into either a rubber stamp or a grudge match between different interpretations of Hungarian Notation.
Many eyes can. How many actually do? Unless you're talking about the really big projects, probably very few indeed -- one, I suspect, in many cases.
It's not fair to cite mainstream developments like Linux or Mozilla as the way all open source is any more than it's fair to cite Microsoft's history on things like security and reliability as the way all closed source is.
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
The gimps ui is totally app centered. The "in charge" window is the toolbox.
:
The gimp could be fixed pretty easily to annoy a lot of people less:
1) Menus across the top of each image window.
The right click to access the main menu is not familiar at all to most people. To fitts law lovers
when gtk supports universal menus, that will work,
currently, it doesn't.
2) The toolbox should not be the master of the universe -
When the last image is closed, the gimp should close.
But I think the whole thing would be even better off with a "normal" interface, ie like Koffice or Gnumeric, with dockable toolboxes, etc.
It should be impossible in a sensibly run closed source project as well. If you have a development team whose egos are such that they can't stand criticism and claim unique ownership, you are on the absolute lowest rung of the smart development process ladder. Teams full of such people will never produce good results, whether their source is closed or not.
As with many of the points raised in a thread like this, though, the problem is with industrial development processes and not actually with closed source. In a decent coding shop, no-one has sole knowledge or control of any area of code. In better shops, reviews are routine, and the good guys actively solicit feedback from their peers.
This is just another case (like the "many eyes" argument, and others) where the argument made in favour of open source looks great at first glance, but doesn't stand up to much scrutiny. To date, the most convincing argument I've seen for why some of the big open source projects have generally good code is still that those who do it frequently do so out of interest, rather than just to pay the rent (though certainly most of the work on the big OSS projects is done by pro's these days).
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
" Not always. A more likely explanation is the 'many eyes' that can review the code."
I went to a speech by Gene Spafford here a few years ago, when the subject of Linux code quality versus other systems (especially MS) came up. Someone mentioned Eric Raymond's "Thousand Eyeballs" theory, that more people looking at the code ensured better quality.
Spaff responded "that does no good if those thousand eyeballs are looking at things like networking your toaster instead of quality and security".
I don't think this point is emphasized enough. It's not enough that lots of people are looking at the code. You need lot's of people with training, expierience, and an eye for problems to look at the code. He pointed out that one of the biggest problems in development is that while people can learn C from a book, and even get good at it, they don't learn proper software engineering techniques, philosophies, and debugging skills that way.
In short, simply being open source and having lots of developers isn't a solution in itself.
Life is hard, and the world is cruel
Two of the three open projects you cite also happen to be bigger than most projects, with more developers working in || at the same time than most projects. Does that help them? Read "the mythical man-month" for the answer. (Hint: no).
You have seriously misread The Mythical Man Month, to the point of absurdity. The Mythical Man Month does emphatically not claim that more people can't do any more then fewer people.
What it says is that programmers are not interchangable, and as a result, adding programmers to an existing project will not get a 1-man-hour to 1-programmer-hour gain. While the old programmers are training the new programmers, and while the communication overhead increases, and while the new programmers are not yet settled in, productivity may fail to increase or may even decrease, over the next few months.
However, once the new guys are settled in and assuming a decent organization, the new organization can indeed move faster.
The Mythical Man Month's main point is to avoid treating programmers, or perhaps more accurately programmer-hours, interchangably. The conclusion based on that is the somewhat more famous part about not adding manpower to a late project because it will make it even more late.
Most strong open-source projects, and probably all the ones in this study, have a strong group of core people who are intimately familiar with what they need to know to continue improving the software. Other people may drift in but will find it difficult to contribute code back. (Seriously, just try to get a patch into Mozilla.) The Mythical Man Month does not apply to steady-state teams of people, only growing ones.
So yes, larger teams help those projects create their large software products. Go look at how much code is in Mozilla; no matter how wizard you are, I don't think a small team of people could even type that much code in anything less then a couple of months. How else do you think they could do it but with a large team?
I think those are probably fossil practices left over from pre-Internet days.
It is important to shield the developers from telephone calls and visits from the customer, because otherwise the developers will get nothing done, not necessarily because they have no time (time_in_day - time_dealing_with_customers may still be large), but because they are constantly being interrupted.
As long as the developer has discipline and the customer realizes that the answers won't be instant, though, there can be great value in having email access to the developers, for both parties.
Hopefully more companies will modernize their policies soon.
The code isn't usually kept in a vault with armed guards protecting it, you know
Boy, are you in for a rude surprise when you graduate.
But seriously, most testers for larger systems are busy enough maintaining test scripts, documentation, result sets etc. to actually try to stay up to date on the build tree & process. Admittedly this is less true for unit test than for system and integration but then unit testers are often just developers in funny hats.
This, of course, begs the question of why this isn't a serious problem for OSS but I'll let somebody else tackle that.
#3, "who write and use OSS", implies something very important: the people who writes it has to use it. That's why they care.
That's not entirely true. Specifically, if your closed source app includes a decent plugin architecture and enough documentation to get going, you're no longer restricted to what the original developers planned. For example, take Winamp. In the broader sense you're still limited to playing media, but the original developers surely didn't write it with playing music from NES ROMs in mind. Yet because they build a decent plugin architecture for various components (input, output, visualization, general), winamp can do just that. Another good example would be Internet Explorer. It was designed to be a best-of-breed web browser, but you can do so much more with it. You can embed it in your own applications, you can extend it, you can write activeX objects that allow it to display other formats than HTML (like pdf or Word documents).
Having source code is not a requirement for extending an application to suit your needs if the developers had enough forethought to allow for that with proper architecture. Granted, if the architecture isn't there then you can't do much, but that's more the exception than the rule these days.