Open Source Studies

e8johan writes "Avaya Labs Research has presented a paper studying the open source process in the cases of Apache and Mozilla. They reach a number of interesting conclusions, the ones I find most interesting are: * Open source projects tend to have a core team of 10-15 coders, producing almost all code. The next layer is a set of developers submitting new features and bugfixes. The next layer is a set of advanced users submitting bug reports. * Open source projects tend to have a lower bug-rate than commercial projects. * Open source projects are generally quicker to respond to user requests. The article also discusses the differences between projects that have always been open source (such as Apache) and projects having a proprietary history (such as Mozilla)."

Comments

of course 15 coders makes for less bugs

[PissedOffGuy]

if these projects average 15 coders, on average they're also significantly less complex projects, and then of course on average theyll have less bugs.

also, if you have a team of people who are PAID to find bugs, theyll find more.

Re:of course 15 coders makes for less bugs

[roachmotel3]

Do you really think that projects like Apache, OpenOffice, Mozilla, Xfree86, and Linux are less complex than say, IIS, Word, IE, or Win98?

I mean, seriously, if you look at functionality, things are getting very close between the OSS world and the Microsoft world.

I'm not saying that there are many straight forward OpenSource Projects, but let's be real -- there is a complete OpenSource O/S, that runs and performs amazingly given a core team of 15 people.

Re:of course 15 coders makes for less bugs

[Tet]

if these projects average 15 coders, on average they're also significantly less complex projects

If you think that many software projects need more than 15 coders, you obviously don't have much experience of software development. In all my years of software development, I've never seen a core team bigger than 10 people. Sure, Word may well have 150 developers working on it. But don't for one second believe they're all actively coding Word in one big team. Most will be working in smaller groups of 4 or 5 on one specific area (for example, the spell checker).

Re:of course 15 coders makes for less bugs

[gpinzone]

Do you really think that projects like Apache, OpenOffice, Mozilla, Xfree86, and Linux are less complex than say, IIS, Word, IE, or Win98?

Yes. They are. Microsoft products don't just end at the product itself. They try to integrate their products into the OS. Internet Explorer just isn't a web browser. If it were, you'd never be able to have the level of security holes as it does. Without getting into a debate over whether or not it makes sense to integrate the browser into the OS, just realize that part of IE development is the OS development itself. On the whole, none of these projects are as complex.

Not just open source

[aengblom]

"Never doubt that a small group of thoughtful citizens can change the world. Indeed, it is the only thing that ever has."

-- Margaret Mead

It coule be better

[bsharitt]

I bet most open source projects would be better if the 10-15 core coders were paid as full time employees. Of course this would require compaines to back them, which is easier said than done.

Complexity != number of coders

[MosesJones]

Get thee to the bible of all things Software The Mythical Man Month. Take 15 motivated and talented indivduals (they have to be both) and they will whoop the arse off a team ten times their size which is filled with average people.

Adding more people _makes_ a project more complex but not in terms of the problem being solved, it makes it more complex because there is more communication and communication is not always accurate, the more communication the more bugs. Its no suprise when you look at some elements of large OSS projects that you see that PersonX does everything on Y, its this "master" concept that helps them deliver. And of course in having an excessively large testing team by commercial standards, testers out-numbers codes by huge ratios, any one been on a commercial project where there was even parity.

OSS is the best way to run a paid or unpaid project IMO, the problem is that it looks so expensive on the surface the companies don't do it. But the Total Cost of Ownership is much higher because of the lack of testing and the lack of review, and of course because instead of 15 developers and 100 testers they have 100 developers and 3 testers, 6 managers, 1 programme manager, two account managers, one account director and two administration assistants.

The common factor between OSS and standard commercial is that no-one does enough documentation.

OSS as an alternative

[OmniVector]

Open source software has been in my mind more of a philsophical debate than one of software production. It seems like computer science mimics things a lot in regular science. A new *thing* is discovered, and becomes a widely used standard incorporated into other programs (aka inventions) and it becomes part of the market place.

According to the article: Proponents claim that OSS software stacks up well against commercially developed software both in quality and in the level of support that users receive...

In many ways this is true, but coming from me, someone who is trying to switch from windows to linux, help is a lot harder to come by than they claim. I've relied much on my friends who have used linux to help me get my system running, and without their help I would have spent weeks on google, newsgroups, forums, doc, and man pages just to get things as simple as my audio drivers for my laptop working.

Support for OSS is minimal at best, and that's to be expected. When you have to pay for software, someone is payed to answer phone calls, to write thorough docs.. because it is their JOB. I know a lot of people, such as those 10-15 dedicated developers like the article says, can do a lot when it comes do documentation and support, but companies beat them hands down in this department. That is a big problem, there needs to be a better system. The irony there is if you make linux easier to use you lose the power of customizing your kernel, or optimizing programs by compiling them on your machine, etc.

If something isn't done though, OSS software will always take more time to setup than commercial software.

Re:OSS as an alternative

[iabervon]

I've relied much on my friends who have used linux to help me get my system running

But that has to be included in the support for OSS. In fact, that's the only kind of support that most people get with just about anything technology-related. Furthermore, it's the best support, because the person actually knows you, and both understands what you're saying and cares that you get it working. With commercial software, you often have to deal with people who don't care if your problem gets solved, so long as they get paid; furthermore, it's much harder to find someone who can actually fix something that's broken.

Customizability and ease of use are not actually in opposition at all; you just need to have the defaults set right. Each new option which gets set, by default, by looking at your usage, improves both ease of use and customizability. Local compilation doesn't make things more difficult, because it can be done without any interaction; there's no reason that, when you download a binary and install it, the system couldn't download the source and compile it in the background, and then replace the binary installation with the compiled one. In fact, compiling a program locally is much likely to work than using a precompiled binary, and compiling most programs doesn't take as long as reading their documentation on recent hardware.

BTW, the unacceptably-slow Linux installation took less time than Windows ever has.

Much like closed source

[mikewas]

The number of developers that are actually contributing seems much like the commercial closed-source projects I've worked on. There's always a small team that really understands the code that does most of the work. The core tends to be about half of the team on small projects. Everybody else performs ancillary functions or just goofs off. On larger projects communication breaks down and the core is limited to a no more than a few subgroups of one to 7 developers each. The subgroups tend to work independently, only occasionally interacting with one another -- usually though some spokesman or leader.

So open source group dynamics are similar to closed source projects. Not really surprising, since both are staffed by people!

Larger more formalized projects, aerospace for example, improve on the above by making subgroups of subgroups. This layering of project & program management really increases the overhead. It seems to slow things down, but at the end you can put things together and have a hope of making it work. It's really a formalization & extension of the way we organize ourselves naturally.

What About OSS Failures?

[theduck]

It's all well and good to focus on the characteristics of a successful OSS project. It's extremely interesting that these projects succeeded without elements that are considered to be fundamental to success in commercial development. However, studying success provides only half the picture. It won't tell you what things to avoid that tend to make OSS projects fail. Without that knowledge, attempts to reproduce and improve upon the methods used by Apache and Mozilla will experience unforseen failures that could have been avoided.

Up to old tricks

[gnovos]

It is often characterized as a fundamentally new way to develop software that poses a serious challenge to the commercial software businesses that dominate most software markets today.

I was under the impression that this kind of approach to building is a fudamentally old way of getting the job done. The Homebrew Computer Club essentally built everything "open source" (well, it was hardware mostly, but same approach). The current resurgence of OSS is not something new and revolutionary, it is instead a rediscovery of old techniques that were coopted by big business.

One thing the report forgot to mention ...

[Taco Cowboy]

The report mentioned many things that we already know. But there's one important thing about the Open Source software the report may have missed :-

The freedom to change / customize the software via the source code.

People can argue that even Windoze can be customized - background, for example - but it ain't the same as cusomization via source code.

Do you ever have the feeling, when you use commercial / close-source software, that some part of it are kinda stupid, cumbersome, or simply plain assinine ?

Do you ever think that if you _just_ have the source code, perhaps you could do some change to it, at least to better suit your taste ?

Well ... With Open Source, we can.

Now, of course, not every one know how to code, and even fewer of us know how to tweak the code to our own liking. But that doesn't change the point that with Open Source, we _can_ change the software anyway we like it.

It's a feeling of having TOTAL CONTROL over the software.

It's a feeling of empowerment.

It's just _the_ thing close-sourced software users don't get to enjoy.

Re:Not to be obvious...

[gpinzone]

It depends how you define successful. Yes, Apache and Mozilla are great products, but if there are so great, why aren't people dropping their closed source software and downloading their open source counterparts in droves? Hell, the two examples given are not only open source, but they're free!

Obviously it all can't be a success. How about the downsides? What about time to market? How long did Mozilla take to deliver a 1.0? What about lack of common features that customers want? (When I say customers, I mean the target audience as a whole, not just the geek community.)

Re:Not to be obvious...

[5KVGhost]

OpenSource development is more successful because the people involved love what they're doing.

Unfortunately that implies a disadvantage, too: Things they don't love doing often don't get done at all.

FUD

[e2d2]

Wow, FUD. According to our study we created to reach our preconcieved notion open source methods are superior.

Where to start?

Open source projects are generally quicker to respond to user requests.
I'm sorry but comparing two open source products to "commercial products" which is who? what product? what project? I don't see any quantative data besides a few lines refering to commercial products as a whole and saying the authors have experience with them is not scientific. I take exception to this because the paper sepcifically tries to appear scientific but yet offers no data comparing either project referenced (Apache and Mozilla) to a commercial counterpart or their ability to respond to bugs.

Open source projects tend to have a lower bug-rate than commercial projects
Again, where is the data? I see the scietific method they use for tracking bugs per line of code and they go into great detail comparing the two projects but yet we see no comparison to commercial project bug counts or the same method applied to commercial projects. The paper is laced with phrases such as "One might speculate". Yeah one might. Of course one might not speculate and offer evidence. If I create a hypothesis should I not have to back it up and test for truth?

And then there is the method of caculating bugs per line of code. They go into great detail about bug counts, when the fix was checked in, the lines of code, etc. But yet how do you measure importance? Some bugs are obviously greater than others. For instance, two teams create two identical applications. One application has 15 bugs and the other application has just 1. They both have the same lines of code, the same project size, same budget, everything is the same. The project with just one bug is obviously superior according to the methods they use, EXCEPT that particular bug allows a remote user to gain Root/Super User access. Which one has failed according to your quantitive data? Which project had the best method? They speak in depth about how this cannot be measured, then show you how they measured it?

Although I think this paper has good intentions and shows insight into some OSS projects,
1. The reference to commercial software as a whole is unfair and offers no value and
2. The method for caculating bugs is not an effective way to measure anything.

This paper is basically the equivilent of Microsoft, Oracle, Sun, or any other entity creating a study that never tests or proves anything and reaching a preconcieved notion. I can see this message being modded as a "troll" but oh well, this paper is not as scientific as it tries to appear.

Lower bug rate kind of a red herring

[dasmegabyte]

"Open source projects tend to have a lower bug-rate than commercial projects"

True. But open source projects are much more precisely targetted, and less functional. Not necessarily in a bad way, but in a way that is very different from marketable commercial software.

Take, for example, IIS vs. Apache. On one hand, yes, Apache is so much better at serving web pages -- faster, more stable, more secure, cheaper etc. But functionally, they don't really do the same thing. IIS encapsulates ASP scripting, database access, file security, web serving, ftp serving, mail serving and a very powerful management interface. Apache is just a core web server. It performs one small task out of dozens, and as such the developers can concentrate on making that work best.

It's hard to do the same with commercial software. You have to keep adding features to stay ahead of the competition -- merely having the fastest webserver is not enough, because hype sells servers, not actual results. For this reason, there are a lot of open source projects that would never survive as viable market solutions. Apache's one of them...considering that "all it does" is serve pages and it relies on "third party" modules to do anything fancier than powerful URL rewrites and server side includes, its market price would be low and thus the margins. Never mind that it's stable as a rock while IIS is as insubstantial as a fart in the wind.

This is a big problem with the adoption of open source as well. You can't just "switch" like you can from, say, Word to WordPerfect. If you use SQL Server and enterprise manager, for example, you can't just "switch" to MySQL. MySQL has no totalitarian interface on par with enterprise manager. It has no massive searchable help database and no "for dummies" option for managing jobs and indexes. If you plug in to MySQL with a SQL Server only toolset, you're in for a shock learning curve, even if the databases themselves are on par with each other and MySQL less buggy. The difference is that what's important to the MySQL developers isn't what sells SQL Server.