Open Source Studies

e8johan writes "Avaya Labs Research has presented a paper studying the open source process in the cases of Apache and Mozilla. They reach a number of interesting conclusions, the ones I find most interesting are: * Open source projects tend to have a core team of 10-15 coders, producing almost all code. The next layer is a set of developers submitting new features and bugfixes. The next layer is a set of advanced users submitting bug reports. * Open source projects tend to have a lower bug-rate than commercial projects. * Open source projects are generally quicker to respond to user requests. The article also discusses the differences between projects that have always been open source (such as Apache) and projects having a proprietary history (such as Mozilla)."

Comments

of course 15 coders makes for less bugs

[PissedOffGuy]

if these projects average 15 coders, on average they're also significantly less complex projects, and then of course on average theyll have less bugs.

also, if you have a team of people who are PAID to find bugs, theyll find more.

Re:of course 15 coders makes for less bugs

[roachmotel3]

Do you really think that projects like Apache, OpenOffice, Mozilla, Xfree86, and Linux are less complex than say, IIS, Word, IE, or Win98?

I mean, seriously, if you look at functionality, things are getting very close between the OSS world and the Microsoft world.

I'm not saying that there are many straight forward OpenSource Projects, but let's be real -- there is a complete OpenSource O/S, that runs and performs amazingly given a core team of 15 people.

Re:of course 15 coders makes for less bugs

[Tet]

if these projects average 15 coders, on average they're also significantly less complex projects

If you think that many software projects need more than 15 coders, you obviously don't have much experience of software development. In all my years of software development, I've never seen a core team bigger than 10 people. Sure, Word may well have 150 developers working on it. But don't for one second believe they're all actively coding Word in one big team. Most will be working in smaller groups of 4 or 5 on one specific area (for example, the spell checker).

Re:of course 15 coders makes for less bugs

[gpinzone]

Do you really think that projects like Apache, OpenOffice, Mozilla, Xfree86, and Linux are less complex than say, IIS, Word, IE, or Win98?

Yes. They are. Microsoft products don't just end at the product itself. They try to integrate their products into the OS. Internet Explorer just isn't a web browser. If it were, you'd never be able to have the level of security holes as it does. Without getting into a debate over whether or not it makes sense to integrate the browser into the OS, just realize that part of IE development is the OS development itself. On the whole, none of these projects are as complex.

Re:of course 15 coders makes for less bugs

[chris_mahan]

I think that you make a good point:

The microsoft products are more complex.

The open-source products are less complex.

I'll have to agree with you on the complexity level.

However, this is most likely due to the microsoft vs unix way of doing things. In unix, a lot of little programs accomplish a lot together. In the MS world, a few monolithic programs accomplish a lot together.

So, taking program to program, the MS ones would be most complex, since they try to put everything and the kitchen sink in every single one of their flagship products.

And so, consequently, these monolithic multi-million code lines programs are more difficult to design, engineer, maintain, and debug.

But there are fewer of them, so more resources (programmers, program managers) can be assigned to each.

In the unix world, there are many more programs. And while each of these programs individually may be less complex, less encompassing in scope, and have fewer features, as a group they are able to outperform microsoft's systems.

Since each program is smaller and more easily defined in scope and requirements, then each program takes fewer programmers to design, implement, and maintain. As a result, it is possible for a smallish dedicated team to design, implement, and enhance a valuable and usable piece of software.

For example, I was surprised two years ago to find out that the core team of Postgresql is relatively small (currently the steering committee and major developers combine to less than 25 people) and yet it is no small feat.

For the unix world, the difficulty comes when there are a great many programs in use that have been developed over the years by different groups of people with different methodologies. These programs must work with each others, store their files in known locations, etc. This is especially important in linux, because there are so many more programs (including multiple guis) that need to share the same directory tree, and each have varying degrees of dependencies.

So the complexity still exists, it just falls outside of the program itself, and is rather a product of the environment.

Of course, having had my share of windows dll hell, I realize the problem of software dependencies is OS-agnostic.

----

I think I'll get back to work now... Is it lunch yet?

I'm an advanced user...

[ajuda]

... and here is my bug sumission. It's spelled COMMERCIAL, not commersial I guess this submission was proprietary.

Not just open source

[aengblom]

"Never doubt that a small group of thoughtful citizens can change the world. Indeed, it is the only thing that ever has."

-- Margaret Mead

Primary Difference

[Anonymous Coward]

Open Source projects tend to have less men with whips walking around your cubical.

"We are slaves."

It coule be better

[bsharitt]

I bet most open source projects would be better if the 10-15 core coders were paid as full time employees. Of course this would require compaines to back them, which is easier said than done.

Re:It coule be better

[JoeBuck]

For some of the best-known free software projects, particularly the Linux kernel and GCC, most of the core coders are paid to work on free software, either full-time or part-time.

Re:It coule be better

[Soko]

Does this make business sense? In a word,

Yes.

Paying people to work on an OSS project is strategic for any business for the following reasons:

1. You get in on the ground floor. If your competitors have core developers on an important Open Source project, you are at a disadvantage. They will have intimate knowledge of the product, since they helped generate the source code. You, on the other hand will have to read and decode the source. You then spend more time - and $ - getting up to speed in supporting your end users. Having a core developer means you're that much closer to the information you need.

2. Standards compliance. If you and 2 or 3 of your competitors are all working on an OSS product, it will become a standard, since everyone has to agree on the functionality of the package. It is impossible to do otherwise. Basically, you all agree to detente - you permenately remove a weapon from the arsenal. This stops an expensive "arms race", and also means less things to worry about and/or reverse engineer. Interoperability is assured then, so it means $ can be spent in more constructive ways.

3. Wealth of focused resources. A compelling Open Source product will attract the brightest users who have a vested interest in your product. The quality of bug reports usually is better and more diverse. For that matter, the coders outside of your organisation also will have a vested interest in your product, since they aren't motivated by getting paid to submit changes and bug fixes. This means development is focused on getting the right product in user's hands - not what marketing thinks is the right stuff. The feedback loop from the field is much better. Less $ wasted on dead end products, or going down the wrong development path.

4. Marketing. If your organisation starts an OSS project and keeps/pays the lead developers, your name is attached to it, even if others contribute to the code. Everyone knows that JFS is and IBM product, as well that XFS is an SGI product. They just happen to be OpenSource. This doesn't result in tangeable $, but other things that can lead to more $ - like goodwill from the development community, end users and sometimes even (gasp!) your competition.

5. Undercutting the competition. If your OSS project provides the same funtionality as a competitors closed source product at the same quality level (most indications state quality will be above Closed Source), you've effectively removed most of the reasons that your competitions product will be bought. If you're paying 2 developers and have 10 other regularily contributing code an OSS product under the GPL, but a competing product needs 20 developers to code (and untold others to support the product and the codrs too), well the math is easy. Cut throat, but effective business strategy.

There are likely many other benefits that come from "owning" OSS code, but I'll stop here for brevities sake.

Soko

P.S. - I haven't listed the downsides, since we, unh, know there aren't any /sarcasm ;-)

Re:It coule be better

[Evan]

Lest anyone think that you're full of theory, here's a concrete example: Zope Corporation and the Zope application server.

Quick background: ZC wrote Zope, released it under a GPL-compatible license, and makes money from large (>$100,000) contracts to build applications using Zope.

1. ZC's engineers know the code better than anyone else, giving them an edge over other contractors who do Zope.

2. There are no direct competitors involved, but customers and community members alike ensure that Zope supports standards (WebDAV, XMLRPC, etc).

3. Several major features of Zope were developed by community members, then adopted into the core. This led directly to some of the authors joining the company.

4. ZC was originally Digital Creations, Inc. They finally changed their name because the association was so valuable. Most of the community called the company "Zope" already anyhow .

5. The fact that every single dollar of a contract with ZC is spent on custom development, and none on licences, is a *huge* marketing advatage.

6. If ZC hadn't made Zope Free, they would never have been able to compete with the likes of BEA and Vignette. The Zope community, and the fact that Zope solutions don't depend on ZC for maintenance, put Zope in a league that no penny-ante proprietary startup could reach.

Complexity != number of coders

[MosesJones]

Get thee to the bible of all things Software The Mythical Man Month. Take 15 motivated and talented indivduals (they have to be both) and they will whoop the arse off a team ten times their size which is filled with average people.

Adding more people _makes_ a project more complex but not in terms of the problem being solved, it makes it more complex because there is more communication and communication is not always accurate, the more communication the more bugs. Its no suprise when you look at some elements of large OSS projects that you see that PersonX does everything on Y, its this "master" concept that helps them deliver. And of course in having an excessively large testing team by commercial standards, testers out-numbers codes by huge ratios, any one been on a commercial project where there was even parity.

OSS is the best way to run a paid or unpaid project IMO, the problem is that it looks so expensive on the surface the companies don't do it. But the Total Cost of Ownership is much higher because of the lack of testing and the lack of review, and of course because instead of 15 developers and 100 testers they have 100 developers and 3 testers, 6 managers, 1 programme manager, two account managers, one account director and two administration assistants.

The common factor between OSS and standard commercial is that no-one does enough documentation.

PDF alert!

[Draoi]

That link points to a pdf file. Pity the /. troll link filter doesn't catch pdf's.

Anyways Adobe has a pdf translation engine here. Just punch in the URL ...

OSS as an alternative

[OmniVector]

Open source software has been in my mind more of a philsophical debate than one of software production. It seems like computer science mimics things a lot in regular science. A new *thing* is discovered, and becomes a widely used standard incorporated into other programs (aka inventions) and it becomes part of the market place.

According to the article: Proponents claim that OSS software stacks up well against commercially developed software both in quality and in the level of support that users receive...

In many ways this is true, but coming from me, someone who is trying to switch from windows to linux, help is a lot harder to come by than they claim. I've relied much on my friends who have used linux to help me get my system running, and without their help I would have spent weeks on google, newsgroups, forums, doc, and man pages just to get things as simple as my audio drivers for my laptop working.

Support for OSS is minimal at best, and that's to be expected. When you have to pay for software, someone is payed to answer phone calls, to write thorough docs.. because it is their JOB. I know a lot of people, such as those 10-15 dedicated developers like the article says, can do a lot when it comes do documentation and support, but companies beat them hands down in this department. That is a big problem, there needs to be a better system. The irony there is if you make linux easier to use you lose the power of customizing your kernel, or optimizing programs by compiling them on your machine, etc.

If something isn't done though, OSS software will always take more time to setup than commercial software.

Re:OSS as an alternative

[grumpygrodyguy]

Actually, my experience with OSS support has been tons better than commercial software. I can email the developers, who often respond in minutes with an answer.

No no, I think you're missing the point. You're already "inside" the Linux community, he's talking about the other 99.99% of the human population. The issue here is useability.

He's not asking how to tweak the source code to get something to compile...he's asking what your mother would ask, what your father would ask, what your brother or sister would ask, and probably what almost everyone living on your street would ask. How can I get my soundcard to work in Linux?

OSS devs are the worst people to ask for help. When you say computers or Linux they start visualizing C code, and talking editors and compilers...they are someplace else entirely. I'm no fan of Microsoft, or AOL...but the main reason why they are successful is not because they sell a better product, but because they sell a product that people can actually use.

Re:OSS as an alternative

[iabervon]

I've relied much on my friends who have used linux to help me get my system running

But that has to be included in the support for OSS. In fact, that's the only kind of support that most people get with just about anything technology-related. Furthermore, it's the best support, because the person actually knows you, and both understands what you're saying and cares that you get it working. With commercial software, you often have to deal with people who don't care if your problem gets solved, so long as they get paid; furthermore, it's much harder to find someone who can actually fix something that's broken.

Customizability and ease of use are not actually in opposition at all; you just need to have the defaults set right. Each new option which gets set, by default, by looking at your usage, improves both ease of use and customizability. Local compilation doesn't make things more difficult, because it can be done without any interaction; there's no reason that, when you download a binary and install it, the system couldn't download the source and compile it in the background, and then replace the binary installation with the compiled one. In fact, compiling a program locally is much likely to work than using a precompiled binary, and compiling most programs doesn't take as long as reading their documentation on recent hardware.

BTW, the unacceptably-slow Linux installation took less time than Windows ever has.

Conflict?

[lseltzer]

One of the authors, Roy Fielding, is on the Apache Board of Directors. I haven't read the paper yet and I'm sure he can be objective, but still.

Much like closed source

[mikewas]

The number of developers that are actually contributing seems much like the commercial closed-source projects I've worked on. There's always a small team that really understands the code that does most of the work. The core tends to be about half of the team on small projects. Everybody else performs ancillary functions or just goofs off. On larger projects communication breaks down and the core is limited to a no more than a few subgroups of one to 7 developers each. The subgroups tend to work independently, only occasionally interacting with one another -- usually though some spokesman or leader.

So open source group dynamics are similar to closed source projects. Not really surprising, since both are staffed by people!

Larger more formalized projects, aerospace for example, improve on the above by making subgroups of subgroups. This layering of project & program management really increases the overhead. It seems to slow things down, but at the end you can put things together and have a hope of making it work. It's really a formalization & extension of the way we organize ourselves naturally.

Google HTML version

[kisrael]

HTML version via since the original is slashdotted and a PDF anyway.

What About OSS Failures?

[theduck]

It's all well and good to focus on the characteristics of a successful OSS project. It's extremely interesting that these projects succeeded without elements that are considered to be fundamental to success in commercial development. However, studying success provides only half the picture. It won't tell you what things to avoid that tend to make OSS projects fail. Without that knowledge, attempts to reproduce and improve upon the methods used by Apache and Mozilla will experience unforseen failures that could have been avoided.

Re:What About OSS Failures?

[zericm]

Not only that, but I'd like to see projects that have deadlines that HAVE to be met, as opposed to your hobby code that's finished when its finished.

I'm working on a closed source project -- a vendor product with a huge amount of customized code -- that has some hard deadlines: first release for testing by mid-August; pilot by mid-September; general release by November 1. Except that we ran into problems getting the team up to speed, so we cut the scope of the project, and decided to skip the august release. And then we ran into more delay and we moved the pilot to the end of September. Then we decided to push the pilot to mid-October. Now we are skipping the pilot and going straight to general release in November.

I can hardly wait for the cries of incompetence, but I respond with the real-world: lay-offs happen. Or key people get new jobs. Or reorgs interfere. Or the business users change the scope (find me a company where business users are ignored). I have as yet to see a plan that accounts for all of these items.

I've worked in large corporations since 1993, as both a programmer and tech lead, with mature and immature development teams. Development is about negotiation. Dates and deliverables are constantly re-evaluated. If a project date can slip then it will slip. If the date is hard, then the scope is cut. In other words, closed source projects are finished when they are finished.

Up to old tricks

[gnovos]

It is often characterized as a fundamentally new way to develop software that poses a serious challenge to the commercial software businesses that dominate most software markets today.

I was under the impression that this kind of approach to building is a fudamentally old way of getting the job done. The Homebrew Computer Club essentally built everything "open source" (well, it was hardware mostly, but same approach). The current resurgence of OSS is not something new and revolutionary, it is instead a rediscovery of old techniques that were coopted by big business.

Open source projects tend to have a lower bug-rate

[vlad_petric]

I quite agree for Apache, but Mozilla ?? Let's stay in the realm of reason.

The Raven

One thing the report forgot to mention ...

[Taco Cowboy]

The report mentioned many things that we already know. But there's one important thing about the Open Source software the report may have missed :-

The freedom to change / customize the software via the source code.

People can argue that even Windoze can be customized - background, for example - but it ain't the same as cusomization via source code.

Do you ever have the feeling, when you use commercial / close-source software, that some part of it are kinda stupid, cumbersome, or simply plain assinine ?

Do you ever think that if you _just_ have the source code, perhaps you could do some change to it, at least to better suit your taste ?

Well ... With Open Source, we can.

Now, of course, not every one know how to code, and even fewer of us know how to tweak the code to our own liking. But that doesn't change the point that with Open Source, we _can_ change the software anyway we like it.

It's a feeling of having TOTAL CONTROL over the software.

It's a feeling of empowerment.

It's just _the_ thing close-sourced software users don't get to enjoy.

"Don't confuse me with facts..."

[dpbsmith]

I'm sure the study will have very little effect on software development practices.

It reminds me of the study cited in DeMarco and Lister's "Peopleware" on the relation between schedule setting and productivity. They compared programmer productivity under four regimes: schedule set by the manager; schedule set by the programmer; schedule set by a neutral third party; and no schedule. The first three alternatives were tightly bunched, with "schedule set by the manager" producing the worst results (but only by a small amount). The fourth, no schedule, result in more than double the productivity of any of the others.

This book has been out for at least a decade, but as far as I know it has not led to the adoption of schedule-free development anywhere...

Re:Open source projects tend to have a lower bug-r

[dhogaza]

The metric they used was bug density, not the absolute number of bugs. In other words, number of reported defects per N lines of code.

Mozilla is a far larger project than the Apache core, so given an equivalent number of bugs per N lines of code you will see a far larger number of bugs.

They did report that to some extent the measurement of bug density wasn't necessarily directly comparable due to the different state of the projects at the time the report was written (Apache == stable, Mozilla == pre-release). If you're interested in more details read the paper yourself ...

Re:Not to be obvious...

[gpinzone]

It depends how you define successful. Yes, Apache and Mozilla are great products, but if there are so great, why aren't people dropping their closed source software and downloading their open source counterparts in droves? Hell, the two examples given are not only open source, but they're free!

Obviously it all can't be a success. How about the downsides? What about time to market? How long did Mozilla take to deliver a 1.0? What about lack of common features that customers want? (When I say customers, I mean the target audience as a whole, not just the geek community.)

critical mass most important element

[truth_revealed]

If your open source project does not have a critical mass of developers it will fail. Critical mass can only be acheived if the program being developed is largely finished or already useful in some way. Everyone likes to back a winner. The exception to this rule is when you have a project started by someone with past successes such as Miguel starting the Mono project after the successful GNOME project. But then you could argue that C# was already a mature specification for Mono to exploit, so it was already a "winner" in a sense. The Parrot project, by comparison has the disadvantage of not having pristine design specification and documentation (courtesy of Microsoft) to mimic.
Often, the majority of work in a pre-critical mass project has already been done by a single individual. This is conveniant because they usually become the defacto project leader. It is just as useful to prevent poor contributions into the project as it is to add quality code to the project. This is due to the fact that no one likes code audits.
And the final mark of a good open source project - the leader cannot be a prick. There's too much bullshit to put up with in day to day life, why would you want to get harassed without being payed?

Re:Not to be obvious...

[timeOday]

Yes, Apache and Mozilla are great products, but if there are so great, why aren't people dropping their closed source software and downloading their open source counterparts in droves? Hell, the two examples given are not only open source, but they're free!

Apache seems an odd example for you to cite. It's the #1 webserver on the planet - in other words, people *are* downloading it in droves. As for Mozilla, remember that its entrenched competitor is also "free."

Re:It's all about the QA!

[grumpygrodyguy]

This is probably the most profound statement about OSS I've seen in this discussion.

OSS projects are not better because the coders are more talented or devoted than closed source projects. They are better because they actually have QA resources that cannot be matched by close source projects.

Yes, but on the flipside OS software is like a box of chocolates, you never know what you're going to get.

Without reliability, business won't endorse open source software. So what ends up happening is you have hundreds of so-called "advanced users" who are basically hobbiests with a sense of adventure spending thier time using faulty software...so that eventually the rest of us will get a useful product. Sure it's thorough, and the bugs do get found...but there are side-effects to this.

For a lot of us who were using Mozilla in the beta stages, it was a complete mess. I remember one bug that was just so rediculous it rendered the entire browswer unusable. As I recall the only way to type a URL into the browser address window was to doubleclick on the text, and hold down the mouse while typing the URL. It was maddening and I stopped using it after a few days. The bug was addressed, and fixed in the next version. But it wasn't for another 8 months(long after the 1.0 release) that I picked up and tried mozilla again.

The point is, I wanted a web browser, not an adventure. Most users want things they can trust. They want working e-mail, office applications, web browsers, etc. Businesses are even more fickle about this. In a business environment software has to be reliable or it could end up costing millions, there is just no room for "software speculation".

OSS projects like Apache and the Linux kernel are stable because they have enjoyed years of meticulous care. I say care because in those cases the user and the developer were the same person. But where are you going to find daring OSS beta-testers for office applications and enterprise level software? These are products that need to work now. 5 years of screw-ups may produce a good product in the end...but most companies can't afford those 5 years.

Re:Not to be obvious...

[5KVGhost]

OpenSource development is more successful because the people involved love what they're doing.

Unfortunately that implies a disadvantage, too: Things they don't love doing often don't get done at all.

Re:Not to be obvious...

[AlienArtifact]

I think there are several factors to consider when evaluating the adoption trends for open source software among businesses and organizations.

Quality: Many businesses make the assumption that commercial software is more rigorously developed and tested than open source software. The rationale here is that a commercial software vendor has an economic incentive to ensure quality and correct problems. There is also an expectation of accountability from an organization that exchanges products for currency.
Support: When investing in a mission critical software package, an organization wants to rely on the vendor to provide assistance and support that may be beyond the capabilities of its own staff. This is coupled with a desire for fast response to critical problems - if your company's livelihood depends on your webserver being up, you want to partner with a vendor that can deliver immediate service and support.
Integration: In today's computing landscape, software does not exist in isolation - software is expected to interface and collaborate with other packages and components that are part of the portfolio of an organization. Unfortunately, integrating diverse software components is a daunting and error-prone task. Organizations tend to favor vendors that provide installation and configuration support as well as consulting services geared toward integrating their products with the systems they already have deployed.
Maintainability: Organizations make software acquisition decisions based on the long-term. From their perspective, they want to invest in a "platform" that has long-term viability. This viewpoint is often associated with buying applications from an established vendor who they expect to be stable and viable enough to last the long-haul.
Usability: When a company makes a purchasing decision for a software package, one of the factors considered is whether the software is usable. This includes how easy it is to configure and maintain, the quality of the GUI (if any), the ease with which you can understand how the software operates, and so on. Open source software, is often viewed as being written "by hackers for hackers". In reality, open source software usually IS targeted to the upper echelon of software users, making it more difficult for organizations lacking highly experienced staff from being able to adequately deploy and use such software.

There are many other factors (both real and perceptual) that impact organization decision-making when it comes to open source software. Vendors tend to market their products to corporate decisions makers aggressively, whereas open source software does not. In many cases the merit of an software solutions is less important than the "relationship" between the IT decision makers and vendors. Not all organizations are aware of the wealth of open source solutions available to them in virtually all domains. And, clearly, if you don't know something exists you won't use it.

The reality is that much open source software meets the quality, support, integration, maintainability, and usability requirements that organizations demand. In many cases, open source software is actually better than the commercial alternatives. The free availability of source code makes it possible for the user community to find more of the potential problems and defects than in closed, black-box commercial alternatives. Skilled consumers can actually submit fixes to the community maintaining the open source solution making turn-around of bug fixes considerably faster. Open source software tends to be more compliant with other industry trends and open source alternatives, vs. the proprietary NIH (not invented here) mentality prevalent among large commercial software vendors.

This is not to say that open software doesn't have its downside. The availability of source code makes it possible for hackers to discover holes and back-doors more easily than with closed commercial software. Installation and configuration is often a tricky, complicated proposition. You can't get an SLA for an open source program. Maintaining, integrating an operating open source software often requires a higher class of user or IT professional than shrink-wrapped software.

Organizations need to learn to better recognize and weigh the benefits of open source against its downside. And while there is a place for both open source and commercial applications in the business environment, organizations need to overcome the "false" impressions they hold of open source software and leverage its benefit.

Re:It's all about the QA!

[Zathrus]

You're a new coder, aren't you?

In most industries the QA department is a joke. A really bad one. QA is the last thing budgeted and the first thing cut -- because when you have a deadline to hit management always thinks that QA is superfluous. I've heard VP's state that it's better to have a buggy product out than no product at all. And the VP in question really didn't care just how buggy it was. With that kind of attitude toward release quality QA can be viewed as nothing but an impediment to getting product out (and, to be fair, some QA departments do view that as their job - preventing anything from being released). So some QA departments just rubber stamp things instead of doing real testing.

On top of that getting good QA people is extremely difficult. A good QA tester has to have enough technical expertise to design test programs, methods, and sets themselves (having the coders do this defeats the purpose of QA), but doesn't want to be a coder fulltime. That particular combination of abilities and desires is very rare indeed.

Finally, and perhaps worst of all, a good QA department is invisible. If QA has done their job then there will be minimal complaints from end users -- sure, there will be issues, but nothing huge. When you never have any huge problems, management tends to forget that it was QA that caught those huge problems before the product shipped.

Where does QA work? Usually in industries like Aerospace, Medical, telecomm, and power generation -- industries which don't have a margin for error. They have decades old QA practices that often got instituted the hard way. They also have relatively little competition and insanely high development costs.

Frankly, OSS does a far better job of QA than most closed shops, because the QA team is not paid by the same company/group developing the software. So there's no incentive to not report a bug, but there's also no incentive to block release -- if a major bug is revealed then Joe Blow user isn't going to get fired for failing to find it.

Oh, I'm sure you won't agree with me. Get another decade or so of experience in the real world and I suspect that you'll think differently.

FUD

[e2d2]

Wow, FUD. According to our study we created to reach our preconcieved notion open source methods are superior.

Where to start?

Open source projects are generally quicker to respond to user requests.
I'm sorry but comparing two open source products to "commercial products" which is who? what product? what project? I don't see any quantative data besides a few lines refering to commercial products as a whole and saying the authors have experience with them is not scientific. I take exception to this because the paper sepcifically tries to appear scientific but yet offers no data comparing either project referenced (Apache and Mozilla) to a commercial counterpart or their ability to respond to bugs.

Open source projects tend to have a lower bug-rate than commercial projects
Again, where is the data? I see the scietific method they use for tracking bugs per line of code and they go into great detail comparing the two projects but yet we see no comparison to commercial project bug counts or the same method applied to commercial projects. The paper is laced with phrases such as "One might speculate". Yeah one might. Of course one might not speculate and offer evidence. If I create a hypothesis should I not have to back it up and test for truth?

And then there is the method of caculating bugs per line of code. They go into great detail about bug counts, when the fix was checked in, the lines of code, etc. But yet how do you measure importance? Some bugs are obviously greater than others. For instance, two teams create two identical applications. One application has 15 bugs and the other application has just 1. They both have the same lines of code, the same project size, same budget, everything is the same. The project with just one bug is obviously superior according to the methods they use, EXCEPT that particular bug allows a remote user to gain Root/Super User access. Which one has failed according to your quantitive data? Which project had the best method? They speak in depth about how this cannot be measured, then show you how they measured it?

Although I think this paper has good intentions and shows insight into some OSS projects,
1. The reference to commercial software as a whole is unfair and offers no value and
2. The method for caculating bugs is not an effective way to measure anything.

This paper is basically the equivilent of Microsoft, Oracle, Sun, or any other entity creating a study that never tests or proves anything and reaching a preconcieved notion. I can see this message being modded as a "troll" but oh well, this paper is not as scientific as it tries to appear.

Lower bug rate kind of a red herring

[dasmegabyte]

"Open source projects tend to have a lower bug-rate than commercial projects"

True. But open source projects are much more precisely targetted, and less functional. Not necessarily in a bad way, but in a way that is very different from marketable commercial software.

Take, for example, IIS vs. Apache. On one hand, yes, Apache is so much better at serving web pages -- faster, more stable, more secure, cheaper etc. But functionally, they don't really do the same thing. IIS encapsulates ASP scripting, database access, file security, web serving, ftp serving, mail serving and a very powerful management interface. Apache is just a core web server. It performs one small task out of dozens, and as such the developers can concentrate on making that work best.

It's hard to do the same with commercial software. You have to keep adding features to stay ahead of the competition -- merely having the fastest webserver is not enough, because hype sells servers, not actual results. For this reason, there are a lot of open source projects that would never survive as viable market solutions. Apache's one of them...considering that "all it does" is serve pages and it relies on "third party" modules to do anything fancier than powerful URL rewrites and server side includes, its market price would be low and thus the margins. Never mind that it's stable as a rock while IIS is as insubstantial as a fart in the wind.

This is a big problem with the adoption of open source as well. You can't just "switch" like you can from, say, Word to WordPerfect. If you use SQL Server and enterprise manager, for example, you can't just "switch" to MySQL. MySQL has no totalitarian interface on par with enterprise manager. It has no massive searchable help database and no "for dummies" option for managing jobs and indexes. If you plug in to MySQL with a SQL Server only toolset, you're in for a shock learning curve, even if the databases themselves are on par with each other and MySQL less buggy. The difference is that what's important to the MySQL developers isn't what sells SQL Server.

Re:Not to be obvious...

[Gerry Gleason]

Taking just Mozilla, since Apache is the most successful product in its category. Mozilla is only recently past 1.0, and the paper (if anyone read it, I just finished and I see over 100 comments) was analysing it before 1.0 came out. Nobody expects OSS development to be as fast as commercial, and it doesn't have to be. OSS is going to be much more concerned with quality, stability than artificially aggressive deadlines.

Also note the fast, and in some cases parallel development of derivative products. I don't have the details, but there are a host of 'Gekko' based browsers, and the direct spin-off Phenix is proceeding very fast indeed. People are reporting it to be useable and fast at the 0.2 release.

There is one issue to worry about (from the paper). One hypothesis is that if a project doesn't achieve critical mass, it won't get enough of a user following to get the many eyes effect. I suspect that this may be weakened by a number of factors. Even if a project doesn't acheive critical mass, it may be reworked in another form because the code is still available for experimentation.

applying free market principles to software dev

[g4dget]

Open source is really in many ways what happens when you apply free market principles to software development: people participate, and their contributions survive, based on actual needs, capabilities, and quality. If open source projects fail to meet the needs of their users, they split, and some branch will adapt, but without throwing everything away.

In contrast, closed source development usually involves assigning people to projects. Their primary motivation isn't the software itself (which they will likely never use), but their job and their stock. Determining features involves a few people guessing hard about what features end users may or may not want. Oh, sure, they listen, but as anybody who has gathered requirements knows, users generally aren't very good at communicating what tradeoffs are important to them. And when closed source projects fail in the market, any new entrant has to start from scratch.

It's not surprising that open source development is winning in the long run. It's central planning (Microsoft, Apple) vs. a free market of ideas (open source) all over again. And we already have a good idea which of the two approaches of organizing large numbers of people around a common goal works better.