Open Source Studies

e8johan writes "Avaya Labs Research has presented a paper studying the open source process in the cases of Apache and Mozilla. They reach a number of interesting conclusions, the ones I find most interesting are: * Open source projects tend to have a core team of 10-15 coders, producing almost all code. The next layer is a set of developers submitting new features and bugfixes. The next layer is a set of advanced users submitting bug reports. * Open source projects tend to have a lower bug-rate than commercial projects. * Open source projects are generally quicker to respond to user requests. The article also discusses the differences between projects that have always been open source (such as Apache) and projects having a proprietary history (such as Mozilla)."

Comments

Not to be obvious...

[roachmotel3]

But it's good to hear it reaffirmed from an outside source what many of us know to begin with -- OpenSource development is more successful because the people involved love what they're doing.

Re:Not to be obvious...

[gpinzone]

It depends how you define successful. Yes, Apache and Mozilla are great products, but if there are so great, why aren't people dropping their closed source software and downloading their open source counterparts in droves? Hell, the two examples given are not only open source, but they're free!

Obviously it all can't be a success. How about the downsides? What about time to market? How long did Mozilla take to deliver a 1.0? What about lack of common features that customers want? (When I say customers, I mean the target audience as a whole, not just the geek community.)

Re:Not to be obvious...

[Lazar Dobrescu]

As far as I can tell, people ARE downloading apache, and using it more than it's closed source counterparts...

Re:Not to be obvious...

[5KVGhost]

OpenSource development is more successful because the people involved love what they're doing.

Unfortunately that implies a disadvantage, too: Things they don't love doing often don't get done at all.

Re:Not to be obvious...

[AlienArtifact]

I think there are several factors to consider when evaluating the adoption trends for open source software among businesses and organizations.

Quality: Many businesses make the assumption that commercial software is more rigorously developed and tested than open source software. The rationale here is that a commercial software vendor has an economic incentive to ensure quality and correct problems. There is also an expectation of accountability from an organization that exchanges products for currency.
Support: When investing in a mission critical software package, an organization wants to rely on the vendor to provide assistance and support that may be beyond the capabilities of its own staff. This is coupled with a desire for fast response to critical problems - if your company's livelihood depends on your webserver being up, you want to partner with a vendor that can deliver immediate service and support.
Integration: In today's computing landscape, software does not exist in isolation - software is expected to interface and collaborate with other packages and components that are part of the portfolio of an organization. Unfortunately, integrating diverse software components is a daunting and error-prone task. Organizations tend to favor vendors that provide installation and configuration support as well as consulting services geared toward integrating their products with the systems they already have deployed.
Maintainability: Organizations make software acquisition decisions based on the long-term. From their perspective, they want to invest in a "platform" that has long-term viability. This viewpoint is often associated with buying applications from an established vendor who they expect to be stable and viable enough to last the long-haul.
Usability: When a company makes a purchasing decision for a software package, one of the factors considered is whether the software is usable. This includes how easy it is to configure and maintain, the quality of the GUI (if any), the ease with which you can understand how the software operates, and so on. Open source software, is often viewed as being written "by hackers for hackers". In reality, open source software usually IS targeted to the upper echelon of software users, making it more difficult for organizations lacking highly experienced staff from being able to adequately deploy and use such software.

There are many other factors (both real and perceptual) that impact organization decision-making when it comes to open source software. Vendors tend to market their products to corporate decisions makers aggressively, whereas open source software does not. In many cases the merit of an software solutions is less important than the "relationship" between the IT decision makers and vendors. Not all organizations are aware of the wealth of open source solutions available to them in virtually all domains. And, clearly, if you don't know something exists you won't use it.

The reality is that much open source software meets the quality, support, integration, maintainability, and usability requirements that organizations demand. In many cases, open source software is actually better than the commercial alternatives. The free availability of source code makes it possible for the user community to find more of the potential problems and defects than in closed, black-box commercial alternatives. Skilled consumers can actually submit fixes to the community maintaining the open source solution making turn-around of bug fixes considerably faster. Open source software tends to be more compliant with other industry trends and open source alternatives, vs. the proprietary NIH (not invented here) mentality prevalent among large commercial software vendors.

This is not to say that open software doesn't have its downside. The availability of source code makes it possible for hackers to discover holes and back-doors more easily than with closed commercial software. Installation and configuration is often a tricky, complicated proposition. You can't get an SLA for an open source program. Maintaining, integrating an operating open source software often requires a higher class of user or IT professional than shrink-wrapped software.

Organizations need to learn to better recognize and weigh the benefits of open source against its downside. And while there is a place for both open source and commercial applications in the business environment, organizations need to overcome the "false" impressions they hold of open source software and leverage its benefit.

Re:Not to be obvious...

[gmuslera]

All right, so open source is used more for web serving and penguin downhill racing simulators. Anything else?

Most of internet infrastructure is open source based, not only web serving, think in DNS (bind most used DNS server, by far) or mail serving (sendmail, qmail, postfix are used by more than 50% of the mail servers, and probably each one of them is more used alone than the most sucessful closed source counterpart).

of course 15 coders makes for less bugs

[PissedOffGuy]

if these projects average 15 coders, on average they're also significantly less complex projects, and then of course on average theyll have less bugs.

also, if you have a team of people who are PAID to find bugs, theyll find more.

Re:of course 15 coders makes for less bugs

[roachmotel3]

Do you really think that projects like Apache, OpenOffice, Mozilla, Xfree86, and Linux are less complex than say, IIS, Word, IE, or Win98?

I mean, seriously, if you look at functionality, things are getting very close between the OSS world and the Microsoft world.

I'm not saying that there are many straight forward OpenSource Projects, but let's be real -- there is a complete OpenSource O/S, that runs and performs amazingly given a core team of 15 people.

Re:of course 15 coders makes for less bugs

[Tet]

if these projects average 15 coders, on average they're also significantly less complex projects

If you think that many software projects need more than 15 coders, you obviously don't have much experience of software development. In all my years of software development, I've never seen a core team bigger than 10 people. Sure, Word may well have 150 developers working on it. But don't for one second believe they're all actively coding Word in one big team. Most will be working in smaller groups of 4 or 5 on one specific area (for example, the spell checker).

Re:of course 15 coders makes for less bugs

[gpinzone]

Do you really think that projects like Apache, OpenOffice, Mozilla, Xfree86, and Linux are less complex than say, IIS, Word, IE, or Win98?

Yes. They are. Microsoft products don't just end at the product itself. They try to integrate their products into the OS. Internet Explorer just isn't a web browser. If it were, you'd never be able to have the level of security holes as it does. Without getting into a debate over whether or not it makes sense to integrate the browser into the OS, just realize that part of IE development is the OS development itself. On the whole, none of these projects are as complex.

Not just open source

[aengblom]

"Never doubt that a small group of thoughtful citizens can change the world. Indeed, it is the only thing that ever has."

-- Margaret Mead

It coule be better

[bsharitt]

I bet most open source projects would be better if the 10-15 core coders were paid as full time employees. Of course this would require compaines to back them, which is easier said than done.

Re:It coule be better

[hyperturbopete]

actually thats an interesting observation.

The core team is 10-15 people who arent even full time! wow :-)

A lot of companies are realizing (or should realize), though, that its a great deal to pay one of their employees (or contract out) to take an OSS project which is almost-right-for-them, and add the last 10% of missing functionality, etc. If they play their cards right, their one developer can leverage all the volunteer expertise out there and work with a huge part-time team backing him.

This can be better than paying for 100% of a commercial alternative.

Thus, many of the more focused OSS developers are actually on someone's payroll.

-peter

Complexity != number of coders

[MosesJones]

Get thee to the bible of all things Software The Mythical Man Month. Take 15 motivated and talented indivduals (they have to be both) and they will whoop the arse off a team ten times their size which is filled with average people.

Adding more people _makes_ a project more complex but not in terms of the problem being solved, it makes it more complex because there is more communication and communication is not always accurate, the more communication the more bugs. Its no suprise when you look at some elements of large OSS projects that you see that PersonX does everything on Y, its this "master" concept that helps them deliver. And of course in having an excessively large testing team by commercial standards, testers out-numbers codes by huge ratios, any one been on a commercial project where there was even parity.

OSS is the best way to run a paid or unpaid project IMO, the problem is that it looks so expensive on the surface the companies don't do it. But the Total Cost of Ownership is much higher because of the lack of testing and the lack of review, and of course because instead of 15 developers and 100 testers they have 100 developers and 3 testers, 6 managers, 1 programme manager, two account managers, one account director and two administration assistants.

The common factor between OSS and standard commercial is that no-one does enough documentation.

Re:Complexity != number of coders

[plalonde2]

I disagree that open source is the best way to run a paid project. Not because the open source model doesn't work, but because it's hard enough to find and retain a core of 10-15 highly motivated, highly talented people; but to keep them interested on payroll to set deadlines is darned near impossible.

Open source developpers don't have the business constraints of hard deadline, make their own hours and release dates and take serious breaks off their projects.

Fundamentally, businesses need to manage their risk, deadlines, and content, in a much tighter way than open source projects allow.

OSS as an alternative

[OmniVector]

Open source software has been in my mind more of a philsophical debate than one of software production. It seems like computer science mimics things a lot in regular science. A new *thing* is discovered, and becomes a widely used standard incorporated into other programs (aka inventions) and it becomes part of the market place.

According to the article: Proponents claim that OSS software stacks up well against commercially developed software both in quality and in the level of support that users receive...

In many ways this is true, but coming from me, someone who is trying to switch from windows to linux, help is a lot harder to come by than they claim. I've relied much on my friends who have used linux to help me get my system running, and without their help I would have spent weeks on google, newsgroups, forums, doc, and man pages just to get things as simple as my audio drivers for my laptop working.

Support for OSS is minimal at best, and that's to be expected. When you have to pay for software, someone is payed to answer phone calls, to write thorough docs.. because it is their JOB. I know a lot of people, such as those 10-15 dedicated developers like the article says, can do a lot when it comes do documentation and support, but companies beat them hands down in this department. That is a big problem, there needs to be a better system. The irony there is if you make linux easier to use you lose the power of customizing your kernel, or optimizing programs by compiling them on your machine, etc.

If something isn't done though, OSS software will always take more time to setup than commercial software.

Re:OSS as an alternative

[Anonymous Coward]

Well, gee, when last have you phoned Microsoft for tech support? Probably never. So, what exactly is different with OSS tech support?

Re:OSS as an alternative

[michaelggreer]

Actually, my experience with OSS support has been tons better than commercial software. I can email the developers, who often respond in minutes with an answer. Also, I do not need to navigate some support labrynth just to see the mailing list of questions, or the FAQ. Commercial apps have little on OSS, from my experience. The best source for debugging Weblogic problems is not sitting on the phone for 2 hours, but going to the mailing lists, like an OSS project.

Re:OSS as an alternative

[grumpygrodyguy]

Actually, my experience with OSS support has been tons better than commercial software. I can email the developers, who often respond in minutes with an answer.

No no, I think you're missing the point. You're already "inside" the Linux community, he's talking about the other 99.99% of the human population. The issue here is useability.

He's not asking how to tweak the source code to get something to compile...he's asking what your mother would ask, what your father would ask, what your brother or sister would ask, and probably what almost everyone living on your street would ask. How can I get my soundcard to work in Linux?

OSS devs are the worst people to ask for help. When you say computers or Linux they start visualizing C code, and talking editors and compilers...they are someplace else entirely. I'm no fan of Microsoft, or AOL...but the main reason why they are successful is not because they sell a better product, but because they sell a product that people can actually use.

Re:OSS as an alternative

[iabervon]

I've relied much on my friends who have used linux to help me get my system running

But that has to be included in the support for OSS. In fact, that's the only kind of support that most people get with just about anything technology-related. Furthermore, it's the best support, because the person actually knows you, and both understands what you're saying and cares that you get it working. With commercial software, you often have to deal with people who don't care if your problem gets solved, so long as they get paid; furthermore, it's much harder to find someone who can actually fix something that's broken.

Customizability and ease of use are not actually in opposition at all; you just need to have the defaults set right. Each new option which gets set, by default, by looking at your usage, improves both ease of use and customizability. Local compilation doesn't make things more difficult, because it can be done without any interaction; there's no reason that, when you download a binary and install it, the system couldn't download the source and compile it in the background, and then replace the binary installation with the compiled one. In fact, compiling a program locally is much likely to work than using a precompiled binary, and compiling most programs doesn't take as long as reading their documentation on recent hardware.

BTW, the unacceptably-slow Linux installation took less time than Windows ever has.

Still Lags

[Pave Low]

While Open Source can make for cool code for the cool apps, it still lags far behind the big boys when it comes to the most mundane things, copy-n-paste, fonts, printing, etc.

While everybody works on creating the cool things, the uncool things get little attention. That's where proprietary software is still superior. You can get paid to do the grunt work. In Open Source, nobody really cares when you do that, and you won't look at leet as the guy making another IRC application.

So the big boys like Microsoft and Apple will always have a leg up.

Much like closed source

[mikewas]

The number of developers that are actually contributing seems much like the commercial closed-source projects I've worked on. There's always a small team that really understands the code that does most of the work. The core tends to be about half of the team on small projects. Everybody else performs ancillary functions or just goofs off. On larger projects communication breaks down and the core is limited to a no more than a few subgroups of one to 7 developers each. The subgroups tend to work independently, only occasionally interacting with one another -- usually though some spokesman or leader.

So open source group dynamics are similar to closed source projects. Not really surprising, since both are staffed by people!

Larger more formalized projects, aerospace for example, improve on the above by making subgroups of subgroups. This layering of project & program management really increases the overhead. It seems to slow things down, but at the end you can put things together and have a hope of making it work. It's really a formalization & extension of the way we organize ourselves naturally.

Re:Much like closed source

[brad-x]

One would hope that the situation surrounding 'goof offs' doesn't exist in the open source world; although I'm sure they equate with developers who work for a while and then lose interest in the project. I can't imagine such people would stick around for long, in any event.

Although this would explain a lot about Mozilla.. :P

What About OSS Failures?

[theduck]

It's all well and good to focus on the characteristics of a successful OSS project. It's extremely interesting that these projects succeeded without elements that are considered to be fundamental to success in commercial development. However, studying success provides only half the picture. It won't tell you what things to avoid that tend to make OSS projects fail. Without that knowledge, attempts to reproduce and improve upon the methods used by Apache and Mozilla will experience unforseen failures that could have been avoided.

Re:What About OSS Failures?

[FortKnox]

Not only that, but I'd like to see projects that have deadlines that HAVE to be met, as opposed to your hobby code that's finished when its finished.

Up to old tricks

[gnovos]

It is often characterized as a fundamentally new way to develop software that poses a serious challenge to the commercial software businesses that dominate most software markets today.

I was under the impression that this kind of approach to building is a fudamentally old way of getting the job done. The Homebrew Computer Club essentally built everything "open source" (well, it was hardware mostly, but same approach). The current resurgence of OSS is not something new and revolutionary, it is instead a rediscovery of old techniques that were coopted by big business.

One thing the report forgot to mention ...

[Taco Cowboy]

The report mentioned many things that we already know. But there's one important thing about the Open Source software the report may have missed :-

The freedom to change / customize the software via the source code.

People can argue that even Windoze can be customized - background, for example - but it ain't the same as cusomization via source code.

Do you ever have the feeling, when you use commercial / close-source software, that some part of it are kinda stupid, cumbersome, or simply plain assinine ?

Do you ever think that if you _just_ have the source code, perhaps you could do some change to it, at least to better suit your taste ?

Well ... With Open Source, we can.

Now, of course, not every one know how to code, and even fewer of us know how to tweak the code to our own liking. But that doesn't change the point that with Open Source, we _can_ change the software anyway we like it.

It's a feeling of having TOTAL CONTROL over the software.

It's a feeling of empowerment.

It's just _the_ thing close-sourced software users don't get to enjoy.

critical mass most important element

[truth_revealed]

If your open source project does not have a critical mass of developers it will fail. Critical mass can only be acheived if the program being developed is largely finished or already useful in some way. Everyone likes to back a winner. The exception to this rule is when you have a project started by someone with past successes such as Miguel starting the Mono project after the successful GNOME project. But then you could argue that C# was already a mature specification for Mono to exploit, so it was already a "winner" in a sense. The Parrot project, by comparison has the disadvantage of not having pristine design specification and documentation (courtesy of Microsoft) to mimic.
Often, the majority of work in a pre-critical mass project has already been done by a single individual. This is conveniant because they usually become the defacto project leader. It is just as useful to prevent poor contributions into the project as it is to add quality code to the project. This is due to the fact that no one likes code audits.
And the final mark of a good open source project - the leader cannot be a prick. There's too much bullshit to put up with in day to day life, why would you want to get harassed without being payed?

It's all about the QA!

[rockmuelle]

"And of course in having an excessively large testing team by commercial standards, testers out-numbers codes by huge ratios..."

This is probably the most profound statement about OSS I've seen in this discussion.

OSS projects are not better because the coders are more talented or devoted than closed source projects. They are better because they actually have QA resources that cannot be matched by close source projects.

Stop and think about this: put a team of 1000 testers on a project who actually understand the software and do not test by a following a checklist of requirements but actually try to use the software and give them direct access to the developers (ie, remove the management/marketing layers that filter bugs). I suspect in this case a closed source project could have the success of an open source project.

Think about it.

-Chris

Re:PDF alert!

[bogie]

Meh, this isn't 1996. We all can read pdf files on our Linux boxes. While html is always preferred for links, there is no need to call the file a "troll link".

Other thing the report forgot to mention....

[Anonymous Coward]

Hello everything.

All the advantages of Open Source Software included in the report are true (IMHO).

But there is one thing the report forgot to mention.

There is few Open Source Documentation. The existent documentation is shitty: poor, bad explained, often out-of-date or incorrect.

And you always gets mad to install and configure the product.

Come on. Don't treat me as a troll. You know that this is true. I have experience with Open Source Products too and I can't remember all the nights I spent without sleeping trying to figure out why some Open Source product doesn't work the way it is supposed to work. And after some days of anguish, found some clue on the Internet that tells me that I have to do something (of course, this information wasn't part of the documentation).

Commercial products are easier because of the extensive documentation. Open Source products are made by people who love programming but don't love writing manuals or documentation.

This is the other side of the story. I'm not against Open Source Products (I use several of them and now I am teaching a course about how to use several of them) but the lack of decent documentation is a major drawback which makes difficult the adoption of Open Source as the general standard.

Only my two cents.

Vicent Palasí

Re:Open source projects tend to have a lower bug-r

[SashaM]

It's much harder to produce a bug free graphical UI application than it is a daemon. Users can (and will) do a magnitude more things than you can receive on an open socket. When "talking" to a user, you will also need to give him a lot more functionality and a much more diverse interface (accessibility, keyboard navigation, mouse navigation) than you will ever need to give another application that's communicating with yours via a clearly defined protocol.

FUD

[e2d2]

Wow, FUD. According to our study we created to reach our preconcieved notion open source methods are superior.

Where to start?

Open source projects are generally quicker to respond to user requests.
I'm sorry but comparing two open source products to "commercial products" which is who? what product? what project? I don't see any quantative data besides a few lines refering to commercial products as a whole and saying the authors have experience with them is not scientific. I take exception to this because the paper sepcifically tries to appear scientific but yet offers no data comparing either project referenced (Apache and Mozilla) to a commercial counterpart or their ability to respond to bugs.

Open source projects tend to have a lower bug-rate than commercial projects
Again, where is the data? I see the scietific method they use for tracking bugs per line of code and they go into great detail comparing the two projects but yet we see no comparison to commercial project bug counts or the same method applied to commercial projects. The paper is laced with phrases such as "One might speculate". Yeah one might. Of course one might not speculate and offer evidence. If I create a hypothesis should I not have to back it up and test for truth?

And then there is the method of caculating bugs per line of code. They go into great detail about bug counts, when the fix was checked in, the lines of code, etc. But yet how do you measure importance? Some bugs are obviously greater than others. For instance, two teams create two identical applications. One application has 15 bugs and the other application has just 1. They both have the same lines of code, the same project size, same budget, everything is the same. The project with just one bug is obviously superior according to the methods they use, EXCEPT that particular bug allows a remote user to gain Root/Super User access. Which one has failed according to your quantitive data? Which project had the best method? They speak in depth about how this cannot be measured, then show you how they measured it?

Although I think this paper has good intentions and shows insight into some OSS projects,
1. The reference to commercial software as a whole is unfair and offers no value and
2. The method for caculating bugs is not an effective way to measure anything.

This paper is basically the equivilent of Microsoft, Oracle, Sun, or any other entity creating a study that never tests or proves anything and reaching a preconcieved notion. I can see this message being modded as a "troll" but oh well, this paper is not as scientific as it tries to appear.

Lower bug rate kind of a red herring

[dasmegabyte]

"Open source projects tend to have a lower bug-rate than commercial projects"

True. But open source projects are much more precisely targetted, and less functional. Not necessarily in a bad way, but in a way that is very different from marketable commercial software.

Take, for example, IIS vs. Apache. On one hand, yes, Apache is so much better at serving web pages -- faster, more stable, more secure, cheaper etc. But functionally, they don't really do the same thing. IIS encapsulates ASP scripting, database access, file security, web serving, ftp serving, mail serving and a very powerful management interface. Apache is just a core web server. It performs one small task out of dozens, and as such the developers can concentrate on making that work best.

It's hard to do the same with commercial software. You have to keep adding features to stay ahead of the competition -- merely having the fastest webserver is not enough, because hype sells servers, not actual results. For this reason, there are a lot of open source projects that would never survive as viable market solutions. Apache's one of them...considering that "all it does" is serve pages and it relies on "third party" modules to do anything fancier than powerful URL rewrites and server side includes, its market price would be low and thus the margins. Never mind that it's stable as a rock while IIS is as insubstantial as a fart in the wind.

This is a big problem with the adoption of open source as well. You can't just "switch" like you can from, say, Word to WordPerfect. If you use SQL Server and enterprise manager, for example, you can't just "switch" to MySQL. MySQL has no totalitarian interface on par with enterprise manager. It has no massive searchable help database and no "for dummies" option for managing jobs and indexes. If you plug in to MySQL with a SQL Server only toolset, you're in for a shock learning curve, even if the databases themselves are on par with each other and MySQL less buggy. The difference is that what's important to the MySQL developers isn't what sells SQL Server.

applying free market principles to software dev

[g4dget]

Open source is really in many ways what happens when you apply free market principles to software development: people participate, and their contributions survive, based on actual needs, capabilities, and quality. If open source projects fail to meet the needs of their users, they split, and some branch will adapt, but without throwing everything away.

In contrast, closed source development usually involves assigning people to projects. Their primary motivation isn't the software itself (which they will likely never use), but their job and their stock. Determining features involves a few people guessing hard about what features end users may or may not want. Oh, sure, they listen, but as anybody who has gathered requirements knows, users generally aren't very good at communicating what tradeoffs are important to them. And when closed source projects fail in the market, any new entrant has to start from scratch.

It's not surprising that open source development is winning in the long run. It's central planning (Microsoft, Apple) vs. a free market of ideas (open source) all over again. And we already have a good idea which of the two approaches of organizing large numbers of people around a common goal works better.