Open Source Projects Manage Themselves? Dream On.

cconnell writes: "One of the most tantalizing statements about open source development is that these projects manage themselves. Gone are layers of do-nothing managers with bloated bureaucracies and interminable development schedules. In their place is a new paradigm of self-organizing software developers with no overhead and high efficiency. There is just one problem with this assertion -- it is not true. This article shows that open source projects are about as far as you can get from self-organizing. In fact, these projects use strong central control, which is crucial to their success. As evidence, I examine Raymond's fetchmail project and Linus Torvalds's work with Linux."

Comments

The modular approach, how to scale open source

[mnf999]

I am not sure I fully agree with the conclusion of this article.
I started and still run a medium sized open source project (50 developers, 10 core, many 10,000 users) jboss.org (www.jboss.org) and we develop an EJB (Enterprise Java Beans) container.

I suppose every project leader has his style and the domain in which they operate dictates the structure they finally adopt but for the J2EE infrastructure we found that the "modular" approach works best for us. I will argue (for those that are looking for a short message) that for "Operating Systems" like efforts (Linux, J2EE) you need a mix of strong central management and a SELF MAINTAINING CODE ARCHITECTURE (modular, ala Linux 2.0) something I believe the article misses.

Here is the deal, we are today in our third iteration. The first iteration was done by self, really a "let's get started" code base that passed the first stage of putting some people together around the project (2 years ago). The second iteration was a clean rewrite with some advanced concepts given by the group and some of its most outstanding contributors. jboss1.0 was released with that code base. It was a fully featured container but we QUICKLY ran into a wall: it was too complex for a casual contributor or even for 3rd party contributors to come in the code base and integrate.

The lesson there for us was kinda trivial, OS like system grow extremelly complex extremelly quickly and there is little chance (or time) for people to wrap their heads around it. There was no significant growth of the code base, not because we were not dedicated (we were full time on it) but because we could not manage/teach the flow of contributions.

We took a good look at what Linux had done over the years and realized that Linux2.0 based systems were relying on a "modular" approach to Systems design. why not try the same for web-systems design? In clear the idea is to isolate the parts of the container (our kernel) so that folks that want to include a Transaction Monitor, or a Database persistence Engine, can focus on a few places and not worry about the whole architecture of the system. It is the equivalent of "divide and conquer". We modularized our container with interceptors and plugins and released an early version of the clean architecture.

The payoff was immediate, we started seeing contributions pouring in, both from corporations and star developers since they could spend one night to understand the scope of their work and start banging on it right away, knowing that the integration was PURELY CODE BASED. That was one of the advantages of Java's Object Orientation and interfaces for us, it was trivial to do.

The point that I am trying to make, and that the article misses imho is that these contributions are sort of automated. Sure we look at all new modules and we OK-notOK projects but we have given RW passwords to our CVS to litterally EVERY contributor on our list (I think we have like 40 guys on CVS). Sure now and then we will see bad fixes, but these are limited and the interfaces and modular approach ensures us that the integration is almost SELF MANAGED.

I don't claim that this applies to every succesful open source project (yes we are succesful ;-) but that the ONLY way to manage complexity of Open source systems is through a modular approach that largely tends to a peer-to-peer model. It is an interesting thing that the code produced through modularization tends to be of great quality as well.

Code can be automated, in the integration sense, and if you don't have that, you won't keep the exponential growth of code base... there is NO management (read no man/woman) that can replace it.

marc

I can think of a Bazaar...

[LionKimbro]

If fetchmail and Linux were not run as bazaar projects, what would a true bazaar project look like? A real bazaar software development method would proceed as follows.

• Someone creates a minimal working version of the software. (This follows Raymond's advice to start with a "plausible promise.")
• >
• The originator releases the working program, a description of how to use it, and all the source code to an appropriate forum such as a newsgroup or public web site. From this point forward, the originator becomes just another member of the user community, with no special status.
• Anyone may download the program and source, try it out, look for bugs, and suggest fixes and enhancements. These ideas are communicated to the entire user community through the forum.
• Anyone at any time, or multiple people at the same time, may decide to create a new version of the program. They do so by using ideas and code from the user community, along with their own contributions. They post the new version to the user forum.
• Most likely, the code "forks" as several people create new releases at the same time. This is part of the bazaar process.
• The user community attempts to settle on the best fork to follow, by trying all available versions and focusing their attention on the best version. No single person or small committee manages this process. Perhaps the best fork is widely recognized and quickly selected, perhaps not.
• Several forks may live in parallel for quite a while. If so, it is the decision of the user community no one fork is the clear winner. When the community tires of parallel forks, they will select one to follow.
• Development continues in this dynamic, organic method. Leaders emerge briefly, as they create a new release or argue for one fork over another, but they then become equal community members again. All decisions about features, design, bug fixes, etc. are made in this way.

Hmm; doesn't this sound suspiciosly similar to the way distrobutions work?

• Red Hat Linux puts together a distribution.
• They make it public and tell us how to use it. At this point, they become just another distrobution.
• Anyone can try it out and change it. (Mandrake does.) I'm not sure where SUSE got it's roots from, but needless to say, there's a lot of inbreeding as well as new code floating about.
• New distributions are posted.
• The distributions fork! (Surprise!) Again: The user community attempts to settle on the best fork to follow, by trying all available versions and focusing their attention on the best version. No single person or small committee manages this process. Perhaps the best fork is widely recognized and quickly selected, perhaps not. Perfect description.
• Coding continues to occur. Distro's come, distro's go.

(The article did have some interesting notes though.)