Open Source Projects Manage Themselves? Dream On.

cconnell writes: "One of the most tantalizing statements about open source development is that these projects manage themselves. Gone are layers of do-nothing managers with bloated bureaucracies and interminable development schedules. In their place is a new paradigm of self-organizing software developers with no overhead and high efficiency. There is just one problem with this assertion -- it is not true. This article shows that open source projects are about as far as you can get from self-organizing. In fact, these projects use strong central control, which is crucial to their success. As evidence, I examine Raymond's fetchmail project and Linus Torvalds's work with Linux."

Comments

The modular approach, how to scale open source

[mnf999]

I am not sure I fully agree with the conclusion of this article.
I started and still run a medium sized open source project (50 developers, 10 core, many 10,000 users) jboss.org (www.jboss.org) and we develop an EJB (Enterprise Java Beans) container.

I suppose every project leader has his style and the domain in which they operate dictates the structure they finally adopt but for the J2EE infrastructure we found that the "modular" approach works best for us. I will argue (for those that are looking for a short message) that for "Operating Systems" like efforts (Linux, J2EE) you need a mix of strong central management and a SELF MAINTAINING CODE ARCHITECTURE (modular, ala Linux 2.0) something I believe the article misses.

Here is the deal, we are today in our third iteration. The first iteration was done by self, really a "let's get started" code base that passed the first stage of putting some people together around the project (2 years ago). The second iteration was a clean rewrite with some advanced concepts given by the group and some of its most outstanding contributors. jboss1.0 was released with that code base. It was a fully featured container but we QUICKLY ran into a wall: it was too complex for a casual contributor or even for 3rd party contributors to come in the code base and integrate.

The lesson there for us was kinda trivial, OS like system grow extremelly complex extremelly quickly and there is little chance (or time) for people to wrap their heads around it. There was no significant growth of the code base, not because we were not dedicated (we were full time on it) but because we could not manage/teach the flow of contributions.

We took a good look at what Linux had done over the years and realized that Linux2.0 based systems were relying on a "modular" approach to Systems design. why not try the same for web-systems design? In clear the idea is to isolate the parts of the container (our kernel) so that folks that want to include a Transaction Monitor, or a Database persistence Engine, can focus on a few places and not worry about the whole architecture of the system. It is the equivalent of "divide and conquer". We modularized our container with interceptors and plugins and released an early version of the clean architecture.

The payoff was immediate, we started seeing contributions pouring in, both from corporations and star developers since they could spend one night to understand the scope of their work and start banging on it right away, knowing that the integration was PURELY CODE BASED. That was one of the advantages of Java's Object Orientation and interfaces for us, it was trivial to do.

The point that I am trying to make, and that the article misses imho is that these contributions are sort of automated. Sure we look at all new modules and we OK-notOK projects but we have given RW passwords to our CVS to litterally EVERY contributor on our list (I think we have like 40 guys on CVS). Sure now and then we will see bad fixes, but these are limited and the interfaces and modular approach ensures us that the integration is almost SELF MANAGED.

I don't claim that this applies to every succesful open source project (yes we are succesful ;-) but that the ONLY way to manage complexity of Open source systems is through a modular approach that largely tends to a peer-to-peer model. It is an interesting thing that the code produced through modularization tends to be of great quality as well.

Code can be automated, in the integration sense, and if you don't have that, you won't keep the exponential growth of code base... there is NO management (read no man/woman) that can replace it.

marc

"management" has a purpose

[Hard_Code]

It took me a while to figure out, but "management" has a purpose. It wasn't until *I* had to be involved in, and manage, projects with *real* deadlines and goals, and a motley assortment of people with different skills and attitudes, that I realized this.

Managers should:

* Define goals, and save us from death by committee.
* Define a path to those goals, so people don't wander off and get lost. "Path" may be formal/informal methodology, standards, technology, milestones for features, etc.
* Keep people on task. Your presence along makes them feel guilty when they are not acheiving goals. I know this, because I waste my time reading Slashdot, and I feel guilty to my manager.
* Facilitate communication amongst people. Complexity of communication scales at n^2. Makes sure people know what others are doing and aren't blocked by dependencies.

This is not exactly coding, but it is definately work that needs to be done. The price a manager pays for not having to be ordered around, is that they are now free to:

* Get blamed when projects fail.

Remember, managers <!=> suits

Not true.

[sheldon]

Go read "Debugging the Development Process" by Maguire.

The book is basically for the audience of "Technical Manager."

They might be called App Architect, Tech Lead, Technical Project Lead, Senior Engineer, whatever.

The role exists, and has existed for years, as the author of this essay points out... Brooks talked about it in the 70's.

Now one thing in a corporate environment, you have all the HR crap.

I think many technical people do not mind falling into the tech lead role. But most technical people don't want to deal with HR crap.

Divide that off onto another person, and you start to achieve success. Your tech leads provide their feedback into the reviews, but this team manager actually handles all the hiring, firing, training, salary, budget, whatever crap.

Many successful companies divide these duties up. Let a technical person manage the software development, and some other non-tech person handle the other crap.

Re:Ask Slashdot

[Shotgun]

1) Not all managers are inept paper pushers.

2) The recent Ask Slashdot I was speaking of concerned a gentleman who had coded up a fix and could not find the right 'manager' to submit it to. The point being that anyone can actually write the code and can solve the problem but it still requires approval of the 'OS manager' to get it in the code base.

3) The bazaar model hypes the idea that everyone gets to decide how to fix a problem and decide how to put it in, not just the blessed few. Few projects follow this methodology, the point of the article and mine.

4)I've dealt with a few OS projects, and from my POV it is easier to talk to upper-management than deal with OS manager/engineers. Engineers tend to have an emotional attachment to the code they've written, and they don't like criticism of their 'baby'. The path I've taken in the past is to fix my local copy of the code and then post my fix to a newsgroup, or even to Slashdot, rather than fight that what I've fixed is an actual bug. Uppermanagement tends to have a myopic view of the world. Convince them that a change saves/makes money and it's in there.

If a company follows the process you outline, it is headed nowhere. Most companies I've seen are much more flexible about the introduction of ideas. At IBM the process was:
submit idea to study group->study group ask customers->study group makes a decision.

At my current job the process is:
submit an idea->start implementing it (unless the idea is completely off-base, the chances rejection are very small)

Haven't read the article yet....

[FascDot Killed My Pr]

...but you've already made at least one fallacy in the intro. Having "strong central control" is orthogonal to "self-organization". The former is about who is in control the latter is about how they got there.

What's the diff? Well, in a non-self-orgranizing project there are by definition external (i.e. non-contributing) organizing forces--this is management. Linux doesn't have that, despite having "strong central control".
--
Linux MAPI Server!
http://www.openone.com/software/MailOne/

Absolutely right

[quigonn]

I am co-developer of fancylogin, a neat login program for Linux, and we have big troubles organizing ourselves. Especially in the beginning we often had dialogues like "did you already implement feature foo? - No, I'm currently working on feature bar - damn, so do I". Software engineering in general can't work in this way, not only at open source projects.

I can think of a Bazaar...

[LionKimbro]

If fetchmail and Linux were not run as bazaar projects, what would a true bazaar project look like? A real bazaar software development method would proceed as follows.

• Someone creates a minimal working version of the software. (This follows Raymond's advice to start with a "plausible promise.")
• >
• The originator releases the working program, a description of how to use it, and all the source code to an appropriate forum such as a newsgroup or public web site. From this point forward, the originator becomes just another member of the user community, with no special status.
• Anyone may download the program and source, try it out, look for bugs, and suggest fixes and enhancements. These ideas are communicated to the entire user community through the forum.
• Anyone at any time, or multiple people at the same time, may decide to create a new version of the program. They do so by using ideas and code from the user community, along with their own contributions. They post the new version to the user forum.
• Most likely, the code "forks" as several people create new releases at the same time. This is part of the bazaar process.
• The user community attempts to settle on the best fork to follow, by trying all available versions and focusing their attention on the best version. No single person or small committee manages this process. Perhaps the best fork is widely recognized and quickly selected, perhaps not.
• Several forks may live in parallel for quite a while. If so, it is the decision of the user community no one fork is the clear winner. When the community tires of parallel forks, they will select one to follow.
• Development continues in this dynamic, organic method. Leaders emerge briefly, as they create a new release or argue for one fork over another, but they then become equal community members again. All decisions about features, design, bug fixes, etc. are made in this way.

Hmm; doesn't this sound suspiciosly similar to the way distrobutions work?

• Red Hat Linux puts together a distribution.
• They make it public and tell us how to use it. At this point, they become just another distrobution.
• Anyone can try it out and change it. (Mandrake does.) I'm not sure where SUSE got it's roots from, but needless to say, there's a lot of inbreeding as well as new code floating about.
• New distributions are posted.
• The distributions fork! (Surprise!) Again: The user community attempts to settle on the best fork to follow, by trying all available versions and focusing their attention on the best version. No single person or small committee manages this process. Perhaps the best fork is widely recognized and quickly selected, perhaps not. Perfect description.
• Coding continues to occur. Distro's come, distro's go.

(The article did have some interesting notes though.)

Ask Slashdot

[Shotgun]

There was an Ask Slashdot question a few days ago about how you make contributions to an OS project. Several of the answers went along the line of read the mailing list and send your bugfix to some of the lesser daemons. Let them take it to the great gods. This hints that an OS project (especially one as important as the kernal) can quickly have a structure identical to Figure 1 of the article.

If you have a problem at work, you talk to your manager, who will speak with the VP, who consults the CEO. How is this any different from emailing the 'less important people on the mailing list', who then contact the component owners, who might be able to get a response from Linus?