Slashdot Mirror
Open Source Projects Manage Themselves? Dream On.
cconnell writes: "One of the most tantalizing statements about open source development is that these projects manage themselves. Gone are layers of do-nothing managers with bloated bureaucracies and interminable development schedules. In their place is a new paradigm of self-organizing software developers with no overhead and high efficiency. There is just one problem with this assertion -- it is not true. This article shows that open source projects are about as far as you can get from self-organizing. In fact, these projects use strong central control, which is crucial to their success. As evidence, I examine Raymond's fetchmail project and Linus Torvalds's work with Linux."
I agree with his central argument, that structure is key, but I'd drive it even further. I'd say that the amount of necessary structure is directly proportional to the amount you can get away with it. Linux and other similar projects, due to their high degree of modularization or lack of size and scope, can afford a relatively loose structure. I do not believe that significantly more involved projects (i.e., development of a high end RDBMS) can afford any of the commonly praised open source management styles.
Though I believe Open Source may some day make great inroads against proprietary software, I think that'll come almost inevitably where structured is lent by those with resources. i.e., corporations. For instance, I briefly read an article in some IT magazine awhile ago that IBM, Redhat, VALinux, and few other corporations were going to build a PARC-like facility where they'd cooperate in the development and improvement of Linux with the intent to create a universal replacement of windows. Assuming the corporations can agree on committing the necessary resources, hiring enough programmers, giving the right amount of force, I could see it succeeding beatifully. But I don't believe we'll ever see Linux, as it is popularly envisioned, usurping the windows hegemony.
But that's NOT how they work. They DO have management and projects and plans. Do you really thing Bob Young's official title is "Guy in the Red Hat"? Do you really think Mandrake has no PHB's? Do you think SuSE has no offices, and every employee logs in from home whenever they feel like it?
The article is right on the money. There is no bazaar. It's more of an open-air cathedral.
A Government Is a Body of People, Usually Notably Ungoverned
You are working on a code project that anyone can look at. You have a 'duty' to make sure that it is the best code possible to make. Also, look at it this way: You have a bunch of geek coders all working on the same project. If you don't have some sort of management and organization system, then the project is pretty much doomed to failure.
There are various Software methodologies to use, each with their own advantages and limitations. Frankly, I don't reccommend Team Software Process (TSP) for Open Source, as it relies too much on time management figures (which may not always be available) and requires too many forms, which may not be the best things for Open Source coders.
Just my $0.02
Kierthos
Mr. Hu is not a ninja.
Consider the Illustra/Informix DataBlade stuff. It makes perfect sense to consider a DataBlade the equivalent of a Linux driver (same with Postgres plugins). As long as the interface is well defined and adhered to, the people developing the datablade kernel don't have to have strict control over the datablades.
The problem is that RDBMSes were designed in the 70s and 80s when modular code wasn't really in vogue. And even if they were, there are minor performance tweaks you can get if you ignore modularization in favor of an additional 1% bump in your TPC-C numbers.
If there were an RDBMS which was designed correctly, it would be a perfect candidate for open source development, as long as there was somebody sitting around saying "This version is stable enough to run your business on", which is the equivalent of the Linus/Alan system of test releases and final kernel versions.
Just because Oracle is designed wrong doesn't mean that the basic premise is any less sound.
I am not sure I fully agree with the conclusion of this article.
;-) but that the ONLY way to manage complexity of Open source systems is through a modular approach that largely tends to a peer-to-peer model. It is an interesting thing that the code produced through modularization tends to be of great quality as well.
I started and still run a medium sized open source project (50 developers, 10 core, many 10,000 users) jboss.org (www.jboss.org) and we develop an EJB (Enterprise Java Beans) container.
I suppose every project leader has his style and the domain in which they operate dictates the structure they finally adopt but for the J2EE infrastructure we found that the "modular" approach works best for us. I will argue (for those that are looking for a short message) that for "Operating Systems" like efforts (Linux, J2EE) you need a mix of strong central management and a SELF MAINTAINING CODE ARCHITECTURE (modular, ala Linux 2.0) something I believe the article misses.
Here is the deal, we are today in our third iteration. The first iteration was done by self, really a "let's get started" code base that passed the first stage of putting some people together around the project (2 years ago). The second iteration was a clean rewrite with some advanced concepts given by the group and some of its most outstanding contributors. jboss1.0 was released with that code base. It was a fully featured container but we QUICKLY ran into a wall: it was too complex for a casual contributor or even for 3rd party contributors to come in the code base and integrate.
The lesson there for us was kinda trivial, OS like system grow extremelly complex extremelly quickly and there is little chance (or time) for people to wrap their heads around it. There was no significant growth of the code base, not because we were not dedicated (we were full time on it) but because we could not manage/teach the flow of contributions.
We took a good look at what Linux had done over the years and realized that Linux2.0 based systems were relying on a "modular" approach to Systems design. why not try the same for web-systems design? In clear the idea is to isolate the parts of the container (our kernel) so that folks that want to include a Transaction Monitor, or a Database persistence Engine, can focus on a few places and not worry about the whole architecture of the system. It is the equivalent of "divide and conquer". We modularized our container with interceptors and plugins and released an early version of the clean architecture.
The payoff was immediate, we started seeing contributions pouring in, both from corporations and star developers since they could spend one night to understand the scope of their work and start banging on it right away, knowing that the integration was PURELY CODE BASED. That was one of the advantages of Java's Object Orientation and interfaces for us, it was trivial to do.
The point that I am trying to make, and that the article misses imho is that these contributions are sort of automated. Sure we look at all new modules and we OK-notOK projects but we have given RW passwords to our CVS to litterally EVERY contributor on our list (I think we have like 40 guys on CVS). Sure now and then we will see bad fixes, but these are limited and the interfaces and modular approach ensures us that the integration is almost SELF MANAGED.
I don't claim that this applies to every succesful open source project (yes we are succesful
Code can be automated, in the integration sense, and if you don't have that, you won't keep the exponential growth of code base... there is NO management (read no man/woman) that can replace it.
marc
The real mnf999 always posts as anonymous coward
It took me a while to figure out, but "management" has a purpose. It wasn't until *I* had to be involved in, and manage, projects with *real* deadlines and goals, and a motley assortment of people with different skills and attitudes, that I realized this.
Managers should:
* Define goals, and save us from death by committee.
* Define a path to those goals, so people don't wander off and get lost. "Path" may be formal/informal methodology, standards, technology, milestones for features, etc.
* Keep people on task. Your presence along makes them feel guilty when they are not acheiving goals. I know this, because I waste my time reading Slashdot, and I feel guilty to my manager.
* Facilitate communication amongst people. Complexity of communication scales at n^2. Makes sure people know what others are doing and aren't blocked by dependencies.
This is not exactly coding, but it is definately work that needs to be done. The price a manager pays for not having to be ordered around, is that they are now free to:
* Get blamed when projects fail.
Remember, managers <!=> suits
It's 10 PM. Do you know if you're un-American?
The author does not seem to realize that the difficulty is in finding the bug in the first place. He seems to think the finding or the insight to find is trivial, and the coding of the fix, once found, is hard. This is the exact opposite of my own experience.
You're definitely right on about projects with no management - i'm sure it's a nightmare. What most projects need is organization that comes from below, not from above. Look at it this way:
Our current business model: Business is a separate layer existing on top of the development and production processes. Administration is for the benefit of business, not development/production, even though business's only logical purpose in existence is to organize and improve development. Administration has the most authority, so naturally they only work to benefit themselves. Developers/producers; i.e the real workers have basically no say in this process, which most likely interferes with the efficiency of their operation.
A better, more sane business model: Since the development of the product is the most crucial aspect of a business's purpose, developers should rule the company, with administrative, marketing, sales, etc. types as advisors on the deveoper's payroll. Money should be distributed by the people who do the most work within the company: the developers and producers! The administrators make suggestions, but have no real authority.
And yes, it could theoretically work, and could help to eliminate the fact that right now, business is just a separate society leeching off of the real society and bringing no benefit to it whatsoever (it's called a PARASITE.)
I've said my piece.
Although, you left some parts off of your corporate example. It should be more like this:
Your problem -> Your manager -> VP -> Big Boss -> another VP -> another manager -> grunt who actually fixes problem.
In the open source world there are few people (usually none) who are pure management. When you send your problem to people working on the linux kernel who forward it to Linus, they actually write the code and can solve the problem. In the open source world your bug reports and comments actually go straight to the ears of the people who not only solve the problems and implement new functionality but also decide what new functionality gets added in. It is the excising of the huge manager -> higher up manager -> yet higher manager -> ..... -> high enough management to make a decision -> lower manager -> yet more lower management -> .... -> manager intermediate step which usually has very little usefulness (often it's usefulness is in fact negative), that makes "The Bazaar" so special.
Managers figure out how to use resources (i.e. people); architects design systems. While there's a little management in Open Source, most of my work with SAX (Simple API for XML) has been a combination of design and implementation. After all, I can hardly threaten to fire people or offer them bonuses or promotions, so I cannot make anyone do anything but what they wanted to do in the first place. That's what Eric meant, I think -- the architecture evolves according to what people are willing to do, not (usually) according to a strong central vision. I remember being part of the thread that annoyed Linus into allowing loadable kernel modules (hardly part of Linus's grand vision at the time).
I chaired a W3C working group for a year and spent most of my time worrying about absurd procedures designed by non-coders to make sure that the coders couldn't do any real work. That was management.
Go read "Debugging the Development Process" by Maguire.
The book is basically for the audience of "Technical Manager."
They might be called App Architect, Tech Lead, Technical Project Lead, Senior Engineer, whatever.
The role exists, and has existed for years, as the author of this essay points out... Brooks talked about it in the 70's.
Now one thing in a corporate environment, you have all the HR crap.
I think many technical people do not mind falling into the tech lead role. But most technical people don't want to deal with HR crap.
Divide that off onto another person, and you start to achieve success. Your tech leads provide their feedback into the reviews, but this team manager actually handles all the hiring, firing, training, salary, budget, whatever crap.
Many successful companies divide these duties up. Let a technical person manage the software development, and some other non-tech person handle the other crap.
1) Not all managers are inept paper pushers.
2) The recent Ask Slashdot I was speaking of concerned a gentleman who had coded up a fix and could not find the right 'manager' to submit it to. The point being that anyone can actually write the code and can solve the problem but it still requires approval of the 'OS manager' to get it in the code base.
3) The bazaar model hypes the idea that everyone gets to decide how to fix a problem and decide how to put it in, not just the blessed few. Few projects follow this methodology, the point of the article and mine.
4)I've dealt with a few OS projects, and from my POV it is easier to talk to upper-management than deal with OS manager/engineers. Engineers tend to have an emotional attachment to the code they've written, and they don't like criticism of their 'baby'. The path I've taken in the past is to fix my local copy of the code and then post my fix to a newsgroup, or even to Slashdot, rather than fight that what I've fixed is an actual bug. Uppermanagement tends to have a myopic view of the world. Convince them that a change saves/makes money and it's in there.
If a company follows the process you outline, it is headed nowhere. Most companies I've seen are much more flexible about the introduction of ideas. At IBM the process was:
submit idea to study group->study group ask customers->study group makes a decision.
At my current job the process is:
submit an idea->start implementing it (unless the idea is completely off-base, the chances rejection are very small)
Aah, change is good. -- Rafiki
Yeah, but it ain't easy. -- Simba
Why the fuck can't you take 2 minutes off to read the damn artcile before commenting? No matter how important what you have to say is, read the article first. Yes, mark me as a troll, I do not care, but I read the entire article, word for word, and it is in my bookmark. I also agree with 95% of the content.
------ Curiosity killed the cat. {satisfaction brought it back | it didn't die ignorant | lack of it is killing mankind
...but you've already made at least one fallacy in the intro. Having "strong central control" is orthogonal to "self-organization". The former is about who is in control the latter is about how they got there.
What's the diff? Well, in a non-self-orgranizing project there are by definition external (i.e. non-contributing) organizing forces--this is management. Linux doesn't have that, despite having "strong central control".
--
Linux MAPI Server!
http://www.openone.com/software/MailOne/
Linux MAPI Server!
http://www.openone.com/software/MailOne/
(Exchange Migration HOWTO coming soon)
I am co-developer of fancylogin, a neat login program for Linux, and we have big troubles organizing ourselves. Especially in the beginning we often had dialogues like "did you already implement feature foo? - No, I'm currently working on feature bar - damn, so do I". Software engineering in general can't work in this way, not only at open source projects.
A monkey is doing the real work for me.
If fetchmail and Linux were not run as bazaar projects, what would a true bazaar project look like? A real bazaar software development method would proceed as follows.
Hmm; doesn't this sound suspiciosly similar to the way distrobutions work?
(The article did have some interesting notes though.)
There was an Ask Slashdot question a few days ago about how you make contributions to an OS project. Several of the answers went along the line of read the mailing list and send your bugfix to some of the lesser daemons. Let them take it to the great gods. This hints that an OS project (especially one as important as the kernal) can quickly have a structure identical to Figure 1 of the article.
If you have a problem at work, you talk to your manager, who will speak with the VP, who consults the CEO. How is this any different from emailing the 'less important people on the mailing list', who then contact the component owners, who might be able to get a response from Linus?
Aah, change is good. -- Rafiki
Yeah, but it ain't easy. -- Simba