Slashdot Mirror
Open Source Projects Manage Themselves? Dream On.
cconnell writes: "One of the most tantalizing statements about open source development is that these projects manage themselves. Gone are layers of do-nothing managers with bloated bureaucracies and interminable development schedules. In their place is a new paradigm of self-organizing software developers with no overhead and high efficiency. There is just one problem with this assertion -- it is not true. This article shows that open source projects are about as far as you can get from self-organizing. In fact, these projects use strong central control, which is crucial to their success. As evidence, I examine Raymond's fetchmail project and Linus Torvalds's work with Linux."
I agree with his central argument, that structure is key, but I'd drive it even further. I'd say that the amount of necessary structure is directly proportional to the amount you can get away with it. Linux and other similar projects, due to their high degree of modularization or lack of size and scope, can afford a relatively loose structure. I do not believe that significantly more involved projects (i.e., development of a high end RDBMS) can afford any of the commonly praised open source management styles.
Though I believe Open Source may some day make great inroads against proprietary software, I think that'll come almost inevitably where structured is lent by those with resources. i.e., corporations. For instance, I briefly read an article in some IT magazine awhile ago that IBM, Redhat, VALinux, and few other corporations were going to build a PARC-like facility where they'd cooperate in the development and improvement of Linux with the intent to create a universal replacement of windows. Assuming the corporations can agree on committing the necessary resources, hiring enough programmers, giving the right amount of force, I could see it succeeding beatifully. But I don't believe we'll ever see Linux, as it is popularly envisioned, usurping the windows hegemony.
The difference between an open-source management model and a traditional management model is that the management in this case comes from the most expert software developers in the group. Raymond, being the primary force behind Fetchmail, and Torvalds, being the primary force behind the Linux kernel, are not traditional management types at all because they get their hands dirty, and they have technical expertise.
I don't want to insult anyone, but generally, people who have Computer Science/Management based degrees DO NOT know as much as people with Computer Science/Engineering degrees. They are taught to manage based on traditional techniques of business management, techniques which generally detract from the overall usefulness and efficiency of a project. They obtain a small amount of technical knowledge, enough to become merely adequate at their field, enough that they are not ignorant of the goals and ideas of a product (not entirely ignorant, anyway), yet they would be completely confused if actually asked to develop a worthwhile piece of the puzzle.
The open source projects mentioned in the Cathedral and the Bazaar, i believe, are similar to traditional projects in that they require orginization and central administration to become useful, but they differ in that people who don't understand the development process are not welcome to participate.
I'll just leave you with this: in our society, actual work is done by a large amount of people; products are produced by a large amount of people, etc. Yet the small few at the top who do not actually produce any products; who do not know how or want to actually work for a living; who attempt to administrate despite their complete lack of knowledge about what is actually produced; those people retain the largest cut of the money and hand out the rest to everyone else who did something. Is there something wrong with this societal model??
When TCP/IP came along, though, this approach was abandoned to great extent. And when Unix GUI programming came along, it was totally abandoned -- there's no such thing as an "X" component.
And finally, Microsoft did the world a grande disservice by deciding that no, threads of execution did not need to be firewalled off from each other so that stray pointers could not crash the entire system, let's run them all in the same memory space! Multi-threaded programs are now the norm, and are a maintenance nightmare. Components in a multi-threaded program can no longer be tested in abstraction, they must be tested amidst the object environment that they were designed for. Successfully writing a component in a multi-threaded program is not just a case of writing some routines to process a stdin and stdout file handle, you now must know about the stuff in the global shared object depository, and how to arbitrate access to them.
A project I'm working on started out as a Windows-style multi-threaded program. It swiftly became obvious to me, however, when my office-mate (an excellent programmer) could not understand my architecture, that the chances of less-than-stellar programmers ever coming into the project and contributing were next to nil. Thus the project was re-written as a Unix-style program rather than a Windows-style program. While newcomers to the project need to have various services running before their programs will work, the startup learning time is greatly reduced, and their components can be written in isolation without me worrying that their component is going to break *MY* component. And on modern systems, this isn't even slow. Granted, sometimes we have 10 processes running (a plumbing nightmare!), but user response is still quite snappy.
Threads: Just Say No.
-E
Send mail here if you want to reach me.
Thanks....they lowercased the page name.
The correct url should be: votenader.org/issues/corporations.html
I believe Nader2000 just mirrors that. Nader2000 is unofficial, but has a lot of info.
It's 10 PM. Do you know if you're un-American?
Are you talking about bugs? This isn't an accurate statement at all concerning bugs. When an engineer hears about a bug, his first reaction is not to deny it's existance, but to fix it. He (or She) want's his 'baby' to be perfect. That's my experience, at least.
Are you talking about major architectural changes? Certainly you shouldn't be asking for major changes if you aren't a major developer. You wouldn't know what you were talking about.
If the problem is that the developer is overworked and can't devote enough time to your particular fix, that's a different issue entirely. Don't make it sound like the developer doesn't want to fix bugs.
Got HTML? Want LaTeX? Try html2latex
While this is a marvelous attitude (and an appealing one for me, coder-in-training), you must realize this is why Mozilla (and other open-source products) haven't shipped. My limited experience in development has led me to conclude the following (at least at the last place I worked) about the process.
:)
Program managers come up with ideas, everyone goes to meetings about them (even testers). Development leads for each feature team (btw, all were developers for 5+ years) then come up with schedules, working with the devs.
The devs (me) try really hard to get it done. There is an absolute ship date.
It slips a little bit. Everything takes longer than imagined.
We cut a whole bunch of stuff so we can ship close to it.
We finally ship a product.
If we were coding for coding's sake, the work would never get done. It would, however, be a programmer's dream.
There's no mystical energy field that controls my destiny. It's all a lot of simple tricks and nonsense.
...that those who are in leadership roles in the open source and free software movements are not only capable of managing people; they are also very technically adept. No PHBs here! In many cases the leaders are the original authors of the project, and in most, the leaders have assumed their role only through the development community's collective respect of the leader's technical merit.
So while our (open-source/free) development communities do, in fact, have leaders, our leaders are, in fact, better.
Washington, DC: It's like Hollywood for ugly people.
I was going to say the exact same thing, that it sounds like how the distributions work. Sure, the linux kernel currently is managed by Linus, but due to the bazarre nature of open source*, as some random CS guy from Finland, he was able to write a kernel and people started adopting it alongside the BSD and herd kernels, which were already in development.
Although he manages the kernel he does not manage any linux distributions.
* I use the term open source here since we are talking about ESR's open source theories. I usually call it free software.
I've sent an email to the author of the article explaining how the *BSD systems (particularly FreeBSD) are orginized. Waiting for an update. :)
- Alfred Perlstein - Programmer and Administrator, Wintelcom.
With good project management, the management is invisible. Management is an enabler, and nothning more (and nothing less!).
Having said that, I feel that open source projects encourage an enabling management, but it's management nevertheless. The people in charge know that they can't motivate people by creating artificial deadlines, or stupid requirements. The motivator is enthusiasm and success.
Open source attracts good management.
(An essential book for Managers, talking about this stuff, is Peopleware, by Tom DeMarco and Timothy Lister. Do yourself a favor and get it for your manager!)
Under capitalism man exploits man. Under communism it's the other way around.
Calling Open Source methods the Cathedral is simply ridiculous. The cathedral is monolithic and it's architecture can't be changed. Bazaars involve negotiation, haggling, keeping a sharp eye out for prevailing opinion, constantly changing power relationships. Which of those two describes the Open Source community we all know.
This is one of those "They're all wrong and I'm right" articles. Only he isn't.
Before we actually start to flame him, maybe we should all read the article first.
I'm not saying whatever he says is all true. But he got some interesting points there.
The 'real' bazaar model in software engineering would not really work. Do you actually think that it would really work, among all the chaos that would result from such a model? Someone, or some trusting entity, will have to put an order to the chaos at the end.
However, there's probably something that Connel failed to point out. The open source model (flat cathedral) eliminate a lot of efficiency out of the traditional "pyramid" model, where a lot of decisions are made by idiot management types who have no technical skills. What results from that is poor quality softwares, coz those management types (most of them) don't know what they are talking about but insist that certain features be in the system or certain architecture be designed in a certain way.
The open source model is centrally controled by capable and smart people who are themselves contributors to the project, not just people who wander around in the office with note book and claim to be project manager. And the final decision on what will go in the source database take into consideration a lot of opinions, suggestions, debates from all participants. As a matter of fact, open source projects that do not do this are loosing contributors, e.g. x86BSD, just like the traditional model that do not allow any say from contributors does not prosper well.
Absolutely! Which is why we have clinicians, nurses and physicians working for us. They use the equipment/software in actual clinical conditions. Then we have external clinicals. Then we have beta sites where the stuff is used in real life situations.
:-)
But what it all eventually comes down to is "the customer is always right, even when they are wrong". If there is a feature that they need/b but they do not use, then they do not use it, QED. And if they don't like it we cannot force them to use it. In one instance we improved our medical imaging quality so much that the customers hated it. Physicians are very set in their ways, and the new improved medical imaging was *different*, so much so that they could not recognize the "texture" of a tumor with the new imaging, but relied on the older blurry "texture". However, younger physicians who are not so set in their ways prefer the new style imaging. Go figure
Open Source non-commercial projects have the freedom to work on whatever they want without worry of financial loss. But commercial projects, whether proprietary or open source, cannot afford thousands of man hours for features that they customers will not purchase.
A Government Is a Body of People, Usually Notably Ungoverned
Sports analogies are out of fashion, though. Oh well.
Traditionally the business model of almost anything as been about producing some widget as cheap as possible and selling a lot of them. The qualities of the widget were unimportant, as long as it wasn't blatantly defective (and even this wasn't a problem, so long as the others in its category were defective).
What made the business model sing was MARKETING -- someome discovered that if you had really catchy marketing it didn't matter what your widgets were, you could make people buy them and create a market for your widgets.
The computer software industry suffers from this attitude. They think that it doesn't matter what kind of quality their software has as long as its not demonstrably worse than their competitors. They'll use MARKETING to to sell their products. cf. Microsoft.
(Incidentally, this attitude towards marketing as king is also what makes software companies drool for cheap guest workers on H1B visas -- you never hear about the shortage of high-quality marketing people, you only hear about large salaries and benefits packages for them).
It puts the software industry in a weird position. You can make a pile of money selling crap software with good marketing, but at the same time the rank-and-file IT people have figured out that the free stuff off the internet (Linux, BSDs, etc) is as good or better than the crap pushed down their throats from the marketing people, and they're defecting to the quality stuff in droves.
I think at some point the software industry may figure out that they need to put the development people in charge of the business process and make marketing and sales respond to development instead of the other way around.
Using marketing to drive your business model is probably entirely appropriate for products with little practical difference (soap, jeans, et al), but I think it might be the ultimate undoing of the software biz.
Putting marketing on a level with development is a sure way to skew the organization and make it marketing-driven. Marketing is fundamentally different from development, and needs to be treated as such, not just botched in as "just another team".
--
It's a
-- Danny Vermin
Of course there is a single website for GNOME. Of course there is a single person managing the Linux kernel. Of course there is only _one_ Nautilus app.
:-)
But if I want to add something to GNOME, I write what _I_ want and offer it to the community. GNOME may or may not accept it into its core, but that's valid: GNOME is the "market visitor", picking out what it likes. If I have a patch for the Linux kernel, I offer it at the mailing list. Linux visits the "marketplace" and picks out the good stuff.
I've seen people "playing marketplace" with their GNOME stuff. A good instance is Galeon; it is not accepted at the GNOME core, but it is very popular amongst individual "visitors". I've seen people offering their stuff at linux-kernel; some of their stuff is accepted, some of their stuff is rejected by Linus but remains a seperate patch. I've seen someone offering to write Samba support for GNOME-VFS, at which the Nautilus folks responded: "that would be great". I've seen distro's choose between package formats. Etc., etc.
Now, if this isn't a marketplace...
Of course there are project managers, but all they really do is "shop". And everyone is free to offer their goods. Never, ever, does a project manager tell somebody what to do in the Open Source world; instead, they only say "I'd like to have this, if there's someone who implements it, I'd be grateful". If there's no-one implementing it, it's not going to be there.
Go look at some Open Source projects' FAQ's, you'll often find these two:
Q.: When will it be finished?
A.: Depends on your input
Q.: Will there be feature X?
A.: No-one is working on it, but you are welcome to do it.
'nuff said, right? A project is nothing more than a "market visitor", that says what it needs and where it should go. The "market people" then can make their offers.
It's... It's...
"We can confirm that Debian does *not* ship the version with the trojan horse. Our version predates it." [CA-2002-28]
Another compelling (and often-quoted) section of The Cathedral and the Bazaar is the discussion about debugging. Raymond says: "Given enough eyeballs, all bugs are shallow" and "Debugging is parallelizable." These assertions simply are not true and are distortions of how the development of fetchmail proceeded. It is true that many people, in parallel, looked for bugs and proposed fixes. But only one person (Raymond) actually made fixes, by incorporating the proposed changes into the official code base. Debugging (the process of fixing the program) was performed by one person, from suggestions made by many people. If Raymond had blindly applied all proposed code changes, without reading them and thinking about them, the result would have been chaos. A rare bug can be fixed completely in isolation, with no effect on the rest of the program.
The above quote from the article suggests that "debugging" consists essentially of making the code changes themselves. - WRONG!
The most excrutiatingly painful part of the "debugging" process is to actually find the bug, not necessarily and exclusively fixing it which I consider more like the fun part of the challenge. Well that can be inverted in some cases too ...
Anyway, the Open Source model, with "many eyeballs" looking at the code, helps you FIND a lot more bugs than one person would, and identify the weaker areas of the code and architecture. Of course the code doesn't fix itself because a bunch of geeks looked at it, and the OpenSource concept never ever claimed to allow that, but it does allow a more thorough debugging process which will eventually make for stronger code.
One of my favorite Open Source projects is the Xalan XSL processor, which I believe is another marvelous example of the success of the Open Source concept. I have tested its compliance against W3C Standards by using a good 99% of XSL-T and XPath's features in some complex XSL-based applications I'm working on, and I can tell you this thing is rock-solid, which I find even more laudable as language processors are some of the more complex applications one can develop.
Extraordinary Vacations. Exceptional Prices
This no manger thing could prove to be a problem. Imagine the scene, three years hence. A street corner, in a cold New England city. A lady walks by a home-impared person. "Got a dime," he asks "for an old mid-level technical manager?" This will be happening all over the country, as companies large and small shed their carbon-exchange units. There will have to be a government program to help these unfortunates of the "Programmer Revolution." It warms the heart.
Rafe Singer: Sorcerer's Apprentice, Doer of the Impossible, Sayer of Sooths, Believer in Murphy's Law
But that's NOT how they work. They DO have management and projects and plans. Do you really thing Bob Young's official title is "Guy in the Red Hat"? Do you really think Mandrake has no PHB's? Do you think SuSE has no offices, and every employee logs in from home whenever they feel like it?
The article is right on the money. There is no bazaar. It's more of an open-air cathedral.
A Government Is a Body of People, Usually Notably Ungoverned
... one of the safest and most productive forms of government.
They are not democracies. Democracy doesn't work for this sort of thing.
Most projects, have 3 letters and one person:
CVS and the guy who approves changes and/or does most of the work.
By the time Open Source (the good stuff) is released its all-ready pretty pollished and/or has an established framework and scope of application.
Hence yes, there is central organization. But the advantage is that said organization comes usually from those with detailed technical knowledge.
-Daniel
But you are missing the one important role of upper management and marketing drones: they are not geeks. Seriously. When you take a look at open source projects without PHBs you'll see geeks writing software for other geeks. (I am a geek, no insult intended). Looking at those OSS projects that aren't geeky, you'll find that they do indeed have upper management.
In the company I work for, we make medical imaging devices. If the hardware and software engineers had their way, we would have a vastly different system, and possibly more robust and powerful. But it would be useless to the physician end-user. We are lucky in that our CEO and many of the top dogs are in fact physicians. Those that aren't are in daily contact with physicians and nurses and hospital admins. If the majority of the users want a feature, we will do that feature, regardless of what the engineers think. This is a Good Thing!
A Government Is a Body of People, Usually Notably Ungoverned
You are working on a code project that anyone can look at. You have a 'duty' to make sure that it is the best code possible to make. Also, look at it this way: You have a bunch of geek coders all working on the same project. If you don't have some sort of management and organization system, then the project is pretty much doomed to failure.
There are various Software methodologies to use, each with their own advantages and limitations. Frankly, I don't reccommend Team Software Process (TSP) for Open Source, as it relies too much on time management figures (which may not always be available) and requires too many forms, which may not be the best things for Open Source coders.
Just my $0.02
Kierthos
Mr. Hu is not a ninja.
I'm just curious - how does the KDE project is managed? They have no one head figure, no "dictator", yet it seem very organized with planning ahead of dates, feature-freezes, code freeze etc
Is it a true bazaar? Or maybe the management is taken by "volunteers" for each task? Someone from KDE could answer?
Consider the Illustra/Informix DataBlade stuff. It makes perfect sense to consider a DataBlade the equivalent of a Linux driver (same with Postgres plugins). As long as the interface is well defined and adhered to, the people developing the datablade kernel don't have to have strict control over the datablades.
The problem is that RDBMSes were designed in the 70s and 80s when modular code wasn't really in vogue. And even if they were, there are minor performance tweaks you can get if you ignore modularization in favor of an additional 1% bump in your TPC-C numbers.
If there were an RDBMS which was designed correctly, it would be a perfect candidate for open source development, as long as there was somebody sitting around saying "This version is stable enough to run your business on", which is the equivalent of the Linus/Alan system of test releases and final kernel versions.
Just because Oracle is designed wrong doesn't mean that the basic premise is any less sound.
I am not sure I fully agree with the conclusion of this article.
;-) but that the ONLY way to manage complexity of Open source systems is through a modular approach that largely tends to a peer-to-peer model. It is an interesting thing that the code produced through modularization tends to be of great quality as well.
I started and still run a medium sized open source project (50 developers, 10 core, many 10,000 users) jboss.org (www.jboss.org) and we develop an EJB (Enterprise Java Beans) container.
I suppose every project leader has his style and the domain in which they operate dictates the structure they finally adopt but for the J2EE infrastructure we found that the "modular" approach works best for us. I will argue (for those that are looking for a short message) that for "Operating Systems" like efforts (Linux, J2EE) you need a mix of strong central management and a SELF MAINTAINING CODE ARCHITECTURE (modular, ala Linux 2.0) something I believe the article misses.
Here is the deal, we are today in our third iteration. The first iteration was done by self, really a "let's get started" code base that passed the first stage of putting some people together around the project (2 years ago). The second iteration was a clean rewrite with some advanced concepts given by the group and some of its most outstanding contributors. jboss1.0 was released with that code base. It was a fully featured container but we QUICKLY ran into a wall: it was too complex for a casual contributor or even for 3rd party contributors to come in the code base and integrate.
The lesson there for us was kinda trivial, OS like system grow extremelly complex extremelly quickly and there is little chance (or time) for people to wrap their heads around it. There was no significant growth of the code base, not because we were not dedicated (we were full time on it) but because we could not manage/teach the flow of contributions.
We took a good look at what Linux had done over the years and realized that Linux2.0 based systems were relying on a "modular" approach to Systems design. why not try the same for web-systems design? In clear the idea is to isolate the parts of the container (our kernel) so that folks that want to include a Transaction Monitor, or a Database persistence Engine, can focus on a few places and not worry about the whole architecture of the system. It is the equivalent of "divide and conquer". We modularized our container with interceptors and plugins and released an early version of the clean architecture.
The payoff was immediate, we started seeing contributions pouring in, both from corporations and star developers since they could spend one night to understand the scope of their work and start banging on it right away, knowing that the integration was PURELY CODE BASED. That was one of the advantages of Java's Object Orientation and interfaces for us, it was trivial to do.
The point that I am trying to make, and that the article misses imho is that these contributions are sort of automated. Sure we look at all new modules and we OK-notOK projects but we have given RW passwords to our CVS to litterally EVERY contributor on our list (I think we have like 40 guys on CVS). Sure now and then we will see bad fixes, but these are limited and the interfaces and modular approach ensures us that the integration is almost SELF MANAGED.
I don't claim that this applies to every succesful open source project (yes we are succesful
Code can be automated, in the integration sense, and if you don't have that, you won't keep the exponential growth of code base... there is NO management (read no man/woman) that can replace it.
marc
The real mnf999 always posts as anonymous coward
It took me a while to figure out, but "management" has a purpose. It wasn't until *I* had to be involved in, and manage, projects with *real* deadlines and goals, and a motley assortment of people with different skills and attitudes, that I realized this.
Managers should:
* Define goals, and save us from death by committee.
* Define a path to those goals, so people don't wander off and get lost. "Path" may be formal/informal methodology, standards, technology, milestones for features, etc.
* Keep people on task. Your presence along makes them feel guilty when they are not acheiving goals. I know this, because I waste my time reading Slashdot, and I feel guilty to my manager.
* Facilitate communication amongst people. Complexity of communication scales at n^2. Makes sure people know what others are doing and aren't blocked by dependencies.
This is not exactly coding, but it is definately work that needs to be done. The price a manager pays for not having to be ordered around, is that they are now free to:
* Get blamed when projects fail.
Remember, managers <!=> suits
It's 10 PM. Do you know if you're un-American?
The author does not seem to realize that the difficulty is in finding the bug in the first place. He seems to think the finding or the insight to find is trivial, and the coding of the fix, once found, is hard. This is the exact opposite of my own experience.
You're definitely right on about projects with no management - i'm sure it's a nightmare. What most projects need is organization that comes from below, not from above. Look at it this way:
Our current business model: Business is a separate layer existing on top of the development and production processes. Administration is for the benefit of business, not development/production, even though business's only logical purpose in existence is to organize and improve development. Administration has the most authority, so naturally they only work to benefit themselves. Developers/producers; i.e the real workers have basically no say in this process, which most likely interferes with the efficiency of their operation.
A better, more sane business model: Since the development of the product is the most crucial aspect of a business's purpose, developers should rule the company, with administrative, marketing, sales, etc. types as advisors on the deveoper's payroll. Money should be distributed by the people who do the most work within the company: the developers and producers! The administrators make suggestions, but have no real authority.
And yes, it could theoretically work, and could help to eliminate the fact that right now, business is just a separate society leeching off of the real society and bringing no benefit to it whatsoever (it's called a PARASITE.)
I've said my piece.
Although, you left some parts off of your corporate example. It should be more like this:
Your problem -> Your manager -> VP -> Big Boss -> another VP -> another manager -> grunt who actually fixes problem.
In the open source world there are few people (usually none) who are pure management. When you send your problem to people working on the linux kernel who forward it to Linus, they actually write the code and can solve the problem. In the open source world your bug reports and comments actually go straight to the ears of the people who not only solve the problems and implement new functionality but also decide what new functionality gets added in. It is the excising of the huge manager -> higher up manager -> yet higher manager -> ..... -> high enough management to make a decision -> lower manager -> yet more lower management -> .... -> manager intermediate step which usually has very little usefulness (often it's usefulness is in fact negative), that makes "The Bazaar" so special.
Case in point, OpenSSL. Great package, and I'm glad the developers went through the trouble to create it, however the documentation is all but nonexistant. I subscribed to the various mailing lists for a while and found that documentation basically was in the oral tradition form, with some third party, out of data stuff available (for SSLeay, the predecessor), and some examples. This is where a formal process in a corporate setting, with a profit motive is useful. Sure, the programmers don't want to document what they've done, management would rather do something else too, but the customer will insist on documentation, so you'd better produce it if you want to make sales.
Take this as a plea to make good documentation a part of your project, not a peripheral activity or one left for someone else to do after you're finished. My thanks go out to the projects which have been fabulously easy to use due to well written docs. I'm not going to name names as I'm sure to miss some.
Any ideas on where I can find it? I tried all the major search engines, and found references to it, but couldn't find the troll itself. To find the reference, search on google for "subtle art of trolling", find the article, and scroll down to the section called "The Successful Troll".
As far as I'm concerned, this is the Troll Holy Grail. Help me find it, please! BTW, can I be on the mailing list? I've never trolled, but I certainly enjoy them now and then.
No, Thursday's out. How about never - is never good for you?
The comparison fails because it tries to map the open source development model to the traditional development model. Of course there are going to be a lot of similarities as much of the open source development model output set easily fits into the definition of the traditional model. However, try mapping the traditional development model to the open source development model. One finds that the traditional model is restrictive and the open source development model is much more flexible.
The traditional model has more of a solid state, whereas the open source model is like a liquid. The management structure only resembles traditional development because those are the roles that people evolve theirselves. Liquids settle if undisturbed. However, no role is required and no pathway through the system is the only pathway. Should a manager fall out of the chain, or change goals, the open source model flows around it and meets at the next link. The article points out that Linux kernel development slowed when Linus had to take some time off. That's not true. The "official" releases dropped off, but development continued and releases were still made by the other managers. The development simply flowed around Linus and he caught up with it when he came back. Of course in the open source model all of the managers are developers. When anyone, not just Linus, disconnects from any network of developers, there is a loss of total production potential. However, if he disappeared from the face of the earth the same thing would happen, except a respected individual would be declared the "official" release source, and any specific skills he had been providing would need to be taken up by new developers. Also on this subject, releases occur at all points, not at the top level. I've made a change to the kernel for the benefit of myself and a very few other people. Although the audience was very small it was still a release, I became the manager. I set the goal, I followed the goal to completion, I checked the quality, and I made the decision to put the patch out. Note that this was also a highly parallel development, as the entire workload was handled by myself. No supervision was required. Being part of the network, I did consult the existing developers though. Eventually my patch was obsoleted by a parallel effort from another part of the development network.
As other posters have pointed out, the control of the management is with the developers. The development model graph really does look like the Bazaar network for many of the projects, it's just that the links between developers are redundant. The connections are there but aren't used frequently unless another link breaks. Keep in mind that large projects like the Linux kernel aren't just the tar ball put out by Linus. There are numerous projects running in parallel that provide modules for the kernel that aren't contained in the kernel, and none of those people have to report to or accept goals from anybody in the core kernel deveopment. It just happens that they do so in order to maintain compatibility. Some of these modules make it into the "official" kernel, even though they were never in the goals of the official kernel.
Be careful here. There is a distinct difference between what a customer needs, and what the customer thinks they need. When a customer comes to you requesting feature X , you, as a developer, should ask for the larger context that X fits into, and the larger need that's being filled. Perhaps X is difficult to implement and fits the need in a narrow fashion, but a different solution Y would fill the need better and may be easier to implement.
Really, the specification process needs to be a two-way street between the developers and the customers, so that the developers produce something truly useful for the customers, and the customers ensure their needs are met.
--Joe--
Program Intellivision!
Managers figure out how to use resources (i.e. people); architects design systems. While there's a little management in Open Source, most of my work with SAX (Simple API for XML) has been a combination of design and implementation. After all, I can hardly threaten to fire people or offer them bonuses or promotions, so I cannot make anyone do anything but what they wanted to do in the first place. That's what Eric meant, I think -- the architecture evolves according to what people are willing to do, not (usually) according to a strong central vision. I remember being part of the thread that annoyed Linus into allowing loadable kernel modules (hardly part of Linus's grand vision at the time).
I chaired a W3C working group for a year and spent most of my time worrying about absurd procedures designed by non-coders to make sure that the coders couldn't do any real work. That was management.
Do the Cathedral and the Bazaar means two different project management models or two different approaches to software development? Sure the open source projects may have a cathedral model, but it lives in a bazaar environment where competition thrives!
I think this is what ESR meant in The Cathedral and the Bazaar.
I fail to see how that saying that and saying bugs are difficult to find are mututally exclusive. Some of my dumbest users find bugs, that doesn't mean I'd want a million of them trying to patch the code. Nor does that mean that the leader developer alone could necessarily detect the bug. When you program, you develop a certain ingrained approach to what you're developing--you implicitely assume on some level that your testing efforts are going to be inclusive of all the users. Thus it is easy to develop certain blind spots, it's hard to look at something with a totally fresh set of eyes. Even if "fixing" the individual bugs is trivial, you have concurrency issues that CVS and the like can only go so far to solve. This is especially true when bugs, as they often do, require a higher level fix than just in the place where it makes itself known.
Go read "Debugging the Development Process" by Maguire.
The book is basically for the audience of "Technical Manager."
They might be called App Architect, Tech Lead, Technical Project Lead, Senior Engineer, whatever.
The role exists, and has existed for years, as the author of this essay points out... Brooks talked about it in the 70's.
Now one thing in a corporate environment, you have all the HR crap.
I think many technical people do not mind falling into the tech lead role. But most technical people don't want to deal with HR crap.
Divide that off onto another person, and you start to achieve success. Your tech leads provide their feedback into the reviews, but this team manager actually handles all the hiring, firing, training, salary, budget, whatever crap.
Many successful companies divide these duties up. Let a technical person manage the software development, and some other non-tech person handle the other crap.
The Debian management is a complete mess for example...
In the old days we had a benevolent dictator (Bruce), and it worked pretty well. Eventually he got tired of getting abused, and we had project leader elections.
Now, a few project leaders later, we got a Debian constitution which makes sure that:
Oh, and that's not it. The packaging manual is also subject to voting...
The sad part is that only a few vocal people within Debian drove the project where it stands now. These people should play nomic and let the developpers do their jobs.
It's almost like a 'worse is better' thing- the Bazaar stuff has a hard time being more sophisticated than Cathedral stuff, but if you have the connectivity, it becomes a hell of a resource just by virtue of statistics. Things that work out really well propagate- if I was giving people my mp3 of my 'Protocol' tape, some people might be looking for jazz, or Scofield, and notice the new tune, listen, go 'woh!' like Keanu Reeves and keep the track in their own collections with great delight- the new track would propagate based directly on how great it was, and if it was no good nobody would keep it. There are spooky parallels with genetic algorithms... only when you have a lot of random noise and junk does a GA-like effect start to happen, because it defines the situation as 'looking for good stuff amid the bad stuff' and forces quality judgements to be made.
I'll just leave you with this: in our society, actual work is done by a large amount of people; products are produced by a large amount of people, etc. Yet the small few at the top who do not actually produce any products; who do not know how or want to actually work for a living; who attempt to administrate despite their complete lack of knowledge about what is actually produced; those people retain the largest cut of the money and hand out the rest to everyone else who did something. Is there something wrong with this societal model??
Slave ethic. If they paid the actual workers at the top of the pyramid, they'd work for a year and retire, and the leadership would no longer have a work force.
At least that's my theory.
I hate it though. Most companies, if you're really good at what you do, they reward you by paying you more money to not do it any more (promotion to management). Where's the logic in any of this?
The answer, of course, is that business doesn't work on logic.
~ radiographite: art by john shepard
however, although organization may arise from chaos, good management is more than mere organization. it is the application of organization towards specific goals. this article points out that the requirement of such application is orthogonal to the source code fungibility, which, once again, underlines the difficulty of cultivating good leadership.
when goodness is lost, there is kindness. when kindness is lost, there is love. when love is lost, there is justice. a true bazaar is just, and no more. this is because it is impossible to expect from everyone compatible expressions of even love. the real world is beautiful like this.
what a thing is called is not the thing.
1) Not all managers are inept paper pushers.
2) The recent Ask Slashdot I was speaking of concerned a gentleman who had coded up a fix and could not find the right 'manager' to submit it to. The point being that anyone can actually write the code and can solve the problem but it still requires approval of the 'OS manager' to get it in the code base.
3) The bazaar model hypes the idea that everyone gets to decide how to fix a problem and decide how to put it in, not just the blessed few. Few projects follow this methodology, the point of the article and mine.
4)I've dealt with a few OS projects, and from my POV it is easier to talk to upper-management than deal with OS manager/engineers. Engineers tend to have an emotional attachment to the code they've written, and they don't like criticism of their 'baby'. The path I've taken in the past is to fix my local copy of the code and then post my fix to a newsgroup, or even to Slashdot, rather than fight that what I've fixed is an actual bug. Uppermanagement tends to have a myopic view of the world. Convince them that a change saves/makes money and it's in there.
If a company follows the process you outline, it is headed nowhere. Most companies I've seen are much more flexible about the introduction of ideas. At IBM the process was:
submit idea to study group->study group ask customers->study group makes a decision.
At my current job the process is:
submit an idea->start implementing it (unless the idea is completely off-base, the chances rejection are very small)
Aah, change is good. -- Rafiki
Yeah, but it ain't easy. -- Simba
I think the author set out with a poor understanding of the process, took a claim about the process much too far, and then set out to disprove a claim that wouldn't have needed to be disproven except that he took it too far in the first place.
Bruce
Bruce Perens.
Of course the myth of self-regulating Open Source projects is just that - a myth.
The reality, and why this model does work over the 'established' corporate model, is manifold:
* Centralized control in OS works, because the people in charge are so by virtue of their qualifications, and because they are accepted as thus by the rest of the developers working on the project.
In corporate environments, traditionally the least qualified individuals are hired to oversee and manage the project - this is even more true for ANY internet related venture. Furtermore, these people are hired because of their resumé, not their qualifications.
* The people doing the work, or managing the project do so because they have a stake in the project (the proverbial itch), and because they approached the project because of interest, not a paycheck.
In corporate environments, hiring chains are so long and convoluted that people are, again, hired by resumé and committee, not based on true qualifications, or interest for a project.
You *can* still get qualified personel in both instanaces using the corporate methods, but either it's rare, or if you get someone who knows what they are doing, their input is squelched by those above them.
Coporate structures are total, centralized empires, with small fiefdoms battling continuously for their own mini-empires, and self-justification - while not getting much of anything done.
OS projects are benign monarchies, that achive an objective because everyone is involved in wanting the project to succeed for their own reasons.
That probably sums it up best.
Harry
I'll believe it when I see it. The modularization that I refer to is not just an issue of organization, the issue (as I see it) is that it requires less effort/drive than a simple highly modularized driver does. Modularization in Linux's scope allows for a certain lackadaisical approach, that I don't believe larger [ even modularized] parts can. There is only so much that one can modularize away before it becomes excessively costly to performance.
/told/ to do it, and /told/ to do it a certain way. Modularizing and laying out a frame work can only go so far; they don't fully describe a path in the way that a manager can. Granted, there may be some significantly large and complex parts to Linux, but a modern day RDBMS requires a certain level of singularly sustained effort.
.01% less likely to fail ACID compliance test, who is going to thank that developer for that, for spending the past 2 weeks of his time to attain a level of safety that only a large corporation is apt to notice [and only then, when things go bad].
The "structure" is necessary not just to restrict additions that don't conform, but to drive, motivate, and encourage. Not every project is going to be particularly thrilling to the developer, sometimes it requires something a little more than the self-motivated desire to "scratch an itch". It's not even necessarily about greed. A developer may well think their efforts are better directed towards something else of their choosing; sometimes they may be right, but I've also known a number of situations where someone with the requisite skills must be
I feel this will be especially true when the open source developers efforts fade more and more into the background amongst many others. When say, Alan Cox writes a driver for my particular piece of hardware, he's sure to recieve plenty of thanks, especially if it's fast. But when one modification of one condition of an INSERT statement that is just
I realize many people will disagree with me there, but I remain skeptical. So I reiterate, I'll believe it when I see it. There may be a way, but not the way popular "Open Source" is envisioned.
For a start, if you can provide the goods you can march straight into the "boardroom" and work right alongside the "CEO" on the kernel.
Bzzt. Wrong. Thanks for playing. The Ask Slashdot question I posted about concerned this VERY issue. The person had a problem that he had a fix for. He marched to the "boardroom", fix in hand, and was soundly ignored by the "CEO" and all his "VPs".
Secondly, with any GPL product, large or small, you have the right to take the whole thing, make your own modifications and push it as the way to go. Try doing that with the real-life component produced by the company of a real life CEO.
Of course, attracting support for your version, your vision, is something else. But if you have the talent and the product, it can be done.
Translation: If you don't like how the project is being managed, you can go start your own company and manage it yourself.
Note that this happens a LOT. Engineers left Palm and created the Handspring. Engineers left Intel and created Weitek (math co-processor for the 386). This doesn't change the fact in any way that large, successful OS projects have a rigid management heirarchy. Be they good a knowledgeable, or be they bad and uninformed, they are management and they are indeed rigid. You will comply with their decisions/edicts, or you will have to start your own company/project.
Aah, change is good. -- Rafiki
Yeah, but it ain't easy. -- Simba
Why the fuck can't you take 2 minutes off to read the damn artcile before commenting? No matter how important what you have to say is, read the article first. Yes, mark me as a troll, I do not care, but I read the entire article, word for word, and it is in my bookmark. I also agree with 95% of the content.
------ Curiosity killed the cat. {satisfaction brought it back | it didn't die ignorant | lack of it is killing mankind
I also feel you're giving too much power to marketing. Marketing is not "on the same level" as development, it exists in a totally different dimension, and should be viewed as a necessary thing given that we're developers, not as an end in itself.
The fact that marketeers think that marketing is an end in itself is a big problem.
I might be misunderstanding you, though, because in the Open Source case you seem to consider marketing as "what the user community wants" rather than as the process of finding out what the user community wants, and selling it to them. (Just because money doesn't change hands doesn't mean the OSS community does no marketing!)
--
It's a
-- Danny Vermin
...but you've already made at least one fallacy in the intro. Having "strong central control" is orthogonal to "self-organization". The former is about who is in control the latter is about how they got there.
What's the diff? Well, in a non-self-orgranizing project there are by definition external (i.e. non-contributing) organizing forces--this is management. Linux doesn't have that, despite having "strong central control".
--
Linux MAPI Server!
http://www.openone.com/software/MailOne/
Linux MAPI Server!
http://www.openone.com/software/MailOne/
(Exchange Migration HOWTO coming soon)
I am co-developer of fancylogin, a neat login program for Linux, and we have big troubles organizing ourselves. Especially in the beginning we often had dialogues like "did you already implement feature foo? - No, I'm currently working on feature bar - damn, so do I". Software engineering in general can't work in this way, not only at open source projects.
A monkey is doing the real work for me.
The difference is that Open Source Projects involve programmers managing programmers. As such, there is a level of understanding that is not present in a corporate bureaucracy, where feature sets and codebases are controlled from "above." While there must a a controlling authority of an OS project, that determines which modifications get included in the codebase, there is no guiding principle for development other than putting out good code--no hidden motives and no idiot managers.
ByteMyCode.com: A Web 2.0 code sharing community.
What we need is a source forge like system that, once a project hs been created, lets anyone upload a patch/update, and have that fork (or a "press release" for it) appear on the origanal front-page.
Its just an naked idea, but it could be developed into a powerful system. The big boys of OS should think about it.
Well, thats my idea. I'm letting go now. Develop it, fork it.
Thad
Thad
You are confusing what the metaphor is. The Cathedral vs. Bazaar metaphor (at least as the author of the paper in question is using it) is between the method of their being constructed, not the final constructions.
I confess this entire metaphor has irritated me for some time. It betrays a lack of understanding how actual cathedrals were actually built. In the days when they were popping up like toadstools (i.e. ~12th century), the "architects" were an exalted flavor of stone masons, and as such shared a technical vocabulary and culture with the skilled craftsmen (implementors) who worked for them.
It irritates me that these medieval engineers (such as Villard de Honnecourt who was the Geeks' Geek of his age) are being lumped in with PHBs and the skilled workers who dressed the stone and raised the arches are being characterized as peasant grunts -- both are thereby slandered.
----------------------------------------------
-*- Any technology indistinguishable from magic is insufficiently advanced -*-
Did you even read the article?
/nobody/ would dispute that he's the central manager, the person who vets all ideas before they can be implemented in the language or interpreter. Hardly the "bazaar" that Raymond posits -- in fact, he's very much a traditional software project manager in this regard.
He doesn't speak to the viability of OSS; he is disputing the often restated premise that open source projects "manage themselves."
You mention XEmacs and Perl; wouldn't you say that both projects have distinct and very stringent management structures? Sure, XEmacs forked from FSF Emacs -- but it's just as strictly controlled.
And while Larry Wall does indeed delegate authority to the various Pumpkings,
The author is making a point about the organization of large software projects, that is orthogonal to the licensing issue. What his conclusions seemingly do imply, though, is that open source is not by itself a revolutionary development in the world of software engineering.
(jfb)
To spur "enterprise Linux," Big Bang, the distributed two-phase commit.
Key to designing any successful software project is having strong input subject area experts. It's no suprise the developers can write good compilers and programming editors, because these things are an integral part of developers' first-hand experiences.
The problems start when developers write software for problems that are outside their personal skills or experiences. Without the structure of a conventional organization to ensure that software get designed and written in a user-centric manner, projects like this are doomed. And even with such an organization, things are probably going to get very painful for everyone involved.
Of course there are the developers who have a deep understanding of disciplines outside of the realm of programming, and these people often go on to create great software outside of the realm of developer tools. Photoshop is a great example of an application originally developed by a programmer-artist. (And maybe Gimp too, but I don't use it, because I'm willing to pay for my tools, and while my primary development and mail-reading paltform is FreeBSD, I do my design work on a Mac.)
Anyway, how many people are there out there who: 1.) are technically clueful enough to program their way out of a paper bag, 2.) have the administrative skills to get a project out of the one-person phase, 3.) are equally talented in some non-technological discipline, and 4.) have been brainwashed by Richard Stallman to such an extent that they don't reaize how successful they could be writing commercial software?
I don't think such people are very common.
I don't entirely buy this. While an absolutely tremendous amount can be done via email, don't knock high-bandwidth communication, i.e. f2f, or the dreaded meeting.
Meetings don't necessarially have to be what you think of them as -- they can be simple hallway chats or dropping by someone's office. But frequently they are *essential* to breaking through conceptual logjams and coming to a consensus.
I'm currently involved with a commercial development project in the US and Europe, and we use email, but we also have weekly videoconferences, and exchange staff every quarter. You just can't beat it.
I think we have here a which came first, the chicken or the egg question?
The fact is that open-source projects can not be controlled by anyone, but everyone has a need that requires other peoples cooperation, so there is a strong pressure to unify to get maximum benefit from a limited number of developers. In a propriatory software world, the pressures are to differentiate yourself from competitors - however old style competition does not make sense in software development.
In a free market society, having seperate factories for say shoes creates pressures to keep costs down for each unit and innovate - however in software that model breaks down - having seperate software development shopes developing the same code is silly because the natural cost of copying is minimal compaired to providing to people who need service or improvements on existing code bases.
Let us not forget that intellectual property is not a basic property right, just because the government calls something a property does not make it so. It was never even designed to be a property right, and it's ignorance of fundameltal economics like natural limits in supply and demand (which is artifically put on information instead of services) dooms it to failure, as well as closed centralized software projects.
One problem is that the role of 'technical manager' doesn't really exist in the standard business world. I see very few companies that have technical managers, or, if they do, have ones that are technically competent.
OTOH, I know people who would be technically competent who shy away from any kind of management role because they no longer get to do interesting things to code. They no longer get to make technical contributions.
The Open Source model solves both these problems. I think the problem of good technical managers being incredibly rare is more a failure of the standard business process to find and select such people than it is a lack of capable people.
I just read the book 'The Cathedral and the Bazaar', and I noticed the same problems with the 'Bazaar' model as this author has noted. There was still a central person who made most of the important architectural descisions.
The Open Source model mainly takes advantage of the fact the the net makes wide hierarchies much more manageable. It also reduces a lot of the corporate politics associated with management hierarchies because there are many fewer steps to the top.
Need a Python, C++, Unix, Linux develop
I haven't read it either, but I was going to say the same thing. Whether there is centralized control or not, and whether the project "organizes itself" are not related.
More to the point, the voluntary participation of all involved means that the better (from the participants' point of view) manager is available to them. If they aren't producing what he wants, he doesn't incorporate their work. If he isn't realizing their goals, they won't contribute. If the whole project is less successful from the user's point of view, they can "fix it themselves" or more likely use the product that is better for them. Don't get me wrong, I know there are bloody wars in any project, large or small. The difference is that when you're in a bad position at work, you have to consider changing your whole life to get around it. With voluntary projects you aren't forced to choose between your livelyhood and a bad manager.
Freedom of choice means none of the traditional metrics of software development matter. It doesn't matter who's code is cleaner, has less bugs, runs faster, looks better, who manages their project the best, who works the hardest, or which tastes better. You have the choice to pick the one that suits your needs, no matter what your role in the system.
This is the same mis-understanding people have about political freedoms. People seem to think that freedom is advertised as "automatically better", but that's not it at all. Freedom means you have to make your own life better, and there's noone else to blame but yourself.
Of course, this is true no matter what your environment is, but with free software and all other freedoms, there's less mis-representation. Noone is claiming to take care of you.
Here are some my random thoughts on management and software production. First two disclaimers...
(1) I'm spoiled. I work in a shop with some excellent managers.
(2) I also work in a somewhat atypical shop with thousands of programmers on the same project. While not exactly unheard of, projects of the size of mine are not exactly the most common.
If it weren't for our managers of excellent quality, little of what we need to get done would get done in an efficient manner. Our managers help prioritize hours to be spent working on defects, new features, enhancement requests, etc. They work with developers to come up with reasonable and workable deadlines. They do a whole lot of logistics so we lazy programmers can just kick back and code.
Now, in the past I have worked with some bad managers. Some of the managers of projects I've been on have been so bad that they would have been better with no management at all.
Good management is an essential part of the software development process. It doesn't necessarily take someone with the title of manager. But it does take someone that can manage the project well. Some few programmers (especially the high profile open source ones like ESR, Linus, and Miguel) also happen to be fairly competent as managers of their respective projects.
However, given that out of all people that do programming tasks relatively few are what I would consider 'competent' and given that out of all the people that do management tasks relatively few are what I would consider competent, the union between competent programmers and competent managers is certain to be pretty small.
have a day....
-l
I agree with the analysis, but not completely with the conclusions. Certainly, even in Bazaar style projects there is central management. And even in Cathedral projects, good ideas originate outside of the central authority guiding the project. The dichotomy between the two is a metaphor to describe the degree of control that the central management of the project exerts, the degree of ownership that is claimed. It helps explain not the actual flow of information (ideas and code), but the psychology of the relationships between the actors.
There are elements of the Bazaar in Cathedral projects. Many of the elements of Emacs began as ideas that were never suggested by Richard Stallman. For all his coding talent and vision for the future of his creation, he is still one man. He eats and sleeps and lives the same 24 hour days as the rest of us. Others come up with answers to their own problems, scratching their own itches. His control has been as a gatekeeper, determining what is in and what is out.
The article itself gives examples of the Cathedral in Bazaar projects. But it overlooks the spirit of the participation. There is a feeling of shared ownership. To the extent that a volunteer participates in the project, he feels a joint ownership of it.
Perhaps I am merely stating my own impressions. I have contributed in minor ways to both styles of projects. Both are open enough for my taste. But I can feel the difference in the degree to which the control is perceived to be exercised. It is really a difference in management style rather than the presence or absence of management. But on such things a project can succeed or fail.
The net will not be what we demand, but what we make it. Build it well.
... am I the only one wondering how you'd set a flamethrower to "stun" ?
If fetchmail and Linux were not run as bazaar projects, what would a true bazaar project look like? A real bazaar software development method would proceed as follows.
Hmm; doesn't this sound suspiciosly similar to the way distrobutions work?
(The article did have some interesting notes though.)
There was an Ask Slashdot question a few days ago about how you make contributions to an OS project. Several of the answers went along the line of read the mailing list and send your bugfix to some of the lesser daemons. Let them take it to the great gods. This hints that an OS project (especially one as important as the kernal) can quickly have a structure identical to Figure 1 of the article.
If you have a problem at work, you talk to your manager, who will speak with the VP, who consults the CEO. How is this any different from emailing the 'less important people on the mailing list', who then contact the component owners, who might be able to get a response from Linus?
Aah, change is good. -- Rafiki
Yeah, but it ain't easy. -- Simba