Shuttleworth on Open Source Development

An anonymous reader writes "Mark Shuttleworth (retired cosmonaut and Ubuntu daddy) has written an informative blog entry about the problems associated with open source development. He found that paying geeks to code without assigning them managers lead to "shiny geek toys", rather than the product he was actually paying for. Shuttleworth says that left-field thinking is required when it comes to managing open source teams. See also Andrew Orlowski's analysis of why AOL eventually killed the Netscape project from a few years ago, where he describes Mozilla developers as "wandering off into Lotus-eating land"."

Comments

Exactly...

[raydobbs]

You can have all the creativity you want - but without proper leadership, all that effort and talent goes wasted. I have a few creative friends that have all these wonderful ideas - but they have no idea on the concepts of project planning or management of resources. Needless to say, their killer applications are still brain children - and not actually out here where the rest of us can use them.

Re:Exactly...

[Stoned4Life]

I think what is, not so much overlooked, but not always considered when thinking of Open Source projects, is that these developers are not working on this Full-Time (usually). This is a project divided among a group who all either share an interest in the resultant, or were brought together to contribute their knowledge in certain fields. It is unlikely that they would have previously held management positions or have acted as project leaders- not to just to govern themselves, but to hold a leadership over a group. What needs to be added to the equation is a person who has experience in managing people to reach a certain goal. I think he had some pros and cons in relying on the group to choose the route to take in designing his project. What needs to be laid out before diving in is the issues that you want to tackle, the ways you plan on tackling them, and the resources you have available. I guess I'm just preaching to the choir when I say all this, but that's just how it has to be. In the end there has to be just one person to make a decision.

Re:Exactly...

[eno2001]

The problem with "leaders" is in how much they actually understand what's going on. Typically the people with strong leadership skills are totally clueless when it comes to understanding what is technically realistic. You've got people out there who are strong leaders but actually believe that if they wanted to have flying minibots all over the place that use anti-grav drives for security services, that it's possible. Hate to break it to them, but it's NOT possible. That's the typical leader with strong skills. What's needed are people with strong technical skills who can lead a group of other coders. That is an impossible requirement because the people who excel at technical abilities are typically horrible leaders. Usually because they don't understand how to interact properly with others, or they can't let people work in their own way and they try to force their staff to be just like them. It's a rare person who understands technology enough to design realistic products AND can manage other people.

Re:Exactly...

[QRDeNameland]

I think one of the reasons that technical people make such poor managers is due to inflexible thinking in many corporate environments. In every place that I've worked, I've seen managers that were quite good at managing projects (in terms of getting the people they manage to do great work that meets or exceeds expectations), but invariably that management position comes along with expanding amounts of corporate middle management stuff (reviews, HR, tracking employee hours, days off, etc., etc.) which such people either hate or are poor at or both.

The result is that many people who could competently manage a technical project tend to avoid such positions because of the BS that goes along with it, and the people who do aspire to the management positions are the ones that aren't that good technically and, though they may be decent managers in the corporate sense, often times are over their heads when providing direction on the actual project.

The best solution in my experience would be to decouple technical project management from corporate/HR management. The person who leads the project should be focused on the project, and let it be someone else's job to deal with the corporate/employee issues.

Old article

[Proud like a god]

From the article:

This entry was posted on Friday, November 21st, 2003 at 6:48 pm...

A little out of touch maybe?

Re:Old article

[skoaldipper]

A tad bit old, yes. The Registry article equally so. Is this blast from the past Tuesday here on /.? If so, might I request an article or two on Deborah Harry of Blondie fame? Circa 1982? I've been thinking about her all day long. Just make something up about guitar technology or somethiing to CYA on a tech news site afterall. And please include pics of Deborah and the guitar, or just Deborah. You decide.

Re:Old article

[Capt James McCarthy]

From the article:
This entry was posted on Friday, November 21st, 2003 at 6:48 pm...
A little out of touch maybe?

No, no, it just took that long to be signed off by all the department heads and then approved by upper management.

I'm not so sure....

[knarph]

that shiny geek toys are all that bad. I can't think of one thing that my grandmother (who is as far from a geek as one can get) uses every day that wasn't once a shiny geek toy to someone.

Re:I'm not so sure....

[knarph]

Sure enough, but if the geek toys were never made, she'd still be using the same crap that was around when she was born, but with a better interface. I'm sure putting a new UI on a steam engine would do it some good, but only to a point.

Re:I'm not so sure....

[Chosen Reject]

putting a new UI on a steam engine would do it some good, but only to a point.

Valve did that and it seemed to work out just fine for them

/ducks

Mozilla - ouch.

[Spy der Mann]

I can relate to that comment, I've been waiting for Mozilla to implement Internet Explorer compatibility (XSLT extensions) and ACID2 compliance for a while. Even with the 10% market share Firefox enjoys, they still don't facilitate the programmers to replace existing IE applications.

I also agree with this:

Creating a neat C++ framework when what the world really needs a non-Microsoft browser is nothing but a deriliction of duty: a piece of vanity code. What we Brits call pointless "willy waving".

I really hated Internet Explorer. When I heard about Mozilla, I tried Milestone 8 (around 1999), and it was slow as a snail on my poor machine. WTF were they thinking? The Netscape code might have been difficult to maintain, but what really needed a revamp was the html renderer.

The reason Firefox did get a huge market share is not because of the XUL framework, but because it was finished. I'm sure all that delay could've been avoided.

Re:Mozilla - ouch.

[jbolden]

At the time Mozilla started there wasn't a freely available cross platform widget set. XUL filled the need. A year or two later they would have used the Gnome projects GTK but didn't exist yet. Besides if the browser project had failed, and Gnome hadn't come along XUL might have been the important technology.

They couldn't pick QT because Netscape was not going to be GPLed and they did want Mozilla to be open source.

Who woulda thunk it?

[smooth wombat]

He found that paying geeks to code without assigning them managers lead to "shiny geek toys", rather than the product he was actually paying for.

Do ya think? How long did it take him to reach that conclusion?

Seriously folks, this is a given and one of the main reasons I don't buy into all the hype about the electronic toy du jour. Everytime I see an article somewhere which says that 'X' is the latest electronic whiz toy that everyone must have I just roll my eyes and move along. (As a side note to marketers, I don't watch your commercials or read your flyers in the paper. You may now explode with unmitigated rage because I'm stealing from you for not watching what you produce.)

I don't want to be forced to buy a DVD player which plays DVDs, mpegs, connects to the net, calls my vet or offers me advice on what wine goes well with acadian rigatoni. I want the machine to play DVDs. Period.

By their very nature geeks (true geeks) will shovel every bell and whistle into a device they can get away with because that is what they do. They want to see how much cruft they can tack onto the hardware simply to see if it can be done. Top that off with manuals (the paper ones if you're lucky enough to get one) which are so poorly written and obtuse that the average user has to take lessons to learn how to program their device, and the market becomes filled with devices whose half-life is as long as the life of a fruit fly.

To all who produce this crap, here's a hint: Stop making a swiss army knife out of every product. If you absolutely must put tinsel on the tree, make three trees. The first is bare bones (i.e. just a cell phone. no music, games, etc). The second has a few more items (include games and music). The third has everything (bleeding edge). If you check your sales figures you'll be surprised to learn which one sells the best (hint: it's not number three).

Re:Who woulda thunk it?

[DiSKiLLeR]

If you check your sales figures you'll be surprised to learn which one sells the best (hint: it's not number three).>

Actually, I think, you will be quite surprised to find out that it actually IS number three.

Re:Who woulda thunk it?

[meringuoid]

By their very nature geeks (true geeks) will shovel every bell and whistle into a device they can get away with because that is what they do.

Only if that device is not a true Device.

A true Device does one thing and does that one thing well; it has clearly defined inputs and does not mind what the input comes from, and it has clearly defined outputs and does not mind what the output goes to.

Then the Geek is happy, for with many such Devices and an assortment of cables the Geek can assemble a composite System that meets his needs exactly. It is the True Way. It is the UNIX Way.

Thus it is that my TV aerial cable goes into the back of the digibox, whose output then goes into the back of the VCR, whose output then goes into the TV card, whose output then goes to mplayer, whose output goes to the screen.

But if a device tries to thwart a Geek in this fine pursuit? Then it is that the Geek takes it up as a challenge to force that device to do his own bidding, to mod it to suit himself, to make it act as a Device and not as a mere device.

Re:Who woulda thunk it?

[Ed Avis]

By their very nature geeks (true geeks) will shovel every bell and whistle into a device they can get away with because that is what they do.

The true geek will make it as minimal as possible, stripping out features until you get down to a barebones command line interface. (That's not what grandma wants either.)

It is often marketing departments who are responsible for your DVD player offering you 'premium' or 'sponsored' content recommending particular wines.

Re:Who woulda thunk it?

[Vellmont]

By their very nature geeks (true geeks) will shovel every bell and whistle into a device they can get away with because that is what they do.


I guess I'm not a "true geek" then. There's definitely a set of people that will do just that. There's also a very large amount of people that follow the mantra "Keep it simple, stupid". You really don't need to look much farther than all the extremely successfull open source software projects to know that what you're saying simply isn't true. Is Linus Torvalds not a "true geek" because he's an extremely practical leader?

I guess I'm not exactly sure why you're even blaming this phenomenon on un-managed geeks, or even open source. Have you opened up Microsoft Word lately? It's full of every bell and whistle you could imagine.

Not only open source projects...

[herve_masson]

I agree with most of what he said, except I don't think its limited to open source projects. I have seen that on purely commercial context as well. The problem is that you *need* some kind of "geek toys" occasionnally, because they sometimes give birth to a very valuable technology (I've seen that many times). That's a complex task to find the fair balance between what is reasonable/valuable and what is not in term of focus diversion, and that's a hell of a management task to deal with people who can't see that balance (either way).

Why blame OSS?

[ameoba]

It sounds like he hired a team of talented but flakey developers and is generalizing this to all OSS development. I don't think the problem has anything to do with OSS - it has to do with a team of guys thinking they have free reign to do what they want with no expectations, deadlines or oversight.

Olde news?

[Anonymous Coward]

Ok. For those that didn't realize it, that blog entry was from 2003. Today, three years later on, where is the SchoolTool project? Did Mark really learn a lesson and develop a solution or did he just relive a trend he noticed so long ago?

Seems to be a long development cycle for a specialized calendar. I'm glad I'm not paying for it.

Re:His project needs an architect

[AKAImBatman]

I think his project ultimately would have been successful if he'd started with a strong architectural design.

Strong architectural design definitely helps. However, it's not the be-all-to-end-all. In OSS development you have to be aware that your programmers are volunteers. They can and WILL step out the door at inopportune times, start arguements over architectural designs, and spend time working on what they think is cool rather than what is needed.

To get a project to absorb much of this chaos, you can do a few things to help the project:

1. Start with an existing codebase, usually an intial version written by a single author.
2. Hire programmers who you can tell what to do (and fire if they don't) to get a core going.
3. Get your architect to contribute code to the vision.
4. Start a competition between areas of the project to see who can hit more milestones. (This isn't easy, but it's great motivation when it works.)
5. State that something is impossible just so that someone produces the "impossible" code. (Sneaky, I know. :-P)

Re:Obviously!

[ErroneousBee]

Orlowski is miles off.

See how Firefox developed once it came out from under a corporate yoke. All those shiny geek toys (XUL, plugins, etc) started getting the attention they needed instead of making it work on an infinite number of badly written web pages.

Orlowski is just a hack who slags things off on the cusp of thier sucess. Hes turned The Register into a personal rant blogg, dont be suprised when it goes bankrupt.

Mark Shuttleworth on the other hand clearly states the problem, gives a lucid account of its causes, and proposes a solution. Maybe you should read his article and actually learn something.

Not some huge revelation...

[dasil003]

I'm not sure exactly how it went down, but it sounds like he hired a bunch of developers for a project and just sent them off to go do it without leadership. This isn't a problem with open source, it's just a boneheaded decision. When you hire someone you have to train them and tell them what you expect. It's no wonder that they gravitated towards whatever they wanted to do since they had no direction.

You can have all the creativity you want - but without proper leadership, all that effort and talent goes wasted. I have a few creative friends that have all these wonderful ideas - but they have no idea on the concepts of project planning or management of resources. Needless to say, their killer applications are still brain children - and not actually out here where the rest of us can use them.

In that case self-management is the key. I've been there. Working for years in an educational environment where the actual workload was less than 20 hours, I had a lot of freedom to take things in new directions. I ended up coming up with some of my best ideas and was able to develop the discipline to implement them. But it was really hard not to get distracted. You have to develop a manager mentality--be results oriented. As a programmer / designer / creative, sometimes spending 8 hours just researching or learning something is well worth it, but at some point you have to jump in and focus hard on the final product until its done. Then you can go back into creative mode and dream up version 2.0.

Re:Not some huge revelation...

[xenocide2]

Part of what makes Linux and GPL'd software so nifty is that with access to the source code one can do all sorts of wonderful and unexpected things. Port wondershape to the wrt54g. Replace svgalib with aalib and seamlessly render images and video streams as ascii art. Fit linux onto all sorts of silly places, including a windows device driver. Tune the linux scheduler parameters using adaptive genetic algorithms. Cook up packages for compiz before the distro puts it into stable. The ability to think outside the box and hack things in this manner is simultaneously the strongest advantage OSS has, and it's greatest obstacle.

The greatest obstacle? Firstly, if you simply hire people familiar with open source, those who recognize the value of the above traits, you're likely to get something that satisfies their needs not yours. Maybe that compiz package requires a lot of extra effort on your part, because nobody's written a script to handle the simple textfile changes nessecary. Secondly, integrating silly hacks back into the core is a challenge. On the one hand, integrating them into the core encourages more, which we like. It also means that the hack gets all the benefits of future improvements. On the other hand, not every hack is easily maintainable, nor easily integrated. Every time you reject a patch, you discourage people from offering in the future, and the risk of someone pissed off and forking your project increases. Not that forking things is always bad, but a fork in spite is bound to not only divide resources but increase overheads, potentially causing a net loss in future value of the software.

Shuttleworth believes he can fix this by making communication among groups more explicit. I doubt it will improve anything. On his next attempt, perhaps he should make his team a stakeholder in a very dear sense. Not bonuses for completion or anything silly. Make them run a school with the software -- Eat their own dogfood. The core team will have to shift their focus on making the software work for them in ways similar to other schools. Maybe they could start a school about hacking on OSS. Teach the newbs the labrynthian ways of the autotools, how to take a tarball and make it a .deb, that sort of thing. Something the team wants to be successful and proud of.

And what else did you expect?

[dchallender]

As far as I could see, very little in the way of specification, design / architecting.
Without a reasonable framework it was inevitable the project collapsed.

The actual coding should be a minor part of a project, the real blood, sweat and tears is the spec and the architecting / design (and usability / test side of things): If that is done well enough then the coding should be a simple join the dots task.

Without architecture / design constraints then you will get toys for the boys (and girls) as there is no pressure / direction on them to do otherwise.

XUL

[Britz]

I always kept wondering what exactly XUL was developed for when a browser was needed. I don't know the timeline, but wasn't Gtk ready about the time they started Mozilla? I know that Qt was for a long time worthless for cross platform free stuff, because Trolltech charged money for the win32 version (which they had every right to do so).

Schooltool link

[BeardsmoreA]

In case people are too lazy to spend 3 seconds on google... (Which from some comments above seems to be the case)

http://www.schooltool.org/

Summary of current status as I read it: SchoolTool still isn't really there, but they did manage to get the spinoff 'SchoolBell' out there, and the SchoolTool work is ongoing and being included in the 'Edubuntu' distro.

A Generic Failure

[jamesl]

This has nothing to do with Open Source. It is about trying to develop a product without a spec., without an architect, without management and without a timeline. Kind of like pointing a group of carpenters at an empty lot and telling them to build a school.

It wouldn't be any more or less successful at Microsoft, IBM or SAS.

Orlowski is sooo wrong (and today we know it)

[Anonymous Coward]

Actually, Orlowski reasons for deriding the Mozilla team in "wander[ing] off into Lotus-eating land" are:

"creating esoteric frameworks". Later we learn that means "Creating a neat C++ framework when what the world really needs a non-Microsoft browser is nothing but a deriliction of duty: a piece of vanity code". Except http://weblogs.mozillazine.org/ben/archives/009698 .html/ shows XUL creation was a direct effect of AOL pressure on advertising and netscape portal integration

"note-perfect bug tracking systems that only a nerd could appreciate". Anyone who ever looked at bugzilla's internal knows it's a quick and ugly hack who could never mobilise a whole team for years.

So when Orlowski writes jokingly "corrupt suits at AOL" "betrayed the Great Noble Project" he's right, and when he rants about nerds he's dead wrong (you'll notice once the inspired suits where taken of the picture the nerds did produce a successful browser).

If anything, returning on Orlowski's pontifications today only emonstrates the depths of his prejudices and cluelessness.

30 year old philosphy...

[deadlinegrunt]

...is still valid? Do one thing, do it well.

Imagine that - simple, solid advice survives time. Reminds me of the Twelve Networks Truths of RFC 1925 Section 2-11

Example: Why start Adept when we have YAST?

[UseFree.org]

Mark Shuttleworth is subsidizing the Kubuntu team is working on a software installer named Adept. I find this to be rather wasteful, since there is already an extremely feature-rich, robust and mature installer from SUSE named YAST. YAST is Free and Open Source (GPL) and it is built on the Qt/KDE framework and integrated in the KDE Control Center, so it would fit very nicely in the Kubuntu environment.

YaST is the app that makes the proverbial "Linux on the Desktop" a reality. It is the most robust, comprehensive and user-friendly configuration tool for GNU/Linux -- software management, hardware detection, system administration and much more. In short, it is everything the average newbie from W$ needs to set up and update his computer without having to touch the command line.

Devising a new GUI app for installing packages is reinventing the wheel by duplicating the gigantic functionality of YAST. This project will only yield a half-baked solution that will get abandoned as soon as it starts tackling the more thorny issues that YAST has already solved.

The YAST code is clean, and has already been used by Linux distros like Yoper, so it is definitely feasible to get it running under Debian/Kubuntu if their devs don't start reinventing the wheel. YAST might be complex, but then any program that excels at setting up and updating a Desktop Linux system is going to become complex no matter what.

You can't have it both ways

[MikeRT]

Developers of OSS often forget that they have two choices in most cases:

1) Meet the needs of their users and especially those who want to use their products
2) Meet their own needs

OSS developers need to stop using the argument that "feature X is missing because we're hobbyists." If you want to compete with the big guys, you need to give your users the features they want. It's certainly your right to prioritize based on your wants, but don't kid yourselves. If you don't give the users what they want... they'll leave.

I feel like I've been had

[phoenixdna]

It was an interesting article and well worth reading. But the fact that this is 3 years old is relevant in my opinion. I'm a little put off by the fact that this wasn't noted in the excerpt for the headline. Submitting this and passing it off as current news for the sake of making a point is bad form. This particular blog isn't new at all and so you know that someone didn't submit this just because they found it for the first time today and thought it was valid. Its a lot more likey that someone wanted to make a public point about open source software and dug this blog up out of the archives in order to do it. It you can find a newer link than this to make your point, then maybe this particular view is outdated and not worth spending very much time on.

Re:A bit old?

[colinrichardday]

Geez, the guy posts a minute after the previous post (so he probably didn't see it) and he gets modded -1 (redundant). Won't anyone think of the latency?

I spoke with the head of SchoolTool

[xeno-cat]

I spoke with the project manager for SchoolTool last year when he was at an educational conference in my area. He said that what Mark basicaly learned with the first (Java based) SchoolTool is not to start a project and then go into space. Goo advice for anyone committing to large product development.

The current SchoolTool is being written in Zope3 and is under tighter development control.

This is very old news and does not reflect the current understandings of either SchoolTool or Marc Shuttleworth. This article could also be called "My first babysteps in the universe of Open Source development", file under ancient history.

Kind Regards

SchoolTool Update

[krasni_bor]

Funny to see this piece dredged up again. I'm the blogger Mark references in the story, Tom Hoffman, and for the past year and a half I've been managing the renewed SchoolTool development effort, after Steve Alexander created a new Zope 3 based architecture.

It is definitely tricky to manage a project with such broad and lofty goals, and we've still had our share of mis-steps and mis-directions. I have a background as a teacher and self-taught Zope hacker, so I've learned a lot of lessons about software development.

Nonetheless, a useful application is in sight. We'll have a beta this spring and serious testing in real schools in the fall of 2006. One key this time around was keeping the burn rate down and not creating specific expectations in schools and with governments that we subsequently failed to meet.

If you're interested in open source software for schools, check out http://schooltool.org./

Peer Review

[demon411]

what we need is not management BUT

-payment to coder only when the product meets requirements
(why did anyone get paid if all you got were shiny toys!)
-select coders who can self manage
-peer review

Peer Review is very important! You could have college students doing it, as long as someone goes in there and checks that the code does what it says it should.

Was in process of moderating but removed my moderation to make this comment

Paying geeks to code without managers

[heroine]

Seems lots of places do quite well with egalitarian structures. No-one complains about India, Japan, China putting out shiny geek toys or wandering into lotus land even though they don't have American "org charts".

The issue is more to do with programmers who can't stay on track rather than programmers who ignore the "org chart".

I wonder if this has anything to do with The GIMP?

[tap]

People have been wanting 16-bit color and CMYK support in the gimp since the previous century. FilmGimp aka Cinepaint was the gimp with 16-bit years ago. Why does the gimp still not have 16-bit color when the code has been around for years?

The answer is GEGL, a non-existant "shiny geek toy". GEGL is supposed to be some amazing framework that will handle image operations the Right Way. It will make 16-bit color, CMYK, and adjustment layers appear by magic. It will be fast and generalized and light-years beyond anything Adobe has and wash your windows for you. Who knows what it is supposed to do now? Unlike the codebase of GEGL, the legend of GEGL grows by leaps and bounds.

It you read the gimp devel list archives, you'll see many cases of people saying, "I want to code CMYK", or, "I have 16 bit support". The developers always send them away, "You are doing things the Wrong Way, you must work on GEGL instead!" The result is, development is killed.

What of GEGL? Years go by and it's nothing more a "design document" aka Musings of a Lotus-Eater, that hasn't been updated since the Clinton administration. A CVS repository that goes eight months at a time between commits. No code that actually compiles and does anything. It's still just a pipe-dream shiny geek toy.

Mark Shutteworth tried to fund someone to work on GEGL. I imagine nothing ever came of it.