Open Source Project Management Lessons

cpfeifer writes "Paul Baranowski takes a moment to reflect on Open Source Project Management in his blog. His reflections are based on the first two years of the Peek-a-booty project." Interesting comments on media coverage, choice of programming language, when to release a project, and more.

Comments

Great, I would love to read all about it

[mao che minh]

I would love to read the article, but my employer uses a proxy filter program that filters out sites with the term "proxy avoidance" in it's META tags (or otherwise prominently displayed within the header). If you are going to run a site dedicated to the development of software that allows one to avoid detection systems and firewalls, then how about making the distribution source (the website) not so obvious and vocal about it's intentions.

Just a suggestion.

Re:Great, I would love to read all about it

[aridhol]

Then how would people find it? It's a catch-22 situation for these guys - either say what it is, and have it filtered, or don't say what it is, and don't let anybody find it.

"C/C++ is no longer a viable development language"

[e40]

OK, fine. But what is? It does no good to make a statement like that unless you back it up with an alternative.

Personally, I dig Common Lisp.

I think not

[The Bungi]

While saying that C/C++ are not "viable" languages any more (and making the classic mistake of bunching them into a single blob), he muses that:

It requires compilation - as your code grows larger, the wait time to see if your code works increases. This delay directly affects how fast your code is developed.
It's really, really, really hard for people to learn it, and this directly impacts the number of developers you will have on an open-source project.
It uses static binding (Isn't that supposed to be a good thing?)
There are no standard libraries for C++, so there's a lot of reinventing the wheel. (Yeah, there's the STL and others, but each one has a huge learning curve associated with it).

So, basically, it has to be compiled (duh). It's hard to learn (no, it's hard to use correctly) and it has no libraries... eh?

I'm sorry, but this guy is not a software developer. The usual comments about "X is the One True Language" notwithstanding, I can't follow that because he thinks it's "too hard" and he thinks it's "not viable" and decides that it simply isn't a good fit for his project, then LanguageX must be dead. Perhaps he'd like to share with us which language his OS is written in. Maybe it's Forth or Scheme. Use the right language/runtime/lib/technology for the job and refrain from saying "X sucks because I don't like it".

Other than the dubious "this is how you do open source" slant I can't see how this article is even worthy of news.

Re:I think not

[The Bungi]

The problem is obviously between the chair and the keyboard, not in the compiler he chooses.

Oh, and har, har.

Re:I think not

[GlassHeart]

A few other points in defense of older languages:

• An older language has had more lines of code written for it, so its weaknesses are better known, and more likely published.
• An older language tends to have more developers, which means any random volunteer is more likely to know it already.
• An older language has already fought "battles" for survival, and has been squeezed out of applications for which it is ill-suited, and continues to exist for a good Darwinian reason.
• An older language is more likely to be standardized, and more widely ported.

Let's take C, for example. An experienced C programmer will point you towards two or three books like van der Linden's "Expert C Programming: Deep C Secrets", or Koenig's "C Traps and Pitfalls". C is rarely used for GUI application development, but still widely used in embedded systems where space and speed are important. C is an ISO Standard, which is important for portability.

One mistake that many people make is to dismiss older languages when a new one appears with all the old features and then some. The old language does not become any less viable the day a new one comes out than the day before. That is, if a project will take you 6 months in C, it'll still take 6 months even after Java (which might cut it down to 5) comes out. The question is whether the unique costs of using Java instead justifies the 1 month saved, not whether C is still "viable".

Interface is Everything

[Snot Locker]

Often time this principle applies to people on the project, not just the software being developed. I've learned from experience that really sharp people with a broken user interface can destroy a project!! You have to try to minimize interaction points with folks like that and find where they can excel and use their talents without creating problems for the others on the team. A difficult nut to crack at times and a far more critical factor to project success that the programming language, source management tools, etc...

Repeating the same old misinformation

[h_jurvanen]

There are no standard libraries for C++, so there?s a lot of reinventing the wheel. (Yeah, there?s the STL and others, but each one has a huge learning curve associated with it).

This is a huge error that casts doubt on the author's credibility. What is commonly known as the STL is the C++ standard library, and it has been since C++ became an ISO standard in 1998. Doubters may consult books like the clearly-named "The C++ Standard Library" (Josuttis, 1999) to get themselves up-to-date.

Maybe that's just another drawback of C++... a lot of people don't know what the hell they are talking about and thus repeat misinformation?

Re:"C/C++ is no longer a viable development langua

[Arandir]

C and C++ are most certainly viable development languages. Let's see now: Linux, BSD, GNOME, KDE, Apache, Mozilla. Even Perl, Python and Ruby are written in C or C++. But maybe the author is saying those projects aren't viable...

Use the right language for the job. If all you're doing is interfacing to a database, then a scripting language may be the most appropriate. But if you're writing system software, then by all means stick with C and C++ with some shell glue.

Compiled languages are damned convenient to the user. "Here's an executable, just run it", versus, "here's a script, go download compile and install the interpreter first, making sure it's the correct version, set up your environment variables correctly, then run the script."

C++ is actually wonderful

[dmeranda]

Modern C++ is a wonderful language, at least I think so. But it is much different than the "old" C++, almost to the point of being a different language. So if you've had bad experiences with C++ in the past, perhaps you should give it another look. And C++ is not dead, there are a lot of interesting advancements in the language, and more properly how to use it. There are a whole lot of generic programming and template patterns which are comming out which show that C++ has a lot more power than people ever thought.

Of course C++ does tend to have some problems with Open Source projects, which C usually doesn't have. So I certainly don't frown on C development either. And plain old C is usually very easy to integrate into other languages/environments.

1. C++ compilers are just now catching up to the standard. gcc 3.3 for instance is pretty darn good now, but lots of compilers have lots of problems.
2. C++ can be very slow to compile (or more technically to link), especially as you use lots of templates.
3. The binary ABI is not universal. It's hard to write shared libraries in C++, so it's not as useful for components as it is an entire application.
4. Dynamic linking in of C++ is next to impossible (dynamic linking often uses the dlopen() system calls, and it how most run-time plugin architectures, such as in Apache or Python work).
5. Although in theory C++ is very portable, in practice that is still difficult (usually a victim of poor linkers). C++ which works fine under Linux may have serious problems under say AIX or HP-UX, unless you test on those platforms.

However, C++ is certainly still alive and very viable on a whole. And O'Reilly just published the new C++ in a Nutshell book which covers the ISO standard C++ very well. Also you should look at the Boost Project if you're looking for more advanced C++ libraries (many of the Boost developers actively participate in the C++ standardaization effort, and Boost is often thought of as the testbed for possible language additions for the next round of standardization).

But I do agree, that you need to pick the language according to the project. There is no one best language. When I look for other open source projects with the intent of being able to take advantage of the openness (i.e., modify the code), I tend to look for projects written in Python. I particularly avoid Perl, becuase it is much harder for me to understand. I also avoid Java because it's a proprietary language with no open source JDE/JDK and I think the language sucks when compared to ISO C++. But again, those are my preferences.

Re: Documentation!

[alexjc]

So, after two years he still hasn't realized the importance of documentation?

"Document it and they will come."

A good project is nothing without it...

Re:"C/C++ is no longer a viable development langua

[Dominic_Mazzoni]

Compiled languages are damned convenient to the user. "Here's an executable, just run it", versus, "here's a script, go download compile and install the interpreter first, making sure it's the correct version, set up your environment variables correctly, then run the script."

Amen. When I'm trying to solve a problem, I like using high-level languages like Perl or Python. But when I need to write an application that I want ordinary users to be able to download and _just use_, I don't have a choice - I have to write it in C or C++.

Luckily, thanks to faster processors (shorter compile times) and tools like valgrind to detect memory errors, programming in C and C++ isn't nearly as bad as it used to be.

convienient

[pyrrho]

no, I don't think bundling that is particularly convienient... indeed, I think that it's more convienient to have a choice and not have things like that tied tightly to the language.

Right now there are well established libraries in C++ for anything you get from standard libraries that are tightly integrated... just with multiple competing established libraries. A wealth of choices.

There is no standard GUI library that ships with all compilers. That will come, but not before it's time. Java gives such a library... too bad it sucks.

What language are (most) all OSes written in?

[jordan]

As much respect as I have for the project and its developers, his broad, sweeping statement to the effect of C/C++ is no longer a viable language is really telling of his ignorance and lack of perspective. Which is too bad, since there are other points he makes that are useful lessons.

To claim the fundamental implementations of all modern OSes (Windows, almost every single UNIX, Linux, and the plethora of other OSes) to be "no longer viable" is way beyond reproach: it's just plain idiotic, and does significant damage to the credibility of his other points.

Unfortunately, my that-was-ridiculously-stupid meter flew off the scale when I read that point, so I stopped reading right there.

Repeat after me: The Interface is Everything

[2TecTom]

As an interface designer and technical writer, this has always been my personal mantra. It's finally nice to see that at least one engineer finally actually gets it!

You probably won't believe how many MMI designers and technical writers are feeling totally vindicated at this point.

Really, it's not often one sees history in the making. ;~)

Re:Repeat after me: The Interface is Everything

[Yaztromo]

As an interface designer and technical writer, this has always been my personal mantra. It's finally nice to see that at least one engineer finally actually gets it!

He gets it, but he doesn't get it. To specifically quote what he wrote:

This is similar to the #1 project management lesson. The program should be fun to work with. There should be buttons and things that blink. The interface should be the first thing you do.

I agree that the UI is extremely important. It bugs me how many Open Source projects don't bother to employ the skills of someone well versed in HCI (although as an Open Source project administrator myself, I can say that finding such people can be difficult. I've been lucky to have attracted the attention of some people who have some interface design experience).

However, it shouldn't be the first thing you do. That's the top-down design paradigm, and it doesn't make for particularily maintainable code. You wind up with a pile of code that is so tightly coupled to the GUI that it's difficult to use the useful portions of the code for any other project.

It's much better to break the design into two parts -- the core "engine" code that actually does the work irrespective of interface, and the UI. Neither should be tightly coupled to the other, but both should be considered equally important.

The problem with many Open Source (and closed source for that matter...) projects is that the GUI becomes tightly coupled to the "engine" code. I've seen this happen in all sorts of poorly planned projects, wether the source is open or closed, but this sort of design seems to harm Open Source code moreso, because the useful bits become difficult to use in other Open Source projects.

Whatever you do, don't code the UI first. This makes it too easy to fall into traps that cause your engine code to become tightly coupled with the UI code, and difficult (or impossible) to use without your UI components. But when you do get to the UI, make sure you have your target audience in mind, take the time to design it for maximum usuability, and make sure it has all the bells and whistles your audience expects. It's how the user interfaces with your program, so it's going to be the biggest element in how they judge the usuability and usefulness of your project.

Yaz.

95-5 Rule

[jmacgill]

Usually it's the 80-20 rule, but in open source projects it's more like the 95-5 rule. Open source projects are usually run by one or two people doing most of the work. If you decide to lead an open source project, you must be willing and ready to accept this.

Actualy, if you are about to set out on a new project, its probably best to tell yourself that you are NOT willing and ready to accept this.

6 years ago I started a project called GeoTools and it was, for the main part, excactly that - two people doing most of the work. This was fine for a few years but over time the user/developer ratio got out of hand.

Eventualy it became all but impossible for the two lead developers to support 300+ users and although other developers wanted to contribute it became dificult to 'train' new developers as the knowledge of how things worked existed mainly in the heads of only two individuals who had done 95% of the work.

Two years ago we took the descision to re-design the toolkit from the ground up with as much input from as many people as possible. Since that time we have strived to make sure that as many people as possible have an input into the design process and we keep that process as open as possible by pubishing the IRC sessions in which discussions take place.

The project now has 9 very active developers who are members of a Project Management Committe and a number of other active contributers as well. The end result is that quiries to mailing lists get responded to far more quickly.

Getting other people to work on your project is often - TO START WITH - more effort than just doing the work yourself, but the pay off is HUGE, as you then have someone else who can explain things to others.

If you ever have a contributor who gets stuck or confused and you find yourself thinking 'oh, it will be quicker/easier for me to do this part myself' STOP. Spend the time, help them work out how to do the modification even if it takes a few hours when you could have done it yourself in minutes becuase after you have invested the time in them, they will be able to add things in minutes too, and they can teach others as well.

If you work on a tight, well defined, non-evolving project then most of my ramblings are probably irelelevent if not they they may be of use. The only danger is in investing time in helping developers who then wander off - it happens, but I tend to find that the more you invest in them, the less likely they are to loose intrest.

Re:95-5 Rule

[JamieF]

Or, you could write some documentation. It's not that hard but for some reason developers avoid it like the plague.

I keep reading these open source project managers bitching about how hard it is to answer all the emails from users and potential developers. Then you look at their project and see this:
FAQ: (under construction)
Documentation: (under construction)
and if you're really lucky they bothered to archive the developer mailing list, which is just about the least efficient way to document the project. (Yes, let's make every developer read all of our conversations ever, including all of the arguments we had and all of the things we decided to do differently later.)

It's not that hard; I've done it. You take a day or two and STOP CODING (oh no!) and write some actual words, document APIs, and lo and behold, you realize that you're also white-box testing your code because you were forced to explain what some code does ("hmm, that actually doesn't make any sense" / "oh crap that won't work in this case").

All of a sudden you can just point people at the docs and add stuff when new questions arise. Sometimes the answers are best expressed as a FAQ, sometimes as part of API docs ("oops, I didn't explain what these constants actually meant"), sometimes as end-user documetnation. But IT'S WORTH IT.

As for the common self-delusion, "we'll write the documentation after we're done with the code", all I can say is, read any process book there is. You're doing it backwards if you start with implementation and then back into writing down what the system does and how it's designed. Those things come first, THEN you write the code that does it. Otherwise you redesign the product dozens of times as you go, each time slapping yourself on the head and saying "oh crap I forgot it has to do this too" or "argh, I need this value from here but the API doesn't allow that". If you write it all down (which first requires that you *think* about it in detail) then you won't forget your good ideas, you can jump all over the place, you can write test harnesses that make sure the code does what the docs say, and you can point newbies at the docs and say "OK cool you can get started on the email notification part, we haven't touched that yet."

irony

[pyrrho]

So much of what is said about C++ here on slashdot looks to me to be fear, uncertainty and doubt I find it ironic.

The complaints about C++ are the kind of complaints users make about linux when comparing it to Windows. "It's much easier to click on a big E than to learn another icon..."

And clearly he's talking about GUI libraries... instead of multiple good C++ and C based multi-platform GUI libraries, he prefers another model, more like javas, where you get one crappy library and save yourself a lot of thinking!

If anyone cares about the quality of the software (more than their experience developing), they'll learn to use a good compiled language, period. If you can cut corners on quality, if you buy the idea that CPU cycles are here to waste, then use something lesser (I do in that case). However, I think we are wasting cycles, and there are some amazing things our software is not doing that it could. Moore's Software Law is that every 18 months software gets twice as inefficient doing the same thing.

Not Project Management

[drmofe]

None of the lessons learned and reported here are directly related to Project Management per se. They are all by and large implementation issues.

There is also nothing new here. This does not advance the state of the art. History does not advance by people relearning the same lessons again and again. Just because they have been reported here does not make this article special in any way. This article could have been written in any of the decades of 70s, 80s, 90s (substituting en vogue languages for C++/Java) and still make sense.

In order to truly advance the state of the art, we have to think in far more advanced ways about project management and software development. True Software Practice and Experience requires much more planning and critical thinking than evident here.

If Open Source is to provide a useful and stable platform on which to build, then we certainly need a better vision of how to build software. Otherwise, we will be doomed to repeat history by implementing old things in different ways and not really gaining any control over complexity.

In summary, we still have a software crisis; Open Source will not change that; and summaries of software development experience that just say "I made the same mistakes as other people did" are not very helpful.

Re: Documentation!

[JamesOfTheDesert]

So, after two years he still hasn't realized the importance of documentation?

"Document it and they will come."

A good project is nothing without it...

Good point; somebody should document this.

Binary vs text protocols

[kingdon]

Well, seems to me that SMTP, NNTP, HTTP, etc are easier to develop for because they are textual.

HTTP may be just about the ideal balance: use text for things which tend to be small, but capable of sending a large payload (e.g. a PNG image which the protocol just needs to wrap, not do anything with) in binary.

The thing about protocol layering and/or marshalling is: do you have good debugging tools (at a minimum, log what is going across the wire and be able to interactively enter data and see results)? For binary protocols like TCP and IP, largely yes ("telnet" client in the case of TCP; tcpdump for both TCP and IP). And maybe some RPC packages have these kinds of thing (I haven't used them enough to know). But an app which rolls their own binary format generally doesn't.

Re:"C/C++ is no longer a viable development langua

[emag]

I'll go off on a different track than the other responses I've seen...

Wouldn't python suffer from roughly the same problem as Java with the JRE? I mean, unless it's compiled (and I'll admit right now, I don't follow python closely enough to know if it has a "compiler" yet), users are going to need to find and download a Python runtime environment of some sort. The latest I've found at python's web site is around 9M. While that's still about 1/3 what a JRE is, it's still either going to be a separate install, or a lot of additional weight to package up with whatever you're distributing.

It would seem that anything interpretted is going to suffer the "separate download" problem... Ease of learning would also be debatable for probably every language out there. I know folks who are wizards with C/C++/Java/perl/$language1 who could couldn't get a working "Hello, world." out of $language2, while at the same time I know others with the skills in $language2 who couldn't do anything in $language1.

It's a shame the linked article's author didn't address what WOULD be an ideal language to use, and enumerate why, but it's probably because any language he did pick would end up sharing criteria with his "these make C/C++/Java bad" statements. :-/

Re:Compilation time bounds productivity?

[be-fan]

This C++ is a weakly typed language is bullshit. I get far more type errors in Python (which I love, btw) than in C++. Dynamic typing is both a blessing and a curse, as is static typing. Now the Java-heads like to say that C++ is a weakly typed language because it lets your do unsafe casts. Well of course it lets you do unsafe casts! How else would you write kernels in C++ if it didn't? The point is that you don't have to use unsafe casts in regular programming.

PS> I have a feeling most of the "wealky typed" arguements come from people who only see the "C" side of C++. For example, I was doing a messaging system the other day. What's the first instinct of a C programmer turned C++ programmer in a situation like that? Have a message contain a void pointer to an untyped buffer, of course. My solution? Use boost::any to encapsulate the message data and use boost::any_cast to do typesafe conversions when the message data needs to be unpacked. Throw in placement new and a sane copy constructor, and you have a perfectly type-safe way to send objects from point A to point B. It's this kind of stuff that you don't see when reading all the legacy C++ code out there, and its largely because of that that C++ gets such a bad rep.

Re:Duh

[Anonymous Coward]

With all due respect, you are clueless.

Once you have a network that you want to maintain, you can't assume that everyone will run the same version of a node. This is a big problem with binary encoding which tends to break compatibility when changed. It's possible to design offset independent binary protocol, but then again, XML is also 0's and 1's. If you gzip your XML, you remove redundancy (thus eliminating most of the network bloat argument) while still maintaining excellent compatibility (new elements are ignored instead of breaking things).

So, next time you get on your high horse, don't... If you don't respect Mr. Baranowski, that's your personal problem. You should ask yourself why he said that instead of assuming that you know the answer.

Re:"C/C++ is no longer a viable development langua

[Newander]

I don't think that's what he meant by a "standard library".

Actually, he mentions the STL by name.

Baranowski's Comments on Languages

[WillyLane]

In regards to his statement "C/C++ is no longer a viable development language" and his further explanation here ...

-I dont agree with his language critique. I think the advantages of static typing far outweigh any time it takes to (as he describes) figure what type to declare a variable. Have you ever tried to debug a large project written in a dynamically typed language? It can be a fucking bitch.

-He also says it is really, really hard to learn C, C++, and Java. He says this cuts down on your developer base. Any developer who doesn't have the capacity to pick up those three languages, I just do not want on my project.

Re:Duh

[crucini]

It looks to me like you just haven't yet learned the lessons he learned. Which is fine - most of us can only learn through pain. HTTP, SMTP and friends were widely accepted because they're text-based. This has great implications. You can simulate a client with telnet. You can paste part of a session into email or chat when discussing a bug. When interpreting the complexities of HTTP 1.1 you don't have to perform a second level of mental indirection to translate a header to a numeric constant you're looking at in a hexdump.

I think the creation of a new binary application-level protocol needs to be specifically justified. Either:

• One end of the link is computationally weak, like a tiny microcontroller.
  
• The ratio of messages to available bandwidth is very high. But remember, binary protocols aren't guarranteed to save bandwidth - the simpler they are, the more wasteful of bandwidth. If you take a huge C struct and put it on the wire, and most of the fields are irrelevant in a particular message, you might have saved bandwidth by using a text key=value scheme.
  

On the whole, binary protocols are like any other optimization, and should be applied judiciously when a performance gain can be expected. If you properly abstract the generator/parser for textual messages, it shouldn't be too painful to make the optimization.

Re:Article text

[Jellybob]

That doesn't make it the ideal language to use for those programs.

The IM programs are certainly good candidates for higher level languages, since most of what they do is text parsing - and I know that at least myself would do some work on GAIM if I could.

Static compilation time

[Per Abrahamsen]

Static compilation time does not grow linearly with project size, unless you either aren't using a dependency tool (like make) or have organized your code to that every module depends on every other module.

With regard to the first: consider learning thee build tools part of the cost of learning a static compiled language. It is not optional.

With regard to the second: If you have so many cross-dependicies, you will run into problems in any language as the project grow. A bug solved in one module may cause new bugs to appear in any other module.

In general, Based on many years experience with C++ and Lisp, I'd recommend compiled and statically typed languages for large projects, because of the discpline they encourage. Interpreted and run-time typed languages are usually superior for small projects.