Slashdot Mirror
eXtreme Programming (XP) in OSS projects?
ivi asks: "I first bumped into the XP approach to system development, some years ago, on Dr Dobbs' now-defunct Seminar-On-Demand site TechNetCast. XP has some short, simple rules for growing software from Users' Stories, eg, with programmers working in pairs,
showing prototypes to Users "early & often" et al. Download this XP site (under 400KB, zipped) for more.
So, who's used or using XP, does it work for OSS projects & what have your experiences been with it, so far?"
So, who's used or using XP, does it work for OSS projects & what have your experiences been with it, so far?
Heh, I'll be the first to say that Windows XP doesn't play well with OSS projects.
and yes, that IS an attempt to be funny, not stupid.
We tried XP approaches in my college programming classes. More often than not, it caused my partners and I to claw each others' eyes out, and generally despise your partner for the remainder of the project.
What worked MUCH better was designing the framework from the beginning, outlining how the different functions/processes/data structures etc. would have to communicate with each other, then splitting the programming tasks up.
After every one made progress, we'd recombine, work out bugs, etc.
Much more organic and realistic than the XP approach, which always seemed very articifical and limited.
I remember reading an article on O'Reilly about XP and Open Source .. go dig it up..
:-).
I don't have any experience with XP in writing open source projects, but I sure see a lot of projects that can benefit from it!!
I would love to see more software with comprehensive tests. It's very satisfying when you hack a minor change into a program and all the tests still pass. Then you can send the patch off to the author knowing it won't break anything (well, at least it won't break any of the tests
In this vein, I believe test-driven development (i.e., write tests, write any code that passes the test, then refactor to clean up the mess) is the most amazing gift from whoever the programming gods are. I'm glad XP has (re-)discovered it.
It forces much simpler, cleaner interfaces, and complete unit tests. There have been several times when I've wished that a particular open source program was split into many small objects that I could mix and match (i.e., I love this package BUT I need to replace the code that saves files to disk with code that saves files to a database..etc).
With TDD your design splits into many small simple classes. I think this would be great for OSS projects. A good example of a well-factored library (off the top of my head) is Mime::Tools for Perl.
Too many OSS developers think "I used Python, I must be doing object-oriented programming" or something along those lines.
Some other aspects of XP such as continuous integration seem useful.
Another thing about XP techniques (especially test-driven development again), is you can code in small bite-size chunks and not leave the code in a broken state. I think this is great for projects you might have "on the side". You can work an hour or two and your code will always be working and passing tests, none of these "egg-juggling" 10-hour coding marathons. You know what I mean, when you change something and it ripples to 20 files and you keep all the things you need to do in your head and you go half-insane running from file to file.
Pair programming, user stories, "Customers", etc., seems less useful.
But yeah test-driven development is worth trying, at least.
My initial impression of XP (I haven't tried it but had it explained to me) is that it takes a lot of the fun and programmer-as-artist aspect out of software development. Also it is good for making a larger number of so-so programmers as productive as a few really good programmers.
However, most OSS projects seem to be labors of love by groups of individuals working together in their spare time. I would bet most of the largest contributors to OSS projects are very good programmers who don't need to be "managed" to the extent that XP does. I don't see how many of the XP ideas could be applied when none of the developers are in the same room on a regular basis (or even available to meet on-line at one time), their time commitments are unpredictable, and in many cases have big egos and work on what they feel like, not what the customers ask for... After all, programmers have day jobs that are like that!
There is really nothing "extreme" or new about XP--it's simply that someone gave a name to decades old practices. Many OSS projects are already using XP, whether they call it that or not.
Are those practices good? That depends on the specific people, user community, and the project. XP is no more a silver bullet than anything else.
Something like... M'aud'ib! -ib! -IB! -IB! -*-KABOOM-*- ! (-:
Got time? Spend some of it coding or testing
Having participated in a number of OSS projects, and even led/maintained a few, I can't see how Extreme Programming can help - the model is clearly better suited for in-house programming than distributed programming.
Many of the same techniques are used in OSS projects though. "Release early, release often" is almost biblical to people from what can soon be called a Linux-background, and another rule is "No functionality is added early.", which fits my personal design philosophy pretty well, and a lot of OSS projects: I can't remember the last time I added functionality to a program.
What happens is a company has a bad experience with programming -- say they fire some old guy to hire a cheaper new guy and discover the new guy can't make head or tails of what is going on. Of course the new guy says this is due to lack of documentation. Or a department realizes that some software doesn't do what they needed, because they never told anyone what they needed and no one asked. Or the realization that a key piece of software that is keeping the whole company running is a mystery to everyone who works there and the source code is long lost.
Well, managers can't really get paid the big bucks for saying "Hey guys, next time let's try to be smart" which is the real solution. So instead they invent a plan, procedure, or rules to prevent it from happening again. Everything you do must be documented. Tens of thousands of dollars on verson trackers. A flow chart describing a series of meetings to produce a requirements document and another flow chart for modifying the requirements document.
Well, pretty soon people realize you have a bunch of people filling out TPS reports and one productive guy who is evading all the documentation and requirements procedures and just getting work done, maybe protected and covered for by a few co-workers and a boss. And someone thinks, jeez, if only everyone was like him.
But you can't just have the managers go out and say, "Alright, bad idea, toss it all out the window, let's just focus on common sense, communication, and being smart." That kind of implies you could hire fewer managers and just spend money on trying to get smart people.
So you have to package the "common sense" proposal in a bunch of buzzwords that make it sound like it's a new 6 sigma or total quality or some other plan for extracting the most out of dumb people by telling them to do smart things. This way you can slip it in past the Organization Men.
One of the most pathetic things you can ever see is a stagnant ossified department, which long ago chased away anyone with any ambition, attempting to implement pair programming and other feature of common sense. If you program in pairs, it's harder to say that person X did this and person Y did that, which managers see as a threat. Talking with the final consumer also undermines certain people's positions, so they usually interpret that "show prototypes early and often" as instructions to go through a formal process of beta release, evaluation, discussion meeting, plan for addressing issues, FASTER. Not fewer levels of BS and more communication, just do it harder and more often.
Open Source projects are usually XP in an extreme way, except for pair programming. Almost all code is written by lone wolfs and casually reviewed by others if at all. Other than that, the quick-release and review, constant prototyping, etc all just naturally happens. Even more so since the programmer is usually the user he is trying to satisfy, which is the tightest loop there is.
Oh, God, NO! Not in OSS, you can't taint it like that!
Oh, wait, [i]that[/i] XP! Oh, okay, I feel better now. Phew...you had me scared for a minute there.
Help protect civil rights from abuse by the TSA - visit TSA News Blog.
http://www.tsanewsblog.com
Should've been "oh, wait, that XP." Too much time on the phpBB boards. *sigh* There goes my "funny" mod...
Help protect civil rights from abuse by the TSA - visit TSA News Blog.
http://www.tsanewsblog.com
From the Download:It is a collection of rules and practices each of which supports several others, and are supported by several others in turn.
A truly organic approach is the best way to program in a team. The people involved should and usually do develop their own system during a project. Having rules of practices that support and are supported is just way to complicated. No 2 people and no 2 teams will ever want to go at a project the same way, it makes teamwork tough enough. You should find a middleground, not a set of "rules" on how you do things that neither person will like.
My experience, after a good few years at it, is the same as what Lambent said - work on the design together, do tasks separately, and recombine. Pair programming works on certain problems, where one of the two has asked for help, but otherwise it's counterproductive and lets both parties get very annoyed with each other.
Some of the other rules in the article are nice in theory, but not really practical in real life. For example: customers aren't always available, and often don't want to be either. Involving the customer too much usually ends up in them wanting something other than what they originally agreed to, but within the same time frame and for the same price (and then they get upset when they can't have it).
Moving people around can backfire too - you can end up with people who have a little knowledge in a lot of subjects. It's not always realistic to expect that everyone knows enough to program every part of the system. Some people really don't have that capacity; if you have a team of top notch people who can, consider yourself lucky.
Some of the other points listed aren't that extreme at all, and should be part of the normal development cycle. Eg: must pass complete testing before release! They also list the point that the rules themselves need to be flexible, and can be changed. I like this; it's necessary when working with people as they don't produce an identical result each time they get the same input!
-- Steve
If your open source project is being developed by a team of people all sitting in the same building, directed by a customer who's paying for the software and who accepts XP, then yes, XP should work just fine.
If your open source project is being developed by volunteers all over the world in different time zones, then forget it. Getting XP to work in that situation is probably possible, but difficult and probably expensive.
But if you, like so many others, are using XP as an excuse for shoddy development practices (the "look, we don't need to do requirements analysis or design or documentation because we're using XP" syndrome), then by all means go ahead. It'll work just fine as an excuse.
So let's get to the details. I understand XP pretty darn well. We deployed it successfully at my shop, making all the classic mistakes (and some of our own) on the way. We finally got everything right, and once we did, it worked really well.
I'm going to assume that "open source project" means a project with volunteers all over the world (or all over the country), and no paying customer.
Problem 1: The customer role
The "customer" in XP is a person who represents the users and can make decisions regarding what to implement and when. It's not necessarily a paying customer or an outside person.
The customer is a key person in XP. If you don't have a customer, you can't do XP and if the developers don't do what the customer requires, you can't do XP.
Developers don't make good customers. They're too involved in the technical side of the project and rarely prioritize well. You really need an outsider who is focused only on the end result.
Developers have to listen to the customer. The customer decides what gets done. The developers have no choice in the matter. They can tell the customer how much it will cost to get something done, but in the end they have to do what the customer wants.
Do you have a customer on your project? Will your volunteer developers do what the customer wants? If either answer is no, forget about XP.
Problem 2: Pair programming
Pair programming gets left out of a lot of XP projects. That's unfortunate, because pair programming is a key ingredient of XP. Without it, the process hobbles along.
There are lots of reasons why people give up on pair programming. The poster mentions one: not leaving that big programmer ego at home (it's just programming, not personal). There are others.
What you need to understand is that pair programming is how knowledge and skills are communicated in an XP project. Pair programming compensates for lack of formal design and documentation. Pair programming cuts down on both trivial and serious bugs (the brain not writing the code is usually thinking about the big picture). We found that code written by pairs was consistently better by all subjective and objective criteria we used.
Can you do pair programming in your open source project? If your developers are all in one place and work at the same time, then it's easy. But if they're not, you'll find it very difficult. Instant messaging or even speakerphones don't really help much. And remember, pairs are supposed to be unstable. If you always work with the same person, you'll lose a lot of the benefits of pair programming.
Problem 3: Communication
XP developers tend to be pretty chatty, in my experience. At our shop, verbal communication was a very important component, and to ensure that it was easy, we all worked in the same room (no partitions at all).
How easy is it for your developers to communicate? Instant messaging helps, but isn't all that good since you can't count on instant feedback. A phone works pretty well, but it should be a speakerphone (a headset is OK, but less effective). Are your developers working at the same time? If they're not, how are they going to talk to each other while they work?
In conclusion . . .
T
I think one of the main points about XP is that everything is customer-driven. In open source projects, there are no customers, which in my opinion explains the bad usability of lots of free software.
They have a page Combining Open Source And XP, which I reproduce here to avoid hammering their server. Posted anonymously because this is a total karma whoring.... not that it'd matter, I've been capped for years, but hey... style matters.
This is mostly musing, rather then a "how to", and assumes a lot of context you may not know unless you know XP, but following the links (like UnitTest, which I particularly recommend) can fill you in.
Finally, before I leave you to the page, I'm doing a project right now that I hope will be open source, and while it's currently just one person (so pair programming is right out), a lot of the other ideas work incredibly well; with Unit Test and merciless refactoring I'm staying on top of a project that's already five or six times larger then any I've ever done on my own, it's in good shape and I understand it, and I can easily triple or quadruple the size before panicking, whereas the "competition" for my project... such as it is... blew up long before even getting as far as I have (mostly becoming Big Balls of Mud, and there was one that used a blob). Even if you can't do "XP", Unit testing (and some degree of Test-first development) and Merciless Refactoring alone can be a huge help on open source projects; the better your code quality the more likely it is you might actually get external developers.
--------------------
The idea is to develop a site similar to http://www.SourceForge.org or http://www.CoSource.com where, however, where the XP practices provide controlling of the OpenSource development approach. The problem for a potential OpenSource customer is that if they want to pay money to have some product developed by OpenSource developers then they also want a guarantee that the product will be completed on time and within budget and to the quality they require. However, money should not become the sole motivating factor, this risks turning an OpenSource project into a ClosedSource project.
CoSource does this but doesn't have any means for the customer to check whether progress is being made (although they define a third party to judge when the project is completed). And this is where XP comes in: through UserStories, UnitTest s and ContinuousIntegration the customer can always check progress. Plus after each 2-4 week iteration they can terminate the project and only pay for the work done to that point.
XP would not be enforced, can't anyway, but the intention is to offer tools which allow the customer and developers to a) communicate and share ideas, b) allow the customer to see whether progress is being made and c) both sides to check the quality. Tools should not be forced upon projects, however, customers would have the right to define which tools should be used for a project (after all they sponsor the projects). However, to a certain extent, a project should be given time to find it's own tools of choice.
Benefits for customers:
The first thing I did were to picture the whole system as a group of functional boxes with interfaces. The next were to define the interface mechanism. Then I grabbed my nearest bosses and made them customers showing them a new release every two weeks. I distributed the tasks of implementing the functional boxes among the programmers in the group who I placed in the biggest room awailable facing each other. In the group we used nightly builds to always have the CVS build clean. My bosses were forced by us to prioritize among tasks and give us input for the next release. One day over a year later we could experience a decreased pressure and a product that we felt confident about. At that point we started to refactor all bits and pieces that were just implemented the easiest way as well as the build system. Now another two years down the line we have catched up and we have got rooms, some still with two programmers and we are increasingly using a traditional way of programming, with a spec phase, an implementation phase and a QA phase etc. We only kept some featured of XP that we like, nightly builds and refactoring.
I think that XP has its cons and pros depending on the situation. We could never have delivered all these demos, prototypes and releases during the course of these three years without XP, simply because we had to attract customers and venture capital all along. We didn't use XP to the full extent either, no automatic testing and no true pair programming but that was only because we didn't have time to change ourselfs. If you sit on a fully financed military project for instance I doubt that XP will work for you. If you lack time or funding or simply needs to go extremelly fast to something that works XP might be the way to go, just ensure that everybody in the project agrees.
Good Luck!
which in my mind isnt' the biggest aspect of XP. Hanging on my office wall is a poster of the main practices of XP, so I'm using that for reference, and it seems to me that a lot of the practices fit.
Ways OSS and XP mesh very well: Simple Design, Refactoring, Continuous Integration, Collective Ownership, Small Releases, a Sustainable Pace, and Coding Standards. These are things that most successful OSS projects do, and they're pillars of XP.
However, OSS misses the others, like Test Driven Development, Pair Programming, and Customer tests. In my mind, the most important aspect of XP is Test Driven Development (not Pair Programming), which I have yet to see a single OSS project do.
XP isn't an all or nothing deal, it's a set of practices. Use the practices, and you'll write good software. I consider OSS development extreme because it follows several of the guidelines, and I think OSS development teams would gain a lot by implementing the rest of the practices. I also think this is a reason why OSS developers write such good software (now imagine what it would be like if we did Test Driven development...)
-- Breaking Windows: Not just for kids anymore KDE
I've a few reasoned objections to XP as well, but my gut reaction to it, and I'm not the only one who has it, is "oh please" along with eyerolling.
I think the real reason is that XP has -- no, XP cultivates -- an image of "Programming for the MTV Generation". Everything about it screamed, well, "EXTREME, DUDE!", from the salesmanship to the cutesy names, all the way down to the methodology itself. Short attention spans seem to dominate the whole XP process. All other methodologies are derided as stodgy and conservative, good only for the Folks Over 30.
Engineering methodologies are a discipline, and yes, a heavy one can crush a light project that just needs to get out yesterday and basically do the right thing in whatever way it takes. But the point of development methodologies isn't to be fun and cool or even interesting to most people. It's to make your product and your process itself stable, reliable, and repeatable. If that's too much to take, then feel free to hack on your own, maybe you're even good enough that that's all it takes. No one wants to bank the entire company on that assumption though. And no one is saying developers should be forced into the bureacratic tedium that a methodology might impose; that's what development managers are for.
Personally I find things much easier to handle when they're designed to spec rather than designed so all the test bars turn green no matter how much I had to hack things up to do so. The documentation sort of writes itself in such cases too, it's mostly a matter of chopping out the detailed bits of the spec that are internal function.
Twisted is fairly big on XPish development techniques. They're big proponents of test-first design and, when possible, like to do pair programming.
The problem is that, generally, pair programming is not really possible for most OSS projects, where developers are spread across the globe & communicating by the net. They've been kicking aroudn the idea of a networked multi-user text editor tho..
my sig's at the bottom of the page.
No, sorry haven't tested all the other aspects of XP yet (see xplanner.org as a nice example of how support might support you in xp projects)... BUT I could tell you dozens of stories about programming in pairs...
first option: your old school buddy, roughly same programming knowledge and project experience like you joins you in an XP project. You hang around for hours, drink a beer with him, tell dozens of old stories... and then you get productive. Damn productive. XP _might_ work great in these conditions. I absolutely enjoyed some examples of programming in pairs when your programming partner is of your kind... BUT
second option: this guy sucks, he's dumb like a piece of shit, has no idea what i'm talking about. You end up doing the whole code and explain to him what you're currently doing. At least that's what you're doing the first two days. You're de-motivated and won't get your work done. That sucks.
Programming in pairs might work, but the bigger the company/project the bigger the chances are it won't.
my 2 cents.
Why the AC post? You have absolutely nailed the corporate experience on the head. It's worked exactly as you describe at every place I've ever worked for, large or small. Common sense is the last thing applied to anything because it's too simple.
A hint to all those in companies now - if you see a company ossifying as described above, get out!! You owe it to your career and your spirit. I have seen too many fine developers wait years too long to get out of a bad situation, a terrible waste of an excellent mind.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
In reality, for most OS projects the developer(s) *is* the customer!! In the ultimate pairing if you will, the developer knows innately what the customer desires because as Pogo said, "We have met the enemy, and they is us.".
Now what that means is that pretty much by definition an OS project usually has the features that make the "customer" quite happy. The variety of UI quality has nothing to do with there being no customer, it has to do with the "customer" feeling too comfortable with how the features are accessed and not having the usual need for a step back by the developer to think about how a customer will really understand a feature, because the developer is intimately aware of how a feature exists and the means of access. They usually just stop developing once they reach a point of comfort.
Good OS projects come about when other "customers" step in from the outside and force that nessicary "out of mind" expereince needed to bring forth a better UI.
"There is more worth loving than we have strength to love." - Brian Jay Stanley