Tips on Managing Concurrent Development?

An Anonymous Coward queries: "I work on a fairly large-sized project with at least a dozen developers. Advanced tools like CVS and ClearCase allow concurrent development, and provide merging tools to merge in different changes to the same file. This can be a significant productivity gain, particularly with files that are unavoidably common to several developers (C header files, most notoriously). During crunch times, such as before delivery deadlines, we often find that we are checking in changes to the same file several times a day, often hourly. The problem does not seem to be with conflicting changes to the same lines of code, but rather with developers knowing the sequence in which concurrent changes will be checked in. It is not possible to always be aware of who is checking in what and when, so programmers submitting patches to the baseline often have to redo those patches multiple times in a day in order to have them applied. Have other programming projects developed solutions for dealing with this problem?" The submitter proposes another solution, below, how well would it work?

"Take, for example, the extreme case of something like Linux (not only concurrent development, but geographically distributed development), how is this managed? One solution we were contemplating was to try to do an 'air traffic control' type of sequencing and conflict resolution. As early as possible in the development stage, we try to identify what will be finished when, and assign a one-up sequence number to each patch. Developers then know that they will be patching against the baseline that was patched by the patch with the previous sequence number. It is hoped that this prevents a lot of rework of patches. A potential problem with this approach is the need for a responsive central authority to assign sequence numbers. Also, such sequence numbers may have to be rearranged in the face of last minute advances and setbacks in developer progress. Despite careful scheduling and detailed design, it may be impossible to know the exact check-in sequence of patches more than a week or two in advance.

Will such an idea be successful, or is it fatally flawed? Are there better solutions to the problem with less effort? Are we treating symptoms and not the disease (i.e., should we be planning better so that we know patch sequences and dependencies early on)? Management likes to keep staff productively occupied and working up until deadlines, so this usually means a lot of checkins within a short period of time, rather than staged checkins. Can checkins be spread out over time while keeping developers productively occupied?"

Comments

12 people a large project ???

[bstrahm]

I don't think so, but then to many people this might be large...

Of course some of these problems sound like lack of planning early in the game...

For example changing headers that two developers need... The only headers that two groups need should be interface headers, these should be set early and not need a lot of change, with any change taking both developers changing the code internally...

Another note, I get really worried when people say that process problems only show up at the end crunch time. If it is crunchtime it is time to use all of your processes, because the processes should be designed to produce the best bug free code the quickest... otherwise it shouldn't be in the process...

That is just my 2c worth however

Better Solution: Use CVS or ClearCase Properly!

[noahbagels]

I currently work in an organization with 75 Engineers (50 USA, 20 India, 5 Asia) - and we use CVS. It's free, easy to use, and has a simple feature set so that more than one person has enough knowledge to do things like branching and merging branches.

We nearly never have merge problems. It is standard procedure that people keep their tree up-to-date with the cvs tree, and thus conflicts rarely arrise. Even at crunch-time, I probably have one merge conflict every 2 weeks, and with CVS - you are notified of the conflict and it is wrapped with CVS comments.

To put this in perspective - while at Oracle with 1000s of engineers working on the same tree, we used ClearCase and it was awesome. The difference here is that there was much steeper a learning curve, and no normal engineers could actually do complex tasks - i.e. create branches etc. We had a complete groud dedicated to ClearCase.


Conclusion:
Educate your engineers - and politely have the senior engineers tell them when they mess up - enforce a policy that people must update the source every day that they plan on checking-in files.

Also - I don't know what CVS versio you are using, but the latest free WinCVS client will not allow you to check in a file with a conflict! It will force you to update/merge/resolve the conflict before updating the tree. I highly recommend CVS and WinCVS due to the ease of use and cost.

Re:Better Solution: Use CVS or ClearCase Properly!

[PD]

We nearly never have merge problems. It is standard procedure that people keep their tree up-to-date with the cvs tree, and thus conflicts rarely arrise

I'll second this: At many companies I have heard over and over that CVS sucks because of the conflicts. When I inquire further, I find out that people never or rarely use the cvs update command to synchronize. cvs update should be executed almost like a nervous habit, or at least a couple times a day. More if code is being checked in frequently, or if you are working on the same code as someone else. Use the watch commands in cvs to be notified when others edit your files.

It's a shame, really, because programmers seem to be afraid of cvs, preferring a more primitive tool such as rcs or pvcs. cvs lets programmers work in the style that is most natural in an open-source arrangement, and in my opinion can be a far more productive environment than a systems that locks the rest of the team out of critical files.

Modular Isolation

[pyrrho]

The only thing I know that really works (i.e. "is simple") is to lessen the conflicts through design... that is, two people shouldn't HAVE to edit the same modules. Or at least not the same lines of the same modules (those are the only merges that are really painful). Similarly, if you have well understood specification for modules then there should not be a problem when the lines edited don't overlap, because the functions and modules will continue to behave to the spec, which is all the other code can expect.

I know this isn't really easy to do (can't be done retroactively), and doesn't really fit all cases (such as near a release when there is a lot of chaos), but it's the only elegant solution I know of, all the rest are more brute force.

Extreme Programming

[Frank Sullivan]

Check out the development techniques of Extreme Programming (just search Google, silly, and buy a book or three). They have a real solid handle on concurrent rapid development.

The real heart of Extreme Programming is "test-first" programming. The entire development process revolves around unit and integration tests, for extremely fine-grained control over code quality. Any changes that might impact other code should break a test. You fix the stuff that breaks, check in your changes, and move on.

Multiple programmers touching the same C files many times a day sounds like you have either design issues, structural issues, or both. That just should not happen, crunch time or not. Heck, crunch shouldn't happen if you're managing your development correctly.

If you're using cvs, conflicts with source checkins should be very easy to resolve. Even if two programmers touch the same file, they shouldn't be in the same function. If they are, you're back to management and architecture problems, and you need to fix those NOW before work grinds to a complete halt.

Why are patches applied multiply?

[Anonymous Coward]

Why in the world should developers be applying the same patch multiple times? You've just said that the problem is not with developers needing to touch the same lines of code -- so, once a patch is in, shouldn't the next person be merging their code with what's already there?
If your problem is with people overwriting the changes that previous submitters made, then you've got a very different kind of mgt problem -- one that can be solved by getting people to use the tools they already have. CVS, for example, lets you merge the current branch head with your working copy, incorporating any changes that may have been made since you checked things out.
Submitters should always diff their current code with the head before commiting a check-in, to see if they are breaking previous changes. This kind of practice is more important when schedules are tight, and you shouldn't let people off the hook because they were in a rush or some other lame excuse.

--tsw

Product Management and Strong Architecture

[dant]

Yikes! You have the process problems of a group five times your size.

If you have several people changing the same file in a given day, then one of two things (probably both) is wrong:

• Coordination between features/projects. Somebody should be keeping an eye on the list of fixes/enhancements that are coming down the line and making sure you don't get too many in the same neck of the woods. This person doesn't necessarily have to be a developer (but should be able to speak developerese), and their whole job is to tell people, 'No, I'm sorry, but there's no room in the schedule for feature X that you want. Can I interest you in feature Y, which is in a different part of the code?'

  Also, if your code is fairly big (more than a few hundred thousand lines), you need to break it into logical chunks and assign somebody to watch every checkin to each chunk. That person is a developer and responsible for making sure new code gets reviewed and unexpected changes aren't being made. If your code is smaller, one person can probably do that.

• Code Architecture. If several functionally-unrelated features end up needing to change the same file, then something is wrong with that file. There's too much going on in it--you've got to be dilligent about keeping your components small and keeping each component in a separate file. See the excellent Lakos book for tips on how and why to do that.

Most likely, your organization went way too fast at some point in the course of setting up the core code architecture and the processes by which you decide what does and does not go into a release. You need to get started fixing both--or this problem will keep getting worse and worse until you're unable to move forward through your own inertia.

On CVS and Clearcase

[ajs]

I've admined both extensively, and I can make a couple of comments here. First, Clearcase is licensed software. Understand that when you get locked out because all of the licenses are in use, you cannot touch your source-code (though someone with a license can copy it into a sandbox for you). Also, Clearcase is a resource pig. It wants a pretty beefy central machine to run on, and if lots of people compile at the same time, the virtual filesystem is not very efficient.

Now on to CVS. CVS is most everything you want from revision control. It's biggest shortcomings are in branch management and the ease with which changes can be made incorrectly. Its ability to interface with well known and standard protocols like rsh, ssh and gzip (which is a format more than a protocol) make it painful to move to anything that's overly proprietary. Its use of your local diff is wonderful ("cvs diff -u" was a revelation for me).

Clearcase manages branches better and can handle non-realtime latency in updates (e.g. you can have two Clearcase repositories at different sites and you can connect them by mailing tapes around or by dialing up once a day). This can be invaluable when you're working in high-security environments, but is otherwise mostly a moot point.

Clearcase has improved in the last few years. They've added some local-checkout features where you don't have to work off of the virtual filesystem, and that helps.

Overall, I'd say CVS is the better system, but Clearcase will sometimes get jammed down your throat, and there are definitely worse fates than to have to get your project working under it.