Tips on Managing Concurrent Development?

An Anonymous Coward queries: "I work on a fairly large-sized project with at least a dozen developers. Advanced tools like CVS and ClearCase allow concurrent development, and provide merging tools to merge in different changes to the same file. This can be a significant productivity gain, particularly with files that are unavoidably common to several developers (C header files, most notoriously). During crunch times, such as before delivery deadlines, we often find that we are checking in changes to the same file several times a day, often hourly. The problem does not seem to be with conflicting changes to the same lines of code, but rather with developers knowing the sequence in which concurrent changes will be checked in. It is not possible to always be aware of who is checking in what and when, so programmers submitting patches to the baseline often have to redo those patches multiple times in a day in order to have them applied. Have other programming projects developed solutions for dealing with this problem?" The submitter proposes another solution, below, how well would it work?

"Take, for example, the extreme case of something like Linux (not only concurrent development, but geographically distributed development), how is this managed? One solution we were contemplating was to try to do an 'air traffic control' type of sequencing and conflict resolution. As early as possible in the development stage, we try to identify what will be finished when, and assign a one-up sequence number to each patch. Developers then know that they will be patching against the baseline that was patched by the patch with the previous sequence number. It is hoped that this prevents a lot of rework of patches. A potential problem with this approach is the need for a responsive central authority to assign sequence numbers. Also, such sequence numbers may have to be rearranged in the face of last minute advances and setbacks in developer progress. Despite careful scheduling and detailed design, it may be impossible to know the exact check-in sequence of patches more than a week or two in advance.

Will such an idea be successful, or is it fatally flawed? Are there better solutions to the problem with less effort? Are we treating symptoms and not the disease (i.e., should we be planning better so that we know patch sequences and dependencies early on)? Management likes to keep staff productively occupied and working up until deadlines, so this usually means a lot of checkins within a short period of time, rather than staged checkins. Can checkins be spread out over time while keeping developers productively occupied?"

Comments

SourceForge 3.0 Enterprise Edition

[Anonymous Coward]

(disclaimer: I am a VA Software employee)

I know this sounds corny, being said on a VA property such as Slashdot, but SourceForge 3.0 is easily the best concurrent development environment i have ever used. It was my love for Sourceforge which made me pursue a job at VA Software in the first place! The fully web-based administration hides all the niggling details of commandline cvs tools and makes managing huge projects a piece of cake.

In short, if you haven't been to VA Software's site, you don't know what you're missing.

Better Solution: Use CVS or ClearCase Properly!

[noahbagels]

I currently work in an organization with 75 Engineers (50 USA, 20 India, 5 Asia) - and we use CVS. It's free, easy to use, and has a simple feature set so that more than one person has enough knowledge to do things like branching and merging branches.

We nearly never have merge problems. It is standard procedure that people keep their tree up-to-date with the cvs tree, and thus conflicts rarely arrise. Even at crunch-time, I probably have one merge conflict every 2 weeks, and with CVS - you are notified of the conflict and it is wrapped with CVS comments.

To put this in perspective - while at Oracle with 1000s of engineers working on the same tree, we used ClearCase and it was awesome. The difference here is that there was much steeper a learning curve, and no normal engineers could actually do complex tasks - i.e. create branches etc. We had a complete groud dedicated to ClearCase.


Conclusion:
Educate your engineers - and politely have the senior engineers tell them when they mess up - enforce a policy that people must update the source every day that they plan on checking-in files.

Also - I don't know what CVS versio you are using, but the latest free WinCVS client will not allow you to check in a file with a conflict! It will force you to update/merge/resolve the conflict before updating the tree. I highly recommend CVS and WinCVS due to the ease of use and cost.

Re:Better Solution: Use CVS or ClearCase Properly!

[PD]

We nearly never have merge problems. It is standard procedure that people keep their tree up-to-date with the cvs tree, and thus conflicts rarely arrise

I'll second this: At many companies I have heard over and over that CVS sucks because of the conflicts. When I inquire further, I find out that people never or rarely use the cvs update command to synchronize. cvs update should be executed almost like a nervous habit, or at least a couple times a day. More if code is being checked in frequently, or if you are working on the same code as someone else. Use the watch commands in cvs to be notified when others edit your files.

It's a shame, really, because programmers seem to be afraid of cvs, preferring a more primitive tool such as rcs or pvcs. cvs lets programmers work in the style that is most natural in an open-source arrangement, and in my opinion can be a far more productive environment than a systems that locks the rest of the team out of critical files.

Source Control + Automated Build & Test

[spullara]

These are the ingredients to make large projects successful from a technical point of view. At the company I work for, we have literally hundreds of people working in the same source tree using P4. It manages merges, versioning, and works flawlessly over the internet (well VPN anyway). It is also much, much faster at syncing to the the depot than CVS because the server keeps track of those files that you are editing and does not need to do diffs with the local filesystem. This is very helpful during crunchtime where you might want to sync serveral times a day (and you have about 10000 files in the system). Also, until your locally edited files are resolved with changes in the depot you cannot submit them, so you don't have the problem of ordering patches properly.

For the second part, I highly reccommend that you have automated build and tests that run after changes have been submitted. You can see how this is done en mass on the mozilla.org site. Also, developers should have access to the same build and test infrastructure on their machines so they can do the build and test before they check in their code.

Finally, you need a good bug tracking system. You might try Bugzilla.

Good luck,
Sam

On CVS and Clearcase

[ajs]

I've admined both extensively, and I can make a couple of comments here. First, Clearcase is licensed software. Understand that when you get locked out because all of the licenses are in use, you cannot touch your source-code (though someone with a license can copy it into a sandbox for you). Also, Clearcase is a resource pig. It wants a pretty beefy central machine to run on, and if lots of people compile at the same time, the virtual filesystem is not very efficient.

Now on to CVS. CVS is most everything you want from revision control. It's biggest shortcomings are in branch management and the ease with which changes can be made incorrectly. Its ability to interface with well known and standard protocols like rsh, ssh and gzip (which is a format more than a protocol) make it painful to move to anything that's overly proprietary. Its use of your local diff is wonderful ("cvs diff -u" was a revelation for me).

Clearcase manages branches better and can handle non-realtime latency in updates (e.g. you can have two Clearcase repositories at different sites and you can connect them by mailing tapes around or by dialing up once a day). This can be invaluable when you're working in high-security environments, but is otherwise mostly a moot point.

Clearcase has improved in the last few years. They've added some local-checkout features where you don't have to work off of the virtual filesystem, and that helps.

Overall, I'd say CVS is the better system, but Clearcase will sometimes get jammed down your throat, and there are definitely worse fates than to have to get your project working under it.