Tips on Managing Concurrent Development?

An Anonymous Coward queries: "I work on a fairly large-sized project with at least a dozen developers. Advanced tools like CVS and ClearCase allow concurrent development, and provide merging tools to merge in different changes to the same file. This can be a significant productivity gain, particularly with files that are unavoidably common to several developers (C header files, most notoriously). During crunch times, such as before delivery deadlines, we often find that we are checking in changes to the same file several times a day, often hourly. The problem does not seem to be with conflicting changes to the same lines of code, but rather with developers knowing the sequence in which concurrent changes will be checked in. It is not possible to always be aware of who is checking in what and when, so programmers submitting patches to the baseline often have to redo those patches multiple times in a day in order to have them applied. Have other programming projects developed solutions for dealing with this problem?" The submitter proposes another solution, below, how well would it work?

"Take, for example, the extreme case of something like Linux (not only concurrent development, but geographically distributed development), how is this managed? One solution we were contemplating was to try to do an 'air traffic control' type of sequencing and conflict resolution. As early as possible in the development stage, we try to identify what will be finished when, and assign a one-up sequence number to each patch. Developers then know that they will be patching against the baseline that was patched by the patch with the previous sequence number. It is hoped that this prevents a lot of rework of patches. A potential problem with this approach is the need for a responsive central authority to assign sequence numbers. Also, such sequence numbers may have to be rearranged in the face of last minute advances and setbacks in developer progress. Despite careful scheduling and detailed design, it may be impossible to know the exact check-in sequence of patches more than a week or two in advance.

Will such an idea be successful, or is it fatally flawed? Are there better solutions to the problem with less effort? Are we treating symptoms and not the disease (i.e., should we be planning better so that we know patch sequences and dependencies early on)? Management likes to keep staff productively occupied and working up until deadlines, so this usually means a lot of checkins within a short period of time, rather than staged checkins. Can checkins be spread out over time while keeping developers productively occupied?"

Comments

Extreme Programming

[Frank Sullivan]

Check out the development techniques of Extreme Programming (just search Google, silly, and buy a book or three). They have a real solid handle on concurrent rapid development.

The real heart of Extreme Programming is "test-first" programming. The entire development process revolves around unit and integration tests, for extremely fine-grained control over code quality. Any changes that might impact other code should break a test. You fix the stuff that breaks, check in your changes, and move on.

Multiple programmers touching the same C files many times a day sounds like you have either design issues, structural issues, or both. That just should not happen, crunch time or not. Heck, crunch shouldn't happen if you're managing your development correctly.

If you're using cvs, conflicts with source checkins should be very easy to resolve. Even if two programmers touch the same file, they shouldn't be in the same function. If they are, you're back to management and architecture problems, and you need to fix those NOW before work grinds to a complete halt.

Product Management and Strong Architecture

[dant]

Yikes! You have the process problems of a group five times your size.

If you have several people changing the same file in a given day, then one of two things (probably both) is wrong:

• Coordination between features/projects. Somebody should be keeping an eye on the list of fixes/enhancements that are coming down the line and making sure you don't get too many in the same neck of the woods. This person doesn't necessarily have to be a developer (but should be able to speak developerese), and their whole job is to tell people, 'No, I'm sorry, but there's no room in the schedule for feature X that you want. Can I interest you in feature Y, which is in a different part of the code?'

  Also, if your code is fairly big (more than a few hundred thousand lines), you need to break it into logical chunks and assign somebody to watch every checkin to each chunk. That person is a developer and responsible for making sure new code gets reviewed and unexpected changes aren't being made. If your code is smaller, one person can probably do that.

• Code Architecture. If several functionally-unrelated features end up needing to change the same file, then something is wrong with that file. There's too much going on in it--you've got to be dilligent about keeping your components small and keeping each component in a separate file. See the excellent Lakos book for tips on how and why to do that.

Most likely, your organization went way too fast at some point in the course of setting up the core code architecture and the processes by which you decide what does and does not go into a release. You need to get started fixing both--or this problem will keep getting worse and worse until you're unable to move forward through your own inertia.