Slashdot Mirror
Tips on Managing Concurrent Development?
"Take, for example, the extreme case of something like Linux (not only concurrent development, but geographically distributed development), how is this managed? One solution we were contemplating was to try to do an 'air traffic control' type of sequencing and conflict resolution. As early as possible in the development stage, we try to identify what will be finished when, and assign a one-up sequence number to each patch. Developers then know that they will be patching against the baseline that was patched by the patch with the previous sequence number. It is hoped that this prevents a lot of rework of patches. A potential problem with this approach is the need for a responsive central authority to assign sequence numbers. Also, such sequence numbers may have to be rearranged in the face of last minute advances and setbacks in developer progress. Despite careful scheduling and detailed design, it may be impossible to know the exact check-in sequence of patches more than a week or two in advance.
Will such an idea be successful, or is it fatally flawed? Are there better solutions to the problem with less effort? Are we treating symptoms and not the disease (i.e., should we be planning better so that we know patch sequences and dependencies early on)? Management likes to keep staff productively occupied and working up until deadlines, so this usually means a lot of checkins within a short period of time, rather than staged checkins. Can checkins be spread out over time while keeping developers productively occupied?"
I don't think so, but then to many people this might be large...
Of course some of these problems sound like lack of planning early in the game...
For example changing headers that two developers need... The only headers that two groups need should be interface headers, these should be set early and not need a lot of change, with any change taking both developers changing the code internally...
Another note, I get really worried when people say that process problems only show up at the end crunch time. If it is crunchtime it is time to use all of your processes, because the processes should be designed to produce the best bug free code the quickest... otherwise it shouldn't be in the process...
That is just my 2c worth however
The only thing I know that really works (i.e. "is simple") is to lessen the conflicts through design... that is, two people shouldn't HAVE to edit the same modules. Or at least not the same lines of the same modules (those are the only merges that are really painful). Similarly, if you have well understood specification for modules then there should not be a problem when the lines edited don't overlap, because the functions and modules will continue to behave to the spec, which is all the other code can expect.
I know this isn't really easy to do (can't be done retroactively), and doesn't really fit all cases (such as near a release when there is a lot of chaos), but it's the only elegant solution I know of, all the rest are more brute force.
-pyrrho
Why in the world should developers be applying the same patch multiple times? You've just said that the problem is not with developers needing to touch the same lines of code -- so, once a patch is in, shouldn't the next person be merging their code with what's already there?
If your problem is with people overwriting the changes that previous submitters made, then you've got a very different kind of mgt problem -- one that can be solved by getting people to use the tools they already have. CVS, for example, lets you merge the current branch head with your working copy, incorporating any changes that may have been made since you checked things out.
Submitters should always diff their current code with the head before commiting a check-in, to see if they are breaking previous changes. This kind of practice is more important when schedules are tight, and you shouldn't let people off the hook because they were in a rush or some other lame excuse.
--tsw
I manage a somewhat larger project (35 developers at two sites), and we have very few issues of the type you describe, despite aggressive development schedules. The key is in architecture and planning. If developers are really performing tasks so different from each other that they aren't already in close communication with one another, you need to ask yourself how they happen to be working in the same source file. With proper architecture and modularization, discrete tasks will naturally land in different parts of your source tree.
There will always be some cases where there's a little overlap, but if your architecture makes these overlaps rare, it becomes relatively easy to see them coming when you lay out your schedule and plan around them.
You might try it, but you'd probably find it lacking. Bugzilla is far from a good bug tracking system. Actually, let me clarify that -- it's a great bug tracking system, if you're tracking bugs in Mozilla. It's horrible for anything else. Data is hardcoded in source files, and if you want to configure it for non-mozilla bug tracking, then you have to edit the source directly. Red Hat and GNOME have obviously put the time in to do so, and have got good results, but for a small business like ours, we couldn't justify the manpower needed to get the system up and running, so we were forced to go with an alternate solution (one I hacked together in PHP one evening -- it may not be pretty, but it works, and gives us 90% of what we need).
"The invisible and the non-existent look very much alike." -- Delos B. McKown
The reason it isn't concurrent is because you have people separately working on patches, but those patches are dependend on one another, so their ordering matters. When ordering matters, the situation is sequential, not parallel. The whole bit about air traffic control and sequence numbers is about serializing the parallel development.
The trick is to decompose the development task into chunks that are in fact parallelizable. In turn, those chunks may have sub-chunks which are not parallelizable; those chunks should be perhaps done by one developer, in the correct order.
No developer should wait around for another's patch, and nobody should develop anything that he or she knows will soon be invalidated by a forthcoming patch so badly that it will have to be substantially reworked. If a unit of work depends on some forthcoming patch so badly, a developer should find something else to work on until that patch arrives. How you know that the patch has arrived is by monitoring your e-mail, or scanning the version control system for changes. The other developers should know that someone is waiting for their patch in order to do the next, dependent part of the change, and broadcast it to the team when they are done.
I worked on a large project (20+ developers) where this situation occured from time to time. There were specific interdependencies between some sources files for various parts of the development effort, and it was easy to step on other peoples feet unless specific steps were taken to prevent this.
Before I describe how we handled this situation, I want to stress the fact that if someone intelligently devides the labor according to how the changes will affect other parts of the code, the need for developers to sqabble over specific changes in specific files should be eliminated.
Labor should be devided at well defined "interface points" where the additions/changes to the interface can be done quickly, satisfying the needs of other developers requiring those interfaces to build against, and then completion of the underlying code can be done with little interference or effect on others.
In short, devide work along interface boundaries, and stub out interfaces with enough code to allow compiling, while developer(s) continue to actually implement the code behind the scenes. Thus, swapping in the actual code has no effect on any one else code, exept that the stubs are now full implementations and work correctly.
Ok, so what happens if the devision isnt clean and you have two people working on the same file?
NOTE: When I am talking about file granularity, or developers "owning" specific files, you can also substiture "subsystem", etc. Sometimes developers are working in entirely different areas of the source tree for the most part, and it makes sense to assign an entire sub tree to a specific developer. This is the devision by interface, which is the usual case.
What we did was assign specific files to specific developers, who have the most work to add/modify to the file. When other developers require changes to a file "owned" by another, they perform the merge, which is verified by both sides to work correctly, and then it is checked in by the "owner".
This was all accomplished using locks (file checked out, ClearCase) and multiple views. The locking of files was a benifit, as it prevented accidental overwrites of other peoples code. Once you check out a file and lock it, no one else (short of the administrator) can check in a modified version and clobber your changes.
A short scenario:
Alice: owns file/tree "something.cpp"
Bob: owns file/tree "modified.cpp"
Tree is something like this:
RootBranch
|
|-- Devel
|
|-- AliceView ---- BobView
| |
[code] [code]
Alice and Bob are both working in a development branch. Alice has the files she is modifying and "owns" checked out and locked. Same for bob.
Bob realizes he needs to make changes to "something.cpp" to support some changes he is making in "modified.cpp".
He checks out a temporary version, unlocked, of "something.cpp" and makes the required changes.
He then notifies alice of the changes, and using the automated merge features she adds his changes, manually resolving conflicts if necessary.
Bob changes his view to use Alice's version of the file with the rest of the code from his view. He builds and verifies that everything is working correctly. Once this is verified, Alice can check in the changes, and Bob can now use the most recently committed version and continue on his merry way.
What this boils down to is basically enforcing ownership through locks to prevent accidental overwrites of others code, and defining clear lines of ownership so that a change is only accepted and merged when the person responsible for that code has tested it (in addition to the developer desiring his minor modifications be included)
That's the a few points that I've found to be helpful in my professional work. Your mileage may vary.
Good luck,
--Mid
[I am not a CVS guru, I just use it.]
If you have to apply patches multiple times, then you're probably patching branches, and developing in the branches. The "CVS Way" seems to be (corrections welcome) to develop in the default branch, and to tag the tree at drop-points -- when you ship the code. If you need to support an old code-drop, you turn the tag into a branch, and then patch the branch.
If you have too many delivered branches being supported at the same time, perhaps you should upgrade those customers to a newer version of the software. They'll appreciate, and it will simplify your situation.
The develop-in-a-branch-and-ship-the-default is appealing, but troublesome.
Otherwise, it sounds like your developers aren't playing nice... developer A patches the tree, and developer B goes to commit his changes, but gets told that there are conflicts and that he needs to update. Not wanting to deal with the conflicts, he copies his important files to a save spot, updates, copies his "important" files back over the top of the conflicted files, and then commits the whole thing, effectively "rolling back" the patch.
If this is what is going on, you need to educate your developers. With a stick.
Over the years, I've discovered that a significant amount of heartburn I may have with CVS comes not from any deficiency of CVS, but from the fact that I frequently fail to use CVS "properly" -- meaning, of course, "as intended".
Pick One: http://www-rohan.sdsu.edu/~stremler/sigs/sigs.html (Note - disable Javascript first!)
I run a small development firm and I wanted to use the enterprise edition. I'll pay a few thousand for something, but $30k for 30 people? I think not.
While I can't say the SourceForge is decent (imho looks like a hodgepodge of thrown together junk), $30k for 30 people is not really a lot. Say salary, benefits, overhead, etc., cost you $1k per week per employee (which is likely very modest). Is something like this going to save each developer a week of time? If there's a tool that good, then it's well worth the money. Sure, it's a big one time cost, but even over the course of a year, it really does pay for itself.