Slashdot Mirror
Tips on Managing Concurrent Development?
"Take, for example, the extreme case of something like Linux (not only concurrent development, but geographically distributed development), how is this managed? One solution we were contemplating was to try to do an 'air traffic control' type of sequencing and conflict resolution. As early as possible in the development stage, we try to identify what will be finished when, and assign a one-up sequence number to each patch. Developers then know that they will be patching against the baseline that was patched by the patch with the previous sequence number. It is hoped that this prevents a lot of rework of patches. A potential problem with this approach is the need for a responsive central authority to assign sequence numbers. Also, such sequence numbers may have to be rearranged in the face of last minute advances and setbacks in developer progress. Despite careful scheduling and detailed design, it may be impossible to know the exact check-in sequence of patches more than a week or two in advance.
Will such an idea be successful, or is it fatally flawed? Are there better solutions to the problem with less effort? Are we treating symptoms and not the disease (i.e., should we be planning better so that we know patch sequences and dependencies early on)? Management likes to keep staff productively occupied and working up until deadlines, so this usually means a lot of checkins within a short period of time, rather than staged checkins. Can checkins be spread out over time while keeping developers productively occupied?"
Maybe we're just stupid. We dont use any CVS or any other versioning type software to keep track of changes. We dont' check in our code or anything.
Luckily our development is done on the web, we just create folders, and move them up when we're done.
I know this sounds corny, being said on a VA property such as Slashdot, but SourceForge 3.0 is easily the best concurrent development environment i have ever used. It was my love for Sourceforge which made me pursue a job at VA Software in the first place! The fully web-based administration hides all the niggling details of commandline cvs tools and makes managing huge projects a piece of cake.
In short, if you haven't been to VA Software's site, you don't know what you're missing.
I wouldn't, however, recommend working with anything from Microsoft. Benchmarks and real-life statistics have shown that their source control solutions are not only slower, but are also less stable and more likely to corrupt your source tree. I hope you have backups!
"I don't know that atheists should be considered citizens, nor should they be considered patriots." - George Bush
I don't think so, but then to many people this might be large...
Of course some of these problems sound like lack of planning early in the game...
For example changing headers that two developers need... The only headers that two groups need should be interface headers, these should be set early and not need a lot of change, with any change taking both developers changing the code internally...
Another note, I get really worried when people say that process problems only show up at the end crunch time. If it is crunchtime it is time to use all of your processes, because the processes should be designed to produce the best bug free code the quickest... otherwise it shouldn't be in the process...
That is just my 2c worth however
I currently work in an organization with 75 Engineers (50 USA, 20 India, 5 Asia) - and we use CVS. It's free, easy to use, and has a simple feature set so that more than one person has enough knowledge to do things like branching and merging branches.
We nearly never have merge problems. It is standard procedure that people keep their tree up-to-date with the cvs tree, and thus conflicts rarely arrise. Even at crunch-time, I probably have one merge conflict every 2 weeks, and with CVS - you are notified of the conflict and it is wrapped with CVS comments.
To put this in perspective - while at Oracle with 1000s of engineers working on the same tree, we used ClearCase and it was awesome. The difference here is that there was much steeper a learning curve, and no normal engineers could actually do complex tasks - i.e. create branches etc. We had a complete groud dedicated to ClearCase.
Conclusion:
Educate your engineers - and politely have the senior engineers tell them when they mess up - enforce a policy that people must update the source every day that they plan on checking-in files.
Also - I don't know what CVS versio you are using, but the latest free WinCVS client will not allow you to check in a file with a conflict! It will force you to update/merge/resolve the conflict before updating the tree. I highly recommend CVS and WinCVS due to the ease of use and cost.
Check out subversion.
It's CVS, but better and based on WebDAV for RPC and BerkeleyDB for storage.
Cheers,
pointer
[%- PROCESS life -%]
The only thing I know that really works (i.e. "is simple") is to lessen the conflicts through design... that is, two people shouldn't HAVE to edit the same modules. Or at least not the same lines of the same modules (those are the only merges that are really painful). Similarly, if you have well understood specification for modules then there should not be a problem when the lines edited don't overlap, because the functions and modules will continue to behave to the spec, which is all the other code can expect.
I know this isn't really easy to do (can't be done retroactively), and doesn't really fit all cases (such as near a release when there is a lot of chaos), but it's the only elegant solution I know of, all the rest are more brute force.
-pyrrho
One solution to avoid patching problem is to use continuous integration. It's an integration technique that builds your source multiple times a day, getting all the latest source code from the CVS tree, and building from that code. If anything fails, the offending developer gets warned. Mozilla uses the same thing, calling it TinderBox. It's one of the principles of Extreme Programming, and seems to work quite well at our company.
Check out the development techniques of Extreme Programming (just search Google, silly, and buy a book or three). They have a real solid handle on concurrent rapid development.
The real heart of Extreme Programming is "test-first" programming. The entire development process revolves around unit and integration tests, for extremely fine-grained control over code quality. Any changes that might impact other code should break a test. You fix the stuff that breaks, check in your changes, and move on.
Multiple programmers touching the same C files many times a day sounds like you have either design issues, structural issues, or both. That just should not happen, crunch time or not. Heck, crunch shouldn't happen if you're managing your development correctly.
If you're using cvs, conflicts with source checkins should be very easy to resolve. Even if two programmers touch the same file, they shouldn't be in the same function. If they are, you're back to management and architecture problems, and you need to fix those NOW before work grinds to a complete halt.
Hand me that airplane glue and I'll tell you another story.
Why in the world should developers be applying the same patch multiple times? You've just said that the problem is not with developers needing to touch the same lines of code -- so, once a patch is in, shouldn't the next person be merging their code with what's already there?
If your problem is with people overwriting the changes that previous submitters made, then you've got a very different kind of mgt problem -- one that can be solved by getting people to use the tools they already have. CVS, for example, lets you merge the current branch head with your working copy, incorporating any changes that may have been made since you checked things out.
Submitters should always diff their current code with the head before commiting a check-in, to see if they are breaking previous changes. This kind of practice is more important when schedules are tight, and you shouldn't let people off the hook because they were in a rush or some other lame excuse.
--tsw
These are the ingredients to make large projects successful from a technical point of view. At the company I work for, we have literally hundreds of people working in the same source tree using P4. It manages merges, versioning, and works flawlessly over the internet (well VPN anyway). It is also much, much faster at syncing to the the depot than CVS because the server keeps track of those files that you are editing and does not need to do diffs with the local filesystem. This is very helpful during crunchtime where you might want to sync serveral times a day (and you have about 10000 files in the system). Also, until your locally edited files are resolved with changes in the depot you cannot submit them, so you don't have the problem of ordering patches properly.
For the second part, I highly reccommend that you have automated build and tests that run after changes have been submitted. You can see how this is done en mass on the mozilla.org site. Also, developers should have access to the same build and test infrastructure on their machines so they can do the build and test before they check in their code.
Finally, you need a good bug tracking system. You might try Bugzilla.
Good luck,
Sam
"If I can see farther it is because I am surrounded by dwarves." -- Murray Gell-Mann
I manage a somewhat larger project (35 developers at two sites), and we have very few issues of the type you describe, despite aggressive development schedules. The key is in architecture and planning. If developers are really performing tasks so different from each other that they aren't already in close communication with one another, you need to ask yourself how they happen to be working in the same source file. With proper architecture and modularization, discrete tasks will naturally land in different parts of your source tree.
There will always be some cases where there's a little overlap, but if your architecture makes these overlaps rare, it becomes relatively easy to see them coming when you lay out your schedule and plan around them.
If you have several people changing the same file in a given day, then one of two things (probably both) is wrong:
- Coordination between features/projects. Somebody should be keeping an eye on the list of fixes/enhancements that are coming down the line and making sure you don't get too many in the same neck of the woods. This person doesn't necessarily have to be a developer (but should be able to speak developerese), and their whole job is to tell people, 'No, I'm sorry, but there's no room in the schedule for feature X that you want. Can I interest you in feature Y, which is in a different part of the code?'
- Code Architecture. If several functionally-unrelated features end up needing to change the same file, then something is wrong with that file. There's too much going on in it--you've got to be dilligent about keeping your components small and keeping each component in a separate file. See the excellent Lakos book for tips on how and why to do that.
Most likely, your organization went way too fast at some point in the course of setting up the core code architecture and the processes by which you decide what does and does not go into a release. You need to get started fixing both--or this problem will keep getting worse and worse until you're unable to move forward through your own inertia.Also, if your code is fairly big (more than a few hundred thousand lines), you need to break it into logical chunks and assign somebody to watch every checkin to each chunk. That person is a developer and responsible for making sure new code gets reviewed and unexpected changes aren't being made. If your code is smaller, one person can probably do that.
The reason it isn't concurrent is because you have people separately working on patches, but those patches are dependend on one another, so their ordering matters. When ordering matters, the situation is sequential, not parallel. The whole bit about air traffic control and sequence numbers is about serializing the parallel development.
The trick is to decompose the development task into chunks that are in fact parallelizable. In turn, those chunks may have sub-chunks which are not parallelizable; those chunks should be perhaps done by one developer, in the correct order.
No developer should wait around for another's patch, and nobody should develop anything that he or she knows will soon be invalidated by a forthcoming patch so badly that it will have to be substantially reworked. If a unit of work depends on some forthcoming patch so badly, a developer should find something else to work on until that patch arrives. How you know that the patch has arrived is by monitoring your e-mail, or scanning the version control system for changes. The other developers should know that someone is waiting for their patch in order to do the next, dependent part of the change, and broadcast it to the team when they are done.
I worked on a large project (20+ developers) where this situation occured from time to time. There were specific interdependencies between some sources files for various parts of the development effort, and it was easy to step on other peoples feet unless specific steps were taken to prevent this.
Before I describe how we handled this situation, I want to stress the fact that if someone intelligently devides the labor according to how the changes will affect other parts of the code, the need for developers to sqabble over specific changes in specific files should be eliminated.
Labor should be devided at well defined "interface points" where the additions/changes to the interface can be done quickly, satisfying the needs of other developers requiring those interfaces to build against, and then completion of the underlying code can be done with little interference or effect on others.
In short, devide work along interface boundaries, and stub out interfaces with enough code to allow compiling, while developer(s) continue to actually implement the code behind the scenes. Thus, swapping in the actual code has no effect on any one else code, exept that the stubs are now full implementations and work correctly.
Ok, so what happens if the devision isnt clean and you have two people working on the same file?
NOTE: When I am talking about file granularity, or developers "owning" specific files, you can also substiture "subsystem", etc. Sometimes developers are working in entirely different areas of the source tree for the most part, and it makes sense to assign an entire sub tree to a specific developer. This is the devision by interface, which is the usual case.
What we did was assign specific files to specific developers, who have the most work to add/modify to the file. When other developers require changes to a file "owned" by another, they perform the merge, which is verified by both sides to work correctly, and then it is checked in by the "owner".
This was all accomplished using locks (file checked out, ClearCase) and multiple views. The locking of files was a benifit, as it prevented accidental overwrites of other peoples code. Once you check out a file and lock it, no one else (short of the administrator) can check in a modified version and clobber your changes.
A short scenario:
Alice: owns file/tree "something.cpp"
Bob: owns file/tree "modified.cpp"
Tree is something like this:
RootBranch
|
|-- Devel
|
|-- AliceView ---- BobView
| |
[code] [code]
Alice and Bob are both working in a development branch. Alice has the files she is modifying and "owns" checked out and locked. Same for bob.
Bob realizes he needs to make changes to "something.cpp" to support some changes he is making in "modified.cpp".
He checks out a temporary version, unlocked, of "something.cpp" and makes the required changes.
He then notifies alice of the changes, and using the automated merge features she adds his changes, manually resolving conflicts if necessary.
Bob changes his view to use Alice's version of the file with the rest of the code from his view. He builds and verifies that everything is working correctly. Once this is verified, Alice can check in the changes, and Bob can now use the most recently committed version and continue on his merry way.
What this boils down to is basically enforcing ownership through locks to prevent accidental overwrites of others code, and defining clear lines of ownership so that a change is only accepted and merged when the person responsible for that code has tested it (in addition to the developer desiring his minor modifications be included)
I've admined both extensively, and I can make a couple of comments here. First, Clearcase is licensed software. Understand that when you get locked out because all of the licenses are in use, you cannot touch your source-code (though someone with a license can copy it into a sandbox for you). Also, Clearcase is a resource pig. It wants a pretty beefy central machine to run on, and if lots of people compile at the same time, the virtual filesystem is not very efficient.
Now on to CVS. CVS is most everything you want from revision control. It's biggest shortcomings are in branch management and the ease with which changes can be made incorrectly. Its ability to interface with well known and standard protocols like rsh, ssh and gzip (which is a format more than a protocol) make it painful to move to anything that's overly proprietary. Its use of your local diff is wonderful ("cvs diff -u" was a revelation for me).
Clearcase manages branches better and can handle non-realtime latency in updates (e.g. you can have two Clearcase repositories at different sites and you can connect them by mailing tapes around or by dialing up once a day). This can be invaluable when you're working in high-security environments, but is otherwise mostly a moot point.
Clearcase has improved in the last few years. They've added some local-checkout features where you don't have to work off of the virtual filesystem, and that helps.
Overall, I'd say CVS is the better system, but Clearcase will sometimes get jammed down your throat, and there are definitely worse fates than to have to get your project working under it.
Conicidence? That's Your call.
I disable sigs...do you?
- Smaller. You only need one executable on your client. And one more for your server. No kernel patches, no drivers, no installation, just the binaries. Ubercool.
- Multi-platform. Perforce has binaries for practically every platform in use out there. Find me another Version Control System with BeOS, QNX, AIX, SCO, MacOS9 and MacOSX support.
- Fast, fast fast. Because of the low communication overhead, it works extremely well across slow/high latency links.
- Ease of use. It's really easy to configure and setup.
- Great support. We've had to go to perforce support twice and both times they've been awesome, with quick responses and knowledgeable people.
- Price. The single server, two client setup is FREE! And per seat licensing + support is very very reasonable. I use the free one on my laptop to version files.
- Plug-ins. Perforce publishes their API, and they have perl, python, ruby and tcl utilities galore.
Now my process recommendation. If you have paid for ClearCase then pay for some more education. ClearCase/Perforce can do lots of what you need automatically. You should practically NEVER have to repeat the same change on a file. You may want to look at some branching/merging techniques which can eliminate the need for colliding checkin's also. Rational has a bunch of whitepapers up on their site, as does Perforce.Most of all, I would advise you to educate yourself on the options/methods of version control. 12 isn't that big. Wait 'til you get to 1200.
There is no trap so deadly as the trap you set for yourself
-Raymond Chandler, The Long Goodbye
That's the a few points that I've found to be helpful in my professional work. Your mileage may vary.
Good luck,
--Mid
I have used both CVS, Clearcase, MKS, straight RCS, and a bit of Visual Source-Safe (Ha!).
What I have found makes the most difference in reducing merge issues is (1), have developers merge on checkin (more on that in a moment), and (2) develop a branching strategy that reduces the need to merge.
On (1). When using ClearCase, have people set checkouts to be "unreserved". When they check out a file it won't lock it so everyone else can use it, and when they check something in ClearCase will force them to perform a merge if it has been updated since they checked it out. Make everyone learn how to use the henious merge tool. Also (and I'm not sure if this is fixed now or if I never figured out how to configure this right) I have had major issues with EOL markers in files and clearcase merge - if a developer edits a file that changes the EOL character(s), then ClearCase merge will claim the whole file differs.
For CVS, it pretty much works naturally the way a system should - the default is to check out something "unreserved" (or at least that's how I remember it!). You can either update a file while you're making changes to keep up to date an make merges smaller, or just wait until you're done - before you can check it in CVS will make you update the file and merge your changes.
In both systems, an approach of having the developer merge the file means the person who really knows what's going on can resolve any conflicts with the merge. Most merges are automatically handled and so often no work is required. I'm not sure how much this really addresses your issue, but it can't hurt to rely more on the source control system helping you manage merges.
For option (2), think carefully about how you want to structure releases. One approach I've used before is having dfferent "levels" of development - you have a production branch, fix branch, development 1, dev 2, etc. You start out code at some given level (say development 2) and as it is completed and tested it moves up through the ranks until it reaches the production level. That worked pretty well and meant a fixed number of branches.
In more recent projects I've been working with a monthly release structure - each months release requires a new branch. This was not my idea, believe me... it might seem nice in theory but in practice you might have three or more months of development at the same time. This leads to something of a merging cross-nightmare. Some sort of branching structure might well help solve your problem, if only in thinking up how you can more clearly seperate changes to a file or set of files.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
[I am not a CVS guru, I just use it.]
If you have to apply patches multiple times, then you're probably patching branches, and developing in the branches. The "CVS Way" seems to be (corrections welcome) to develop in the default branch, and to tag the tree at drop-points -- when you ship the code. If you need to support an old code-drop, you turn the tag into a branch, and then patch the branch.
If you have too many delivered branches being supported at the same time, perhaps you should upgrade those customers to a newer version of the software. They'll appreciate, and it will simplify your situation.
The develop-in-a-branch-and-ship-the-default is appealing, but troublesome.
Otherwise, it sounds like your developers aren't playing nice... developer A patches the tree, and developer B goes to commit his changes, but gets told that there are conflicts and that he needs to update. Not wanting to deal with the conflicts, he copies his important files to a save spot, updates, copies his "important" files back over the top of the conflicted files, and then commits the whole thing, effectively "rolling back" the patch.
If this is what is going on, you need to educate your developers. With a stick.
Over the years, I've discovered that a significant amount of heartburn I may have with CVS comes not from any deficiency of CVS, but from the fact that I frequently fail to use CVS "properly" -- meaning, of course, "as intended".
Pick One: http://www-rohan.sdsu.edu/~stremler/sigs/sigs.html (Note - disable Javascript first!)
As the author of the original 386BSD 0.1 PatchKit software, I have to say that your "air traffic control" approach will not work.
The 386BSD 0.1 patchkit used a serialization of patch numbers, with central assignment. The reason for this was that the patch dependency management was done by manually applying patches posted to Usenet, and then diffing the modified version of the code against a version with the previous N-1 applied.
Effectively, it was a "human CVS repository" system.
Ir was necessary, because the latency in the Usenet system meant that you couldn't "lock down" a file or set of files for some major change: you had to do what you wanted to do against what you had, which was almost never "the most currnet concensual version" of the code, and then hope someone else didn't win the race to "the repository" (at the time, terry@cs.weber.edu's incoming email, and then, later, Rod Grimes', Nate Williams', and then Jordan Hubbard's... no one wanted it for very long).
This led to all sorts of problems; the major one was that the patch kit format was "reverse engineered" (not hard; the patch tools, except the creation software itself, were widely distributed), and a group started releasing patches in the "1000+" ID range, under the incorrect impression that the concern was over the patch namespace collision, not topological application problems. This eventually led to a big argument, and other people going off to play in their own sandbox.
You've probably heard of "NetBSD". A couple (not all, of course) were motivated by communit rejection of the 1000+ numbered patches, which, while they were not colliding in serial number space, seriously blew out topological dependency space for modified files.
In any case, that's exactly what you are doing with your code, when you plan on assigning patch numbers based on expectation of completion.
With the number of people you have, the comments about contested interfaces being agreed to beforehand, and the comments about you having no real problem here in the first place are probably accurate.
You can basically take a couple of approaches.
The first is: don't accumulate patches, just check the code in. This respolves the problem of stale patches by not permitting them to become stale in the first place.
The second is: "cvs tag" before any major commits, so that there is a baseline from which to work to resolve conflicts.
Really, you should not be accumulating large patch sets, with as few people as are involved.
If you have a huge offline latency from a developer or group of developers (e.g. you send a CDROM to Antarctia, and two months later the send back a CDROM with their patches on it), or if you have a huge number of developers, you should reconsider your chioce of tools.
The 386BSD patchkit serialization of patch sequence numbers through a couple of human beings was a serious mistake. It had the emergent property of having a tiered set of priviledge. I'm convinced that this is what resulted in the current "core team/committer/less-than-dirt" striation in the BSD camps today.
I mention this, because CVS has a similar, though somewhat less profound, emergent property of "The One True HEAD Branch". By its nature, it encourages a single direction for all experimentation and all forward looking thought, denying nourishment to any contradictory lines of inquiry, by chopping off the roots. CVS is, in a nutshell, anti-research. It prevents people from going off 90 degrees from where everyone else is headed, and discovering new territory.
Perhaps you've heard of OpenBSD. It emerged because there was "One True HEAD Branch" in NetBSD (an early adopter of CVS, in Open Source-land), and several people felt strongly enough that the focus of the project should be secure systems research, that the resulting code directions were incompatible.
Tools issues are at the base of nearly any strong divide you can name in an Open Source community.
Linux currently has issue, where Linus is investigating the use of Larry McVoy's BitKeeper (Larry was smart, in that early on, he recognized the emergent properties tools choices force onto projects, and tried to design around the problem). It turns out that a single human CVS repository doesn't scale infinitely.
FreeBSD is in the throes of a "To use Perforce or not to use Perforce" decision. Perforce supports seperate lines of concurrent developement.
It fosters, as my former boss' boss, Ray Noorda, used to say, "coopetition": help each other make the best implementation according to their design, and then may the best design win.
Perforce lets this happen, but it also tends to balkanize developement, if not everyone is using the tool. There are complaints in FreeBSD that significant work is taking place in Perforce branches that aren't visible to normal CVS users. The Perforce users complain back that there would be no need for Perforce, if the develeopement were permitted in the main CVS tree -- along with the breakage that would entail. Both arguments have merit. Right now, there is a truce... more of an agreement to disagree, and not force the issue today, but a promise that the battle will be fought to the death at some later date.
For your project, a tool which supports multiple concurrent "One True HEAD Branches" seems like it fitys the bill (though as I wrote that, I still asked myself why, with so few people, it was an issue for you in the first place).
Whether the tool you pick is Perforce, Bitkeeper, or some other tool that can support that developement model is irrelevent.
What is relevent is that you understand that our tools shape the way we think about solving problems, and if you have already arrived at an approach that doesn't -- or *can't* -- fit into the shape dictated by CVS, then it's probably time to look at another tool.
Not matter what you do, I can guarantee you that layering another, less adequate, tool on top of an already inadequate tool, will not fix your problem.
I can also guarantee you that if you can't change your model to fit an existing tool, you're going to find yourself in the source code control tools business, instead of the business you intended to be in.
Probably, you should rethink whatever premise it is that's resulting in large, infrequently integrated patch sets. If it's just your release engineering department not wanting to do their work on a branch, well, that's tough. Branch tag for releases as a matter of policy, and move on. If on the other hand, it's something more profound, perhaps you need to rethink your assumptions in favor of what the tools can do, vs. what you would like them to be able to do.
Alternately, welcome to the source code control tool business.
-- Terry
But that doesn't scale... So you've got to modularize the project so that each team works against their own baseline. Any changes that you do to another teams part, must be given to them, for them to check out, and integrate themselves. The important thing is communication, it often happens that they are working on fixing the same problem, but in another and better way, and can give you back their fix instead.
So when you're having problems with your revision control system, what you are really experiencing most of the time, is problems with communication.
Ideally, all developers should be in the same building, they should work at (mostly) the same time (normal office hours), and they should all be friends, and keep a list of each others telephone-numbers, and eat lunch at the same time. Development should be split into projects (having 10-50 people on each project) that are mosty independent. All the people on the project should meet weekly, and discuss their project, important changes, etc... On each project there can be one or more teams, each consisting of four to eight people, who should have offices really close to each other. On each team, development should be split into sub-teams, consisting of 1-3 persons (depending on difficulty and experience), who should share an office, and thus communicate even further. And just as important: this should not be formalized (at least not too much). People should rotate around somewhat, not being stuck with the same people all the time, to get them to know other parts of the project, and other people to communicte with.
The important lesson I am trying to give is that the most beneficial communication, is often the informal. While having a tool help you with managing deltas is surely helpful, it can't solve every problem in the world. You need to work together, but you also need to modularize, and most importantly, you all need to be friends...