Stop Breaking the Build

Cap'n Grumpy writes "You know the score - you've just finished some coding, do a final cvs update before commiting, and all of a sudden all hell breaks loose. Your code now refuses to compile, or xunit starts flashing up red - test failures! One of the other members of your team has checked in something which breaks the build, and they just went out for lunch ... Argh! Did you know there is a solution to this problem? It is a system which makes it impossible for people to check in code which does not compile or test successfully. It allows coders to review others coding efforts code before it goes into the baseline, rather than after. It organises your checkins into logical change sets. It enforces continuous integration. It is linux based, and GPL'd. It's called Aegis."

Comments

Other ideas...

[Pentagon13]

Some more ways to improve the checkin process:

run the code through a lameness filter before allowing the check-in

enforce a 20 second delay from the time you want to check-in to the time you are actually allowed to check-in

make sure a different developer checks-in the exact same code a day or two later

Re:Other ideas...

[Anonymous Coward]

Please try to keep checkins on topic.

Try to update other people's checkins instead of creating new files.

Read other people's checkin notices before checking in your own to avoid simply duplicating what has already been said.

Use a clear subject that describes what your checkin is about.

Offtopic, Inflammatory, Inappropriate, Illegal, or Offensive checkins might be purged. (You can maintain everything, even purged checkins, by adjusting your threshold on the User Preferences Page)

Ads as articles

[Cyclone66]

Is it just me or does this sound like an advertisement?

Part of the problem is CVS

[ClosedSource]

Allowing multiple developers to edit the same file at the same time is inherently more dangerous than a more conservative approach. Open Source has it's own special needs, but for closed source development you should rarely need to edit the same file unless your team is poorly organized or system poorly designed.

Re:Part of the problem is CVS

[etcshadow]

"for closed source development you should rarely need to edit the same file unless your team is poorly organized or system poorly designed"

Oh, come on. That's such a load. I'll agree that CVS is a big part of the problem, but not because you shouldn't let more than one developer edit the same file at the same time. Rather, simply because CVS sucks.

I hate to admit it, because I love the open source movement, and I know what an important role CVS plays in it, but CVS relly does bite compared to some of the commercial alternatives. I mean, no trackable atomic changes? No means for integration with job tracking? A shitty-beyond-belief branch methodology? Poor tracking of and integration of changes across branches? Crappy permissioning structure?

Where I work we use Perforce, which I absolutely adore. Does not have any of the issues I listed above, works in unix and windows, is command line easy and has a pretty damn good GUI (compare to WinCVS, ack!), is wonderfully scriptable, and is really not that expensive (although I can definitely understand the desire to spend $0).

Anyway, you can't really expect to have a large group of developers both iterating and maintaining a fairly large codebase without ever needing to edit overlapping files... not unless you keep each function/subroutine/method in its own file. Even then, I imagine you'd still run into occasional change resolution issues. The better way to deal with the problem is not to close your eyes and put your hands over your ears; it's to outfit yourself with decent tools capable of dealing with real life. I know that with perforce, and I would imagine with most other half-decent source management systems, simultaneous edits are really not that big of a deal. Unless you've actually edited the same lines in the same file, the user doesn't typicaly have to do a damn thing. And regardless, it is still the fault of the first user to check in a busted file. Again, atomic changes mean that if it compiled for you, and you check it in, then compiles for everyone else.

Re:Part of the problem is CVS

[ikeleib]

Having multiple developers tinkering with the same part of the same file is a project management problem- not a tool problem.

Re:Part of the problem is CVS

[Twylite]

but for closed source development you should rarely need to edit the same file unless your team is poorly organized or system poorly designed

...or if you happen to have a high level of code reuse; or if you have doing firmware, software and driver development in parallel; or if you have a small but busy team; or if you have a large but busy team; or ... or ... or.

This is a ridiculous statement. There are any number of reasons that multiple developers will work on a single file at once, especially in a well structure organisation or development. Development, code inspection, fixes in response to testing and maintenance fixes (bringing a patch from a release into current) can ALL happen simultaneously in a development tree, and can ALL happen simultaneously in one file. They just shouldn't often happen simultaneously in one method/function.

Re:Part of the problem is CVS

[Circuit Breaker]

While I agree on most things, WinCVS is certainly NOT the GUI you should be comparing to - TortoiseCVS is.

I have used SourceSafe, Perforce (very lightly), WinCVS, and Tortoise. Tortoise is above and beyond the rest. I still use the command line sometimes for intricate log grepping, but for everyday usage, Tortoise is simply amazing.

TortoiseCVS

Re:Part of the problem is CVS

[renehollan]

Lesse...

I have used, in my time, Clearcase (which I rather liked despite the high price tag and apparent inefficiency repository-side), CVS, and most recently Perforce. For all the complaining about CVS lacking sophistication, it does get 95% of the job of source code control done. But, none of these address the root problem, and it's unfair to pick on one for not addressing it, implying that the others do a better job.

Inconsistent checkins of code, even code in different files, can break builds, unit, and integration tests. Consider the development process:

You check out a read-only version of the top of the development tree, that builds clean, and passes all the tests. Great. You get a write checkout on the stuff you want to change. Even if others don't touch those files (which can be difficult to enforce for some kinds of files, like headers of unique enumerations that everyone updates), you can still break things.

When you test build, you test build with your stuff changed, and everything else frozen. There is no guarantee that when you check in your changes, the break will build because something else on which your code depends got changed. Like a header file. I suppose you could get a write lock on all the files on which your code depends, but that still isn't good enough.

Consider a processing sequence where two function in two code files get called. One of them is supposed to increment some global "the sequence was run" counter. Both do. Oops. Builds fine, fails regression testing.

Short of locking the entire source tree when one developer changes something, you can't avoid this in general. Oh sure, you can resync all the files you didn't change, and run a build and regression test before checking your change in, but, lo, you'd have to lock the entire source tree during that time (or at least see if it changed again after your build and test, and repeat the process if it did, possibly indefinately). Serializing an otherwise parallel development process that way is murder on productivity: even if you run all the sanity builds at night, you now have a one day turnaround just to test build all changes, and hope they make it into the source repository. Kinda sucks, when you just changed a few lines.

Any automated solution will have to rely on serialization of source tree access, at some level. If the project can be broken down into independent components, serialization within a component can be relatively painless, with somewhat less frequent serialization across components (so called "integration" which the nightly build strives to avoid because it happens all the time). Experience shows that, unless you want to plan "integration" phases, this defeats a large benefit of the nightly build process, though, it is not unreasonable for very large projects, with clearly independent parts.

So, what's the solution?

The same, it's always been: divide and conquer.

While "integration" phases are to be shunned, and "repository serialization" kills productivity, one can take a statistical approach: instead of verifying everything before a checkin to a frozen repository, you design your project to try to isolate the effects of implementation changes from one another. Early on, this may be difficult as interfaces are still being tweaked, but you have less code then, and builds happen quickly. This is a large part of what a project architect is supposed to do.

If done correctly, a local delta built with a recent snapshot of the source repository is not likely to break the build and unit test of a more recent snapshot if it builds and passes regression tests against the somewhat older snapshot. This isn't foolproof, of course, but a proper design will have the necessary isolation: an implementation change in one area, or the addition of a new interface should be oblivious to your code.

Now, if interfaces need to change, or new features are to be added that have to be coordinated among developers (i.e. someone adds a feature and someone else writes code to use it), then you need greater coordination among all those that might be affected. Such work can take place on a side branch, and merged to the main development tree only when it is internally consistent. Again, this is generally the responsibility of the common lead programmer of those implementing the changes that need to be coordinated (perhaps the project architect, but in large projects, that kind of detail can be delegated).

Where trouble brews is when such a synchronized change has to take place across development teams: it usually is more effective if one person handles the integration of the new feature and code that uses it. Because the feature user has the need, it may be him or her. However, review of code to be added to a base for which another team is responsible should, of course, rest with that team: "You code didn't have this which I needed, so here's the patch, wanna check it out?" Yes, this can cause political friction, but in a mature development team, that won't happen. When communication between teams is minimal, hostile, or non-existent, and such functionality is to be added, is when builds break, and regression tests fail -- and the finger pointing begins.

Re:Part of the problem is CVS

[etcshadow]

Well, obviously no source code control software is gonna compensate for developers who can't write good code, use common sense, and follow *simple* process. Granted, requiring complex process of your developers is asking for trouble, but you can't live without simple rules. Some simple rules really require being backed up by the capabilities of the software, though. An example that comes to mind: consistent atomic checkins.

It goes something like this: say you change a function signature. It is your responsibility to grep for all the uses of that function and change them. It is also your responsibility to check in all of those changes atomically. That is: an all-or-none checkin of a group of files all at once. That group is also bound together into the future (the relationship is not severed after checkin).

Another simple point of process that saves your ass is JOB TRACKING. If your source control repository doesn't link into a job tracking system, then I pitty you. I've been there, and it sucks. It took a while for us to work out exactly what was missing and how to get it... but now that we have implemented it, it makes life livable again. The idea with a job tracking system is: assosciate all of your changes with a job. If you want a bleeding edge revision, then sync to the head revision and don't be surprised when stuff breaks. If you want a rough-around-the-edges version to test against, then sync to the highest revision that is entered in the job tracker. Use the various life-cycle statuses in the job tracker to sync to various points. On the whole: Get all files in QA status, or all files in QATESTED status, or CODECHECK status (or however you choose to name these things in your job tracker), or whatever status you want.

In that way, you don't easily break the build, because before you even try to build off of something, you've tracked its code review and its unit testing. Of course, there are always the possibilities for unit testing problemss, but they are usually going to end up being the fault of a developer not following simple processes. In the example I used above where we changed the signature of a function, and updated all of the calls to that function in the same atomic change... you could have had another developer creating a *new* call to that function in their own working copy of the code. You would, of course never catch that, and they might not either... but hopefully whoever is doing code reviews has an eye open to things like function signature changes, and can catch it at that point.

Make it clear to your developers that changes not assosciated with a job will never see the light of day. Every night, review any untracked changes and email the developer, asking them what the hell.

It's true, there is no way to make everything 100% bullet-proof against checking in bad code. Of course, that's why we do things like freeze integration and test before a release.

Re:Doesn't work on Windows

[Khalidz0r]

Well, I think there is more work done on unix systems, and more programs are developed for unix systems than there is for windows systems.

The server side of computing is not really as narrow as you have explained it.

Another thing is, no, not everybody uses Windows, I do use it, but since we are here in slashdot, I'm telling you that most of the users around you use, and love, linux and apple systems, and use them as workstations as well (I have a linux work station although it's not my main pc too).

The idea that developing programs for windows is more important is wrong, I really don't like to spend my time developing a program to help Microsoft gain more power in the monopoly.

I am not much into software development for the time being, but if I get into that I'll always make sure to build programs for linux BEFORE Windows (or make portable solutions).

It's just that Unix is a nice OS for MANY things, not some things!

Request Tracker

[babbage]

This is what Jesse Vincent has been using for RT: Request Tracker development for several months now, rather than CVS. Apparently it's much nicer than CVS, but it's exotic and not many people know about it or how to submit patches with it, so RT3 from what I can tell is kind of a one man project at the moment. In any case though, I've heard nothing but good things about Aegis, and it seems like a tool worth checking out if you have a software project to manage.

(And for that matter, if you need to track software bugs & other issues, RT rocks. Don't bother with Bugzilla, it's not half as good as RT is for most of the same tasks. And no, no one is paying me to endorse RT or anything, it's just great software and, in reference to Aegis, I respect the judgement of the guy developing it...)

Re:Request Tracker

[phamlen]

At the risk of being modded Off-Topic, I will also state that RequestTracker (a system that's currently being built by Aegis) is a great tool for tracking software bugs, issue tracking, trouble-ticketing, and general to-do list issues.

It rocks, it's free, and it does virtually everything that you need it to do without complexity!!!

Pair check-in

[UberChuckie]

There is a rule for check-ins when my team is trying to stablize code; that is have another developer go through your changes in case you did something silly. It also serves as a mini code review. This includes making sure the code builds. :)

Re:Doesn't work on Windows

[Quixotic Raindrop]

Yeah, it's a troll. Still, I feel like munching on a little troll.

In fact, there are more installed UNIX servers in large scale operations than there are Windows servers (I work for a company that sells hundred-terabyte disk storage systems, to those exact operations, and more than 60% of our non-mainframe-using customers are using Solaris, AIX, and HP/UX, with Windows rolling in at about 35%).

On the whole, Windows is completely unsuited to enterprise-level programs and projects. It has a laughably low limit on the number of attached disk devices, as well as ludicrious limits on how big those disks can be. Sharing disks between clusters of Windows servers is tenuous, at best, and not recommended for high-risk environments.

Unix(es), on the other hand, grew up in the enterprise, and is quite well suited to that environment. Just as an example, I am aware of at least ten multi-national banks that operate only Unix in their transaction processing centers, which is one of the most demanding enterprise solutions available. Only Unix. No Windows in any of their datacenters.

The fact that there may, or may not, be a system like Aegis for Windows is irrelevant to your original message, which explicitly anointed truth to things which are wholly untrue.

Re:Used something like this

[bsmoor01]

Yeah, but what if your build takes over an hour? (Like where I work) Say you need something checked in, like now. You're sure it works, but you have to hang out for an hour to make sure it went in? There is a certain level of trust you need in a development environment. Waiting for it to completely rebuild simply isn't always an option.

Sure, you can generally do partial builds. But what happens when you just had to change that include file that somehow effects nearly everything else? You're stuck for that hour.

slashdot business model

[Anonymous Coward]

1) Post advertisements as news articles
2) ???????????????????
3) Profit!!!!!!!!!!!!!!!!!

Re:Used something like this

[ObviousGuy]

Yes, you're blocked from checking stuff in until the test build finishes, but even though you are SURE it works it's better to have a sanity check just in case.

Have some coffee, fill out the TPS report, look at pending bugs. There's a few things you could be doing during that hour.

Solution

[jsse]

...and they just went out for lunch ... Argh! Did you know there is a solution to this problem?

Go to lunch.

Re:Eclipse Plugin?

[aminorex]

Does your development environment include a command-line?

If so, then it's trivial to use another version control system.

Assuming that eclipse CVS support invokes the cvs
command for its operations, it would be trivial to
make a cvs-compatible commmand line for aegis.

Cookies

[IanBevan]

Well, as odd as this might sound, when anybody checks anything to our source control that breaks the build, they have to go out and buy the entire development team a coffee and/or chocolate cookies. This worked like a charm. It not only raises awareness and makes people more careful, it has actually increased moral in the team as we look forward to the weekly build to see who has cocked it up :-)

Krispy Kremes

[mpechner]

2-3 times a month we get Krispy Kremes. That is the penance for breaking a nightly build. Engineer or build meister. Screw it up and bring in the donuts.

Another one..

[droyad]

The base ball bat. At my work after we beat the first few developers to a pulp after they checked in broken files, we found that the rate of broken files decreased dramatically

Actually, that's not far off what CVS does

[devphil]

run the code through a lameness filter before allowing the check-in

Okay, I agree with you that the /. editors were on crack when they made that call. But the idea of "filter before checkin" has been in CVS for a long, long time.

Basically, there's a file in CVSROOT that can call an arbitrary program. If that program succeeds, checkin proceeds. If it doesn't, it doesn't. :-)

I was very surprised to see this article treat such an idea as something new.