Notes From the Cathedral

A reader wrote to us about "Notes From The Cathedral", which is sort of a tell-all from a former developer of a "major software company". An interesting, if somewhat sad, read.

Comments

Govt does code reviews

[Mark F. Komarinski]

For all the hollering about the US Govt, the Dept. of VA (worked for a few years for them) did things the "right" way:

1) Big 'ol binder that had coding style, approved APIs, things like that
2) All the functions must declare inputs, outputs, and globals used before you even start coding. These declarations also make it into the code.
3) That pseudocode goes through a peer review with coders from other facilities.
4) Once the code is written, it goes through another peer review, and there are usually a few rewrites after that review.

The reason for much of this is *this* is the code that runs 170+ hospitals around the country. Don't want rogue code screwing up the systems.

The upshot out of all this is that the code is extremely easy to update, easy to debug, and the support people have an easier job.

The value of process

[staplin]

I agree with you totally. I started out working for a large well established government contracting company. The whole system was slow and process bound. Code reviews were project milestones, as were such things as functional specs, high level designs, detailed designs, and test plans. The paperwork was occasionally a pain in the ass that kept you from doing any coding.

Now I'm working for a small, relatively young company, just starting to ramp up from when it was just a coder or two. I was looking for change, and I found it. Now, code reviews are few and far between, the customer may see a design spec (if they are technically saavy), and test plans are usually incomplete or so vague you could pass them with "HelloWorld" if you squint.

And I find myself missing a lot of the benefits of having a real process, despite the increased paperwork. As we ramp up, we're trying to instill the same kind of discipline that we've seen work at other companies, without sacrificing that ever so important "internet-time" project scope. It may seem to save time to drop all that overhead, but in the long run, the code you write is higher quality, more maintainable, and has fewer bugs than the stuff you crank out blindly...

I read "The Mythical Man-Month" last spring, and was shocked at how much of it was almost directly applicable to many of the small companies I've seen. Too much work for too few people, poorly documented and only margionally tested.

But to get back to the topic, OSS isn't always the solution. Many of the projects I've worked on couldn't be open sourced... too many proprietary issues, government content, or built to work with expensive, proprietary, third-party systems almost no OSS developer would have experience with, let alone have a personal copy of to test the software.

And when OSS won't work for you, the important thing to do is identify the pieces that OSS has that you don't, like lots of peer review, and institute a way to make use of those things.

Re:9 to 5

[Malcontent]

If your life belongs to you that's one thing. If you are married and have children you have a responsibility to them that is much more then the responsibility you have to your boss. In the long run you gain nothing by cheating your children out of a father and cheating your wife out of a husband. When you are old or sick your boss is not going to give damn about you.

IMHO if you bought a child into this world it better be your top priority in life to make sure it gets raised properly, responsibly, and lovingly. Anything else has got to be pretty far down on your list of things to do.

A Dick and a Bush .. You know somebody's gonna get screwed.

There are no 9-to5ers

[DunkPonch]

"There are no 9-to-5ers in the Open Source Community."

Well, there are if you mean 9 p.m. to 5 a.m. :)

Commercial vs. OSS

[mickwd]

I've always felt that a good indicator of whether a (commercial) project will succeed or not is the ratio of talented techies to technical also-rans. (Of course, there are many other factors as well, such as realistic goals and timescales, and good management).

A project will usually succeed if there are enough skillful hackers to "carry" the rest of the team. Why are there "also-rans" writing commercial software ? Well, it's a job, and often highly-paid too. Often people's circumstances change - keen hackers can lose the urge if other things take over in their lives - having kids, for example, or other personal issues. And bad management can be a hell of a demotivation. Also, let's face it, there are plenty of bullsh*tters in our industry.

But why is OSS better ? Because there are no (or at least much fewer also-rans). Many (most?) people write OSS software because it's enjoyable. People who enjoy coding are usually good at it. Are they good at it because they enjoy it, or do they enjoy it because they're good at it ? I don't know. But OSS, almost by its very nature, has attracted, and will continue to attract very good technical brains.

-

Linux - the Unix defragmentation tool.

notes from my cathedral

[ethereal]

I work for probably the most process-heavy, Dilbert-inspired corporate monolith around. We do have an intricate process involving frequent peer reviews of code, documentation, and architecture, combined with gate-keeping by senior designers who determine what is good enough to go into the final product. And y'know what, correct application of the correct process really does make our lives as software engineers easier. Reviews of requirements and architecture up front reduce the amount of re-coding we have to do at coding time, so much so that our time breakdown is something like 40% up-front documentation (including test cases), only 10-20% coding, and we have the rest of the time left to exhaustively test the thing. Sure, things aren't always so rosy for every single project and feature, but overall my impression is that you can get a lot of gain out of your process if you invest in it up front.

Some of the gains aren't evident right away, but full test case suites, requirements, and architecture documents make your results more maintainable. We still have 10 year old code in our product; because it was documented, we know that it still works, and we have tests to prove it. We're not in a situation of "don't change this magic code or the product will never work again".

The most important thing, though, is management support. It's true, we could turn around our product three times as fast without all of the process overhead, and I think sometimes management is tempted to go that way. But without documentation, etc., we wouldn't be able to keep up that pace for 10-15 years on a single product. Our release dates would gradually slip more and more, as the combination of all of our earlier shortcuts came back to hurt us. I see this happening to a lot of PC software companies (I'm in embedded systems) and it's painful to watch.

You really have to have a corporate-wide committment to software quality, up-front design and documentation, and a repeatable process, but if you're really in the software business to stay, I think it's worth it. SW engineering process is the stuff that was both tremendously boring and incredibly obvious in school, but it does work well if you take the time to use it.

Re:Ahhh, Someone help me!!@#!

[ethereal]

Some random thoughts:

• You need management support pronto, because what you have to do is get enough time to understand what you already have done, look for big architectural flaws that will force you to work around them for the rest of your life, and think about where you're going. If you don't control your schedule (and it sounds like you don't) you have to convince the folks who do that there is a benefit to proper planning.
• If this is more than a one-off project, if somebody will have to maintain it, you should try to get some more time to document your architecture and design decisions and write some tests. You can get tools that make generating test & test drivers a lot easier. You can save time now by not documenting and not producing test cases, but you will lose two or three times that much time in the long run. If you know there will never be any maintenance on your product, then you can't make as good of an argument for these things.
• Tailor your process. You don't have to do line-by-line Fagan reviews of your code if the extra attention to detail isn't finding more bugs, for example.
• Contrariwise, if you find that you're using code to paper over serious architecture flaws, then you know to have more complete architecture reviews next time. But your time in peer reviews has to be well spent - if you're on a deadline, have everyone review ahead of time and just hit the biggest issues at a meeting, with minor issues discussed over email. If you make peer reviews a hell to conduct and attend, you won't gain anything from them. If you make reviews easy to attend, you will generally get good comments. Invite subject matter experts if you have 'em; even if they aren't on your exact project (or sometimes because they aren't) they often bring up vital issues that you'd never think of.
• Document how you've tailored things, so that everyone knows what steps they have to take their code through to get it in. Enforce some sort of policy so that there is at least one peer review of everyone's code before it goes into the product, because once it's in you may never come back to look at it until it's too late to fix it.
• You really need management support or ownership of your schedule in order to make time to do these things. Once you have time to do these things you will find yourself worrying less about bits of code and more about architectural issues up front. This is good because anybody can write code, but not everybody can take the larger view of the product as a software system and pinpoint the weak links or missed requirements in your plans.
• Marketing owning the user interface isn't necessarily a bad thing. What's bad is if they are allowed to dictate your schedule so that you end up using poor design or bad code to make up the time. Again, you need management support so that you can tell marketing there will be a cost for any new/changed requirements, and the cost will include time to fully document those requirements (including peer review if possible), time to implement them, and time to test them. Marketing isn't automatically bad, they just don't have any SW engineering experience to know what is possible and what the costs of things are. You have to reiterate this to them until they realize that their actions have consequences. It's sort of like training a puppy.
• If you're spending most of your time coding, trying to get it to compile, or debugging, then you didn't spend enough time up front. Aim to spend most of your time testing, because you'll need it. The worse your development process, the more test time you'll need unless you want to compromise product quality. This is another good topic to bring up to management.
• Adding staff to a late project makes it later - but not always. If you divide up responsibilities, you can have new staff do coding from your design documents while the more experienced staff write test cases or look into design/architecture problems. It sounds like your project might be in that sort of situation - you can get coders fairly cheaply (in terms of time, although maybe not in terms of salary) but people who understand the SW system as a whole and are conversant with the design philosophy have better things to do than generate code or test.
• If you have really good requirements and/or test cases, you can hand off testing to an outside group entirely, but that requires more time up front and more budget. It is a good way to find a lot of bugs quickly, though.

• Wow, that got long. Hopefully some of it was useful.

Re:9 to 5

[NaughtyEddie]

Anyone heard of XP (eXtreme Progamming)? See e.g. "eXtreme Programming eXplained" by Kent Beck. One of XP's ten "rules" are "only work 9 to 5".

I acknowledge many of the problems this guy is talking about, and XP is a promising way of making this work in the real-world of software schedules. Also, some organizations (e.g. Data Connection Ltd, a former employer of mine) have amazing setups. Code review is mandatory - they don't try and tart it up as "objectives" either, it's part of the discipline of working professionally. It confounds me how many paid programmers don't even understand the meaning of the words "professional" and "discipline"!

Incidentally, I don't hold with his notion that a thousand developers adding features willy-nilly is better than a tight team with a coherent design philosophy, but - hey - you can hide all that crap behind a user interface and no-one's any the wiser. Hmm. Didn't Microsoft try something similar?

[The opinions stated in this article are not endorsed in any way by Data Connection Ltd. Just thought I'd make that clear!]

Fixing Someone Else's Broken Code; Resources

[goingware]

I've been working as a programmer for 13 years now, and for most of my career, the way I described it is this:

I've spent most of my career fixing somebody else's broken code.

This is not to say that my own has always been of the very highest quality, but in fact I decided early on to try to come to a fundamental understanding of what was wrong with software development, to get very good at debugging (I say that debugging is a specialty on my homepage) and to learn to write better code.

In my early years I was initially very shocked at what I'd discover in production use at companies. Over the years I just learned that that's standard practice, in commercial software, in-house software, and even in scientific software (where I have become convinced, because of my experiences with high-energy physics data analysis, that many scientific papers are published with erroneous measurements because of software bugs).

Early on I read that something like 90% of software development is spent doing maintenance programming. Some of this is doing legitimate upgrading, but a lot of it is just spent fixing bugs, and even a lot of time spent doing upgrades would be more productive if the code were of better quality.

After reading this figure and having so many experiences with software bugs, both other people's and my own, I decided very early on to get very good at debugging.

One of the first things I did was adopt the regular use of "lint" for checking my C code. I would integrate lint targets into my makefiles and after editing a source file I would type "make lint" before compiling to objects and lint would check all the files that were out of date with the object modules. Pretty quickly I got to where I could write code that was nearly always lint-clean - but the existing code I worked on would make lint scream with hundreds if not thousands of complaints, often serious things like variables being used before they are initialized.

One of the first solid clues I got about software quality came from Robert Ward's book "Debugging C" - now out of print, it predated the common use of source code debuggers and talked about how to write your own stack crawls and other tools.

Ward emphasized the use of the Scientific Method in debugging, and because I was trained in physics, this came very naturally to me; before that I'd mostly floundered and used printf a lot.

I've gotten very good at debugging and have even worked full-time as a debugger at Apple Computer where I was a "Debug Meister" and my business card gave my title as "Cybernetic Entomologist".

I can easily get highly paid consulting work doing debugging for companies desperate to ship a product (and have in the past) but I don't really like to do it for various reasons, some of the same reasons I quit my debugging job at Apple.

One is that if I only do debugging I don't have something to point to at the end of the day and say "I wrote that". I could say "that works because of me", but sadly there's usually lots of bugs left that I didn't have the time to find so I don't really feel proud of the result. The other problem is that the bugs are usually not there because of something interesting, it's not like the code is mostly good but there's some subtle flaws, rather the code is a heap of dung and I can go in with a pitchfork and do debugging wholesale until I get tired of it and the client or manager decides the rate new bugs are being found is low enough they can feel OK about shipping it.

I don't feel good about contributing to such shoddiness. If a company is not good enough to support quality in their corporate culture I don't want to come in and put on a band-aid for them. It would be an entirely different thing if a company hired me to restructure their development process so that quality was a goal that was achieved through direct application of process but gee whiz no one has ever asked me to do that for them.

I do have to say though that the best thing that ever happened to me is that I became a "technology prostitute" as the author of the original article puts it. One benefit of this is that the process is entirely of my own creation, and almost all of the work I've been given has been to write entirely new products from scratch, so I can engineer in quality from the beginning.

Here's a few recommendations I have. Get good tools. Besides a compiler, editor and debugger, you need a static code checker and you also need dynamic testers. The ones I know about are (I haven't used them all yet):

• PC-Lint static code checking for C and C++. It runs on Windows but Flexe-Lint comes as shrouded source code and is highly portable.
• Spotlight dynamic tester for Mac PowerPC - I use this every day and recommend it highly
• BoundsChecker dynamic tester for Windows
• Purify dynamic tester for Unix (but apparently not Linux) and Windows NT
• Optimizeit dynamic tester for Java - do you know many Java programs have memory leaks? Can you understand why? Not just Java but any garbage collected program.

There's also memprof and electric fence, which I think run on Linux. It would be possible to modify gcc so that you program when built with debugging flags and linked with a special library would have all memory references boundschecked before allowing the access. This could work with both read and write access and would catch overruns by as little as one byte - Spotlight does this by patching the executable powerpc code and rewriting the debugging symbol table.

Finally, to really come to understand the software quality problem in the industry and what you can do about it, read The Forum on Risks to the Public in Computers and Related Systems also available on the Usenet News as comp.risks. The book The Software Conspiracy exposes the complete disregard the commercial software industry has for serving the consumer by providing quality products - I haven't read it yet but it looks interesting.

A very interesting methodology that emphasizes personal responsibility and puts the fun back into programming as well as maintaining quality from the very start is Extreme Programming. I'm starting to adopt extreme programming (the the extent a one man operation can - I can't work in pairs :-/ ) and find it a tremendous benefit.

Uh Oh

[dmccarty]

If Open Source was 1000 monkeys typing randomly in diff files, I agree that it would be doomed to fail.

Oh no! We're doomed!
--

Coding in the "real world"

[sfbanutt]

I've been a professional software developer for nearly 20 years now and everything he writes rings very true. It's actually worse than that in many cases. I can count on the fingers of one hand the number of times I've actually been given a specification for a project, or even a reasonable list of requirements. And the established deadlines are typically even worse, you end up doubling or even tripling your estimates for project length simply because you know that whatever time you ask for will be cut in half (or more). Code reviews are something you read about in software engineering texts, if you actually know someone who is able to do them, you feel privileged. It's no wonder that almost all software projects are over budget and late, the budget and timeframe were unrealistic to begin with and the requirements and specifications never existed!

9 to 5

[Anonymous Coward]

The IBM group which writes code for the space shuttle works 9am to 5pm almost exclusively, and they produce very high quality code. There's nothing wrong with coders who work 9-5, theres nothing wrong with coders who have lives outside the office. Sometimes, they're even more productive and predictable than those who put in, and brag about, their 36 hour coding stretches.

Re:9 to 5

[remande]

There are at least two types of code out there: money-critical (failures cost money) and life-critical (failures cost lives). The Shuttle code lives clearly in the second category; most of us are programming in the first category.

Building life-critical software is fundamentally different from making money-critical software. This is because, for life-critical software, people will trade off time-to-market and feature base to get quality (absence of bugs).

Open Source doesn't even provide this. Think about it. If you needed a pacemaker (life-critical), would you want it to run on Linux? I wouldn't. Linux has uptimes of over a year; I need uptimes measured in decades. Frankly, no top 10 OS is qualified for life-critical stuff. I'll trust Linux with my data, even my money, but not with my pulse, my shuttle, or my airliner avionics.

When you build life-critical software, you do it anal-retentively, line-by-line, and work at a snail's pace. The result is something that does one thing (or a few things) very well under incredible circumstances.

Try working that way in the commercial world of money-critical software, and you will be beat to market every time. And not by a little. I bet that it would take decades to build a word processor to compete with today's stuff, if it had to have life-critical reliability.

The code is different, the process is different, the people are different.

Re:9 to 5

[Alternity]

Agreed. What is wrong IMHO is a coder who doesn't care about the code he has to produce for his company. It happened to me a few times that I was working on something I found so interesting that I did not care to stay a few extra hours at work. What is wrong is to see a programmer that spend his whole day looking at code only wishing the day could end so he can stop looking at that aweful code he has to work on.

Programmers working from 9 to 5 each day? I have no problems with that...
Programmers who cannot wait 5 to run away from the code they're working on? Now there's a problem... Those need to realize that they're are lots of other jobs with projects they might like more out there.

Of course everything up there was IMHO

Other downsides of the Cathedral...

[Alternity]

In the cathedral Engineering decision are often made by people who have no idea how to develop software, do not care how to develop software and would by no mean consider it appropriate to consult the coders before making decision.

Thus you end with administrative request about a near impossible feature to be implemented in an astoundingly short delay withouth enough resources. That added with the fact that a lot of programmers in the Cathedral are only waiting 5pm to go back home (and hack in code they really like??) forces people to cut around the corners and take shortcuts. After all the holy priests of the cathedrals do not care if the program is working fine... all that matters is if the feature is implemented and if the software has enough eye-candy to impress other priests in other cathedrals who will pay for our stuff.

Of course it might not be that way in all cathedrals... but the more I move from catehdral to cathedral the more I become atheist :o)

Cathedral has one good thing going for it: Focus

[dmccarty]

The Cathedral does have a very good ingredient in its mixture that OSS programmers don't have, and that is focus. Corporations have the ability to assign teams of 50, 100, 300 programmers to an enormous project and finish it in 6 months to 1 year, something that would be daunting to the Linux community (look how long the 2.4 kernel update has taken).

Now I know that people will want to jump in and say "But look at all the bugs! Look at the poor quality! 2.4 will be released as better software because of the delay!" but the fact remains that the Cathedral gets products out the door while most OSSers live in the near future. "Just wait till the next version," we say, "when it will have [X] feature."

To a large degree, when we buy a piece of software, like Word 2.0, we don't just buy Word 2.0. We also buy into the idea that we bought a piece of software that works decently now, and the creator is committed to eventually upgrading the software until it's the word processor to end all word processors. And so we're at Word 9 today, and the cycle continues. When software is looked at this way, getting a product out the door becomes more important than waiting for people to finish it in their free time. That's why Cathedrals (read: corporations) will make money and that's why they'll stay in business.

I'll add two caveats to the above:
1: This doesn't necessarily apply to security products. Thank God Microsoft didn't introduce us to sockets.
2: Another reason why the Cathedral will always exist is because of competition. If you have highly sensitive secrets it doesn't make sense to publish your source. Security through obscurity might not be a good thing, but that doesn't mean I go around giving everyone copies of my door key and daring them to find out where I live.
--