Notes From the Cathedral

A reader wrote to us about "Notes From The Cathedral", which is sort of a tell-all from a former developer of a "major software company". An interesting, if somewhat sad, read.

Comments

Govt does code reviews

[Mark F. Komarinski]

For all the hollering about the US Govt, the Dept. of VA (worked for a few years for them) did things the "right" way:

1) Big 'ol binder that had coding style, approved APIs, things like that
2) All the functions must declare inputs, outputs, and globals used before you even start coding. These declarations also make it into the code.
3) That pseudocode goes through a peer review with coders from other facilities.
4) Once the code is written, it goes through another peer review, and there are usually a few rewrites after that review.

The reason for much of this is *this* is the code that runs 170+ hospitals around the country. Don't want rogue code screwing up the systems.

The upshot out of all this is that the code is extremely easy to update, easy to debug, and the support people have an easier job.

Re:9 to 5

[Malcontent]

If your life belongs to you that's one thing. If you are married and have children you have a responsibility to them that is much more then the responsibility you have to your boss. In the long run you gain nothing by cheating your children out of a father and cheating your wife out of a husband. When you are old or sick your boss is not going to give damn about you.

IMHO if you bought a child into this world it better be your top priority in life to make sure it gets raised properly, responsibly, and lovingly. Anything else has got to be pretty far down on your list of things to do.

A Dick and a Bush .. You know somebody's gonna get screwed.

notes from my cathedral

[ethereal]

I work for probably the most process-heavy, Dilbert-inspired corporate monolith around. We do have an intricate process involving frequent peer reviews of code, documentation, and architecture, combined with gate-keeping by senior designers who determine what is good enough to go into the final product. And y'know what, correct application of the correct process really does make our lives as software engineers easier. Reviews of requirements and architecture up front reduce the amount of re-coding we have to do at coding time, so much so that our time breakdown is something like 40% up-front documentation (including test cases), only 10-20% coding, and we have the rest of the time left to exhaustively test the thing. Sure, things aren't always so rosy for every single project and feature, but overall my impression is that you can get a lot of gain out of your process if you invest in it up front.

Some of the gains aren't evident right away, but full test case suites, requirements, and architecture documents make your results more maintainable. We still have 10 year old code in our product; because it was documented, we know that it still works, and we have tests to prove it. We're not in a situation of "don't change this magic code or the product will never work again".

The most important thing, though, is management support. It's true, we could turn around our product three times as fast without all of the process overhead, and I think sometimes management is tempted to go that way. But without documentation, etc., we wouldn't be able to keep up that pace for 10-15 years on a single product. Our release dates would gradually slip more and more, as the combination of all of our earlier shortcuts came back to hurt us. I see this happening to a lot of PC software companies (I'm in embedded systems) and it's painful to watch.

You really have to have a corporate-wide committment to software quality, up-front design and documentation, and a repeatable process, but if you're really in the software business to stay, I think it's worth it. SW engineering process is the stuff that was both tremendously boring and incredibly obvious in school, but it does work well if you take the time to use it.

Re:Ahhh, Someone help me!!@#!

[ethereal]

Some random thoughts:

• You need management support pronto, because what you have to do is get enough time to understand what you already have done, look for big architectural flaws that will force you to work around them for the rest of your life, and think about where you're going. If you don't control your schedule (and it sounds like you don't) you have to convince the folks who do that there is a benefit to proper planning.
• If this is more than a one-off project, if somebody will have to maintain it, you should try to get some more time to document your architecture and design decisions and write some tests. You can get tools that make generating test & test drivers a lot easier. You can save time now by not documenting and not producing test cases, but you will lose two or three times that much time in the long run. If you know there will never be any maintenance on your product, then you can't make as good of an argument for these things.
• Tailor your process. You don't have to do line-by-line Fagan reviews of your code if the extra attention to detail isn't finding more bugs, for example.
• Contrariwise, if you find that you're using code to paper over serious architecture flaws, then you know to have more complete architecture reviews next time. But your time in peer reviews has to be well spent - if you're on a deadline, have everyone review ahead of time and just hit the biggest issues at a meeting, with minor issues discussed over email. If you make peer reviews a hell to conduct and attend, you won't gain anything from them. If you make reviews easy to attend, you will generally get good comments. Invite subject matter experts if you have 'em; even if they aren't on your exact project (or sometimes because they aren't) they often bring up vital issues that you'd never think of.
• Document how you've tailored things, so that everyone knows what steps they have to take their code through to get it in. Enforce some sort of policy so that there is at least one peer review of everyone's code before it goes into the product, because once it's in you may never come back to look at it until it's too late to fix it.
• You really need management support or ownership of your schedule in order to make time to do these things. Once you have time to do these things you will find yourself worrying less about bits of code and more about architectural issues up front. This is good because anybody can write code, but not everybody can take the larger view of the product as a software system and pinpoint the weak links or missed requirements in your plans.
• Marketing owning the user interface isn't necessarily a bad thing. What's bad is if they are allowed to dictate your schedule so that you end up using poor design or bad code to make up the time. Again, you need management support so that you can tell marketing there will be a cost for any new/changed requirements, and the cost will include time to fully document those requirements (including peer review if possible), time to implement them, and time to test them. Marketing isn't automatically bad, they just don't have any SW engineering experience to know what is possible and what the costs of things are. You have to reiterate this to them until they realize that their actions have consequences. It's sort of like training a puppy.
• If you're spending most of your time coding, trying to get it to compile, or debugging, then you didn't spend enough time up front. Aim to spend most of your time testing, because you'll need it. The worse your development process, the more test time you'll need unless you want to compromise product quality. This is another good topic to bring up to management.
• Adding staff to a late project makes it later - but not always. If you divide up responsibilities, you can have new staff do coding from your design documents while the more experienced staff write test cases or look into design/architecture problems. It sounds like your project might be in that sort of situation - you can get coders fairly cheaply (in terms of time, although maybe not in terms of salary) but people who understand the SW system as a whole and are conversant with the design philosophy have better things to do than generate code or test.
• If you have really good requirements and/or test cases, you can hand off testing to an outside group entirely, but that requires more time up front and more budget. It is a good way to find a lot of bugs quickly, though.

• Wow, that got long. Hopefully some of it was useful.

Re:9 to 5

[NaughtyEddie]

Anyone heard of XP (eXtreme Progamming)? See e.g. "eXtreme Programming eXplained" by Kent Beck. One of XP's ten "rules" are "only work 9 to 5".

I acknowledge many of the problems this guy is talking about, and XP is a promising way of making this work in the real-world of software schedules. Also, some organizations (e.g. Data Connection Ltd, a former employer of mine) have amazing setups. Code review is mandatory - they don't try and tart it up as "objectives" either, it's part of the discipline of working professionally. It confounds me how many paid programmers don't even understand the meaning of the words "professional" and "discipline"!

Incidentally, I don't hold with his notion that a thousand developers adding features willy-nilly is better than a tight team with a coherent design philosophy, but - hey - you can hide all that crap behind a user interface and no-one's any the wiser. Hmm. Didn't Microsoft try something similar?

[The opinions stated in this article are not endorsed in any way by Data Connection Ltd. Just thought I'd make that clear!]

Coding in the "real world"

[sfbanutt]

I've been a professional software developer for nearly 20 years now and everything he writes rings very true. It's actually worse than that in many cases. I can count on the fingers of one hand the number of times I've actually been given a specification for a project, or even a reasonable list of requirements. And the established deadlines are typically even worse, you end up doubling or even tripling your estimates for project length simply because you know that whatever time you ask for will be cut in half (or more). Code reviews are something you read about in software engineering texts, if you actually know someone who is able to do them, you feel privileged. It's no wonder that almost all software projects are over budget and late, the budget and timeframe were unrealistic to begin with and the requirements and specifications never existed!

9 to 5

[Anonymous Coward]

The IBM group which writes code for the space shuttle works 9am to 5pm almost exclusively, and they produce very high quality code. There's nothing wrong with coders who work 9-5, theres nothing wrong with coders who have lives outside the office. Sometimes, they're even more productive and predictable than those who put in, and brag about, their 36 hour coding stretches.

Other downsides of the Cathedral...

[Alternity]

In the cathedral Engineering decision are often made by people who have no idea how to develop software, do not care how to develop software and would by no mean consider it appropriate to consult the coders before making decision.

Thus you end with administrative request about a near impossible feature to be implemented in an astoundingly short delay withouth enough resources. That added with the fact that a lot of programmers in the Cathedral are only waiting 5pm to go back home (and hack in code they really like??) forces people to cut around the corners and take shortcuts. After all the holy priests of the cathedrals do not care if the program is working fine... all that matters is if the feature is implemented and if the software has enough eye-candy to impress other priests in other cathedrals who will pay for our stuff.

Of course it might not be that way in all cathedrals... but the more I move from catehdral to cathedral the more I become atheist :o)

Cathedral has one good thing going for it: Focus

[dmccarty]

The Cathedral does have a very good ingredient in its mixture that OSS programmers don't have, and that is focus. Corporations have the ability to assign teams of 50, 100, 300 programmers to an enormous project and finish it in 6 months to 1 year, something that would be daunting to the Linux community (look how long the 2.4 kernel update has taken).

Now I know that people will want to jump in and say "But look at all the bugs! Look at the poor quality! 2.4 will be released as better software because of the delay!" but the fact remains that the Cathedral gets products out the door while most OSSers live in the near future. "Just wait till the next version," we say, "when it will have [X] feature."

To a large degree, when we buy a piece of software, like Word 2.0, we don't just buy Word 2.0. We also buy into the idea that we bought a piece of software that works decently now, and the creator is committed to eventually upgrading the software until it's the word processor to end all word processors. And so we're at Word 9 today, and the cycle continues. When software is looked at this way, getting a product out the door becomes more important than waiting for people to finish it in their free time. That's why Cathedrals (read: corporations) will make money and that's why they'll stay in business.

I'll add two caveats to the above:
1: This doesn't necessarily apply to security products. Thank God Microsoft didn't introduce us to sockets.
2: Another reason why the Cathedral will always exist is because of competition. If you have highly sensitive secrets it doesn't make sense to publish your source. Security through obscurity might not be a good thing, but that doesn't mean I go around giving everyone copies of my door key and daring them to find out where I live.
--