Slashdot Mirror
Notes From the Cathedral
A reader wrote to us about "Notes From The Cathedral", which is sort of a tell-all from a former developer of a "major software company". An interesting, if somewhat sad, read.
← Back to Stories (view on slashdot.org)
For all the hollering about the US Govt, the Dept. of VA (worked for a few years for them) did things the "right" way:
1) Big 'ol binder that had coding style, approved APIs, things like that
2) All the functions must declare inputs, outputs, and globals used before you even start coding. These declarations also make it into the code.
3) That pseudocode goes through a peer review with coders from other facilities.
4) Once the code is written, it goes through another peer review, and there are usually a few rewrites after that review.
The reason for much of this is *this* is the code that runs 170+ hospitals around the country. Don't want rogue code screwing up the systems.
The upshot out of all this is that the code is extremely easy to update, easy to debug, and the support people have an easier job.
-- Ever notice that fast-burning fuse looks exactly the same as slow-burning fuse? I didn't... (Edgar Montrose)
Sorry, no.
That might be a (loose) specification, it might be a partial prototype. If that's what you'd think of as a design then I'd rather not work with you if that's OK.
Design needs to be many things which that simply cannot be. It needs to go over the requirements that have been established and properly sort out what needs to be done to sort these out. The structure needs working out, the interfaces need specification. If all you have is a vague description of the issue and a code fragment you really can't expect to sort out the design with any serious chance of success. After a while of designing the system that way you can expect to hit problems as what is a fundamental feature of one component's implementation may become a block to another working. Whereas if it's been properly designed and analysed the chance of this happening is substantially lower.
The other side is that, when it comes to maintaining the software, your approach to design means there's no concrete explanation of exactly why something does what it does - or even what's there. By forcing the maintainers to work out the code with no substantial, clear documentation to explain what's going on, you end up with a much larger effort and a lower probability of success as the maintainers aren't necessarily going to pick up on the subtelties. After all, look back at code you wrote a while back. How well can you understand it at first reading, undocumented? And you wrote it.
You _really_ need more documentation and design than that if you're going to do a seriously good job, long-term.
Greg
(Inside a nuclear plant)
Aaaarrrggh! Run! The canary has mutated!
I know it isn't politically correct, but I can't help but regard that article as so much tripe. With all the "Rah, rah, we're right and you're not" nonsense you hear from Open Source advocates, you'd expect the wins to be much larger and more obvious. The Linux kernel is a nice piece of work. Ditto for Perl, Python, and Apache. Other free software is serviceable but of questionable overall quality and design, including gcc and others that I won't mention for fear of starting tangential flame wars. And there are lots and lots of Open Source projects that are complete and utter garbage, handily beat out by many commercial offerings. The evidence isn't completely in favor of either side.
So why hasn't Open Source shown itself to be an across the board win? I can think of a few reasons:
1. While some figures, most notably Linus Torvalds, have proven themselves to be software engineers of the highest caliber, there's a decided lack of experience among open source programmers. There's much enthusiasm, yes, but there's a pervasive reliance in staying up all night and writing lots and lots of code to fix problems. As such, there's just as much code and feature bloat in the open source world as in this so-called cathedral, including the lack of reliability that comes with such practices. It's not like thousands of people are wanting to sift through voluminous and poorly written code to fix bugs.
2. The "scratch an itch" philosophy applied to programming tools, but it doesn't seem to be applying to other applications. There's a definite "We've gotta out do Microsoft!" mantra, resulting in people working on UIs who know nothing about UI design, and students working on applications without understanding the intended user base.
3. For reasons that still aren't clear, there's a startling lack of creativity in the open source world. There's a lot of copying existing commercial software, but when that isn't the goal there's much floundering about. Look at the attempts to write free games. You certainly could write a game that isn't Tetris, Asteroids, Missile Command, Arkanoid, Tron lightcyles, etc., but there's no desire. The desire is to have an engineering problem to work on, not to create.
Computers are gadgets that require specific instructions to reach specific objectives.
People who know how to give specific instructions and objectives often become programmers, because they are well-suited to the work.
Non-programmers are often people incapable of expressing ideas with the mathematical accuracy that computers require. That's why they hire programmers.
If you are a programmer working directly under a non-programmer boss, it is your job to translate between their fuzzy ideas and the computer's concrete logic.
Programmers who can make this difficult transition are the ones who get paid huge salaries to be project leaders.
Programmers who can't do it are probably better off working for one of those project leaders, so they will be safely isolated from the sort of people who picked on them in Junior High School.
Information wants to be anthropomorphized.
I'm sorry, but come on. The way that is worded is so silly.. "You don't care what operating system you program for???".. I thought it was called being a versatile programmer, not a whore. It's only being a prostitue if you maintain a really sad, religious devotion to your OS of choice.
And noone tell me "it was a joke", becuase if you read the article, it isn't. That thing isn't funny at all. This guy is the definition of an overzealous open source nut.
sig:
sig:
See the "..for smart people" banners Wired runs here? Look elsewhere guys.
Code reviews can only be a waste of time, if the staff are anal. You have to be willing to give up some privacy to put the software ahead of personal interests. Currently the number one enemy of open source is self importance. Look at your company, and you'll see why! It's not security... it's the damn fools who are afraid to stand up and be recognized for their controbutions, or the guys who are afraid the rest of the staff are copying.
This does not apply in a rich social environment.
If you are lucky enough to find yourself working for a company that permits open source, then you may find yourself walking a short plank to corporate security repromands. Be sure you know the limitations of what can and can not be made public, and where such data can be published, and get it in writing from your boss.
There's more. What the open source movement means to me, is a freedom from reinventing the wheel, with knowledge as a tradeoff. No matter how plain your source is, someone in some place can use it to make themselves look good by copying your style.
Take the open source movement a step further; involve Napster and we have a real use for the technology that has every record label peeved.
Sorcery -- a program for finding source code on the internet.
Spread the love!
I agree with you totally. I started out working for a large well established government contracting company. The whole system was slow and process bound. Code reviews were project milestones, as were such things as functional specs, high level designs, detailed designs, and test plans. The paperwork was occasionally a pain in the ass that kept you from doing any coding.
Now I'm working for a small, relatively young company, just starting to ramp up from when it was just a coder or two. I was looking for change, and I found it. Now, code reviews are few and far between, the customer may see a design spec (if they are technically saavy), and test plans are usually incomplete or so vague you could pass them with "HelloWorld" if you squint.
And I find myself missing a lot of the benefits of having a real process, despite the increased paperwork. As we ramp up, we're trying to instill the same kind of discipline that we've seen work at other companies, without sacrificing that ever so important "internet-time" project scope. It may seem to save time to drop all that overhead, but in the long run, the code you write is higher quality, more maintainable, and has fewer bugs than the stuff you crank out blindly...
I read "The Mythical Man-Month" last spring, and was shocked at how much of it was almost directly applicable to many of the small companies I've seen. Too much work for too few people, poorly documented and only margionally tested.
But to get back to the topic, OSS isn't always the solution. Many of the projects I've worked on couldn't be open sourced... too many proprietary issues, government content, or built to work with expensive, proprietary, third-party systems almost no OSS developer would have experience with, let alone have a personal copy of to test the software.
And when OSS won't work for you, the important thing to do is identify the pieces that OSS has that you don't, like lots of peer review, and institute a way to make use of those things.
IMHO if you bought a child into this world it better be your top priority in life to make sure it gets raised properly, responsibly, and lovingly. Anything else has got to be pretty far down on your list of things to do.
A Dick and a Bush .. You know somebody's gonna get screwed.
War is necrophilia.
"There are no 9-to-5ers in the Open Source Community."
:)
Well, there are if you mean 9 p.m. to 5 a.m.
The real DunkPonch is user 215121. Everyone else is Bruce Perens.
like this originally ended with "For these reason, I QUIT!"
A project will usually succeed if there are enough skillful hackers to "carry" the rest of the team. Why are there "also-rans" writing commercial software ? Well, it's a job, and often highly-paid too. Often people's circumstances change - keen hackers can lose the urge if other things take over in their lives - having kids, for example, or other personal issues. And bad management can be a hell of a demotivation. Also, let's face it, there are plenty of bullsh*tters in our industry.
But why is OSS better ? Because there are no (or at least much fewer also-rans). Many (most?) people write OSS software because it's enjoyable. People who enjoy coding are usually good at it. Are they good at it because they enjoy it, or do they enjoy it because they're good at it ? I don't know. But OSS, almost by its very nature, has attracted, and will continue to attract very good technical brains.
-
Linux - the Unix defragmentation tool.
I find that in the design stage, there is no useful distinction between "at work" and "at home". You're working 24 hours, with ideas popping up from all over the place. Might as well leave at 5 and be more comfortable while you think; for that matter, you might as well leave at noon, and sit in your back yard with a beer and a pad of paper close at hand, staring at clouds and contemplating data structures.
However, in the coding stage, I find that immediately dropping everything and leaving at five works great, as long as things are rolling along when you leave. The best time to quit is when you're getting lots done and you know exactly what you're going to do next. Tomorrow morning, you'll know exactly what you've got to start with, and once you do that, you're in the flow again.
If you just keep coding untill you can't get any further, you'll come in the next day, tired out and wondering where to begin. You'll spend hours just getting rolling again.
That's been my personal experience, anyway. I've done some pretty neat stuff on 40-hour coding sprees, but usually the project died there. Sure, I'd get a week's work done in a day, maybe more, but somehow I'd never seem to get anywhere else with the hack-attack code. Eventually, I'd end up rewriting the project from scratch.
---
Despite rumors to the contrary, I am not a turnip.
I work for probably the most process-heavy, Dilbert-inspired corporate monolith around. We do have an intricate process involving frequent peer reviews of code, documentation, and architecture, combined with gate-keeping by senior designers who determine what is good enough to go into the final product. And y'know what, correct application of the correct process really does make our lives as software engineers easier. Reviews of requirements and architecture up front reduce the amount of re-coding we have to do at coding time, so much so that our time breakdown is something like 40% up-front documentation (including test cases), only 10-20% coding, and we have the rest of the time left to exhaustively test the thing. Sure, things aren't always so rosy for every single project and feature, but overall my impression is that you can get a lot of gain out of your process if you invest in it up front.
Some of the gains aren't evident right away, but full test case suites, requirements, and architecture documents make your results more maintainable. We still have 10 year old code in our product; because it was documented, we know that it still works, and we have tests to prove it. We're not in a situation of "don't change this magic code or the product will never work again".
The most important thing, though, is management support. It's true, we could turn around our product three times as fast without all of the process overhead, and I think sometimes management is tempted to go that way. But without documentation, etc., we wouldn't be able to keep up that pace for 10-15 years on a single product. Our release dates would gradually slip more and more, as the combination of all of our earlier shortcuts came back to hurt us. I see this happening to a lot of PC software companies (I'm in embedded systems) and it's painful to watch.
You really have to have a corporate-wide committment to software quality, up-front design and documentation, and a repeatable process, but if you're really in the software business to stay, I think it's worth it. SW engineering process is the stuff that was both tremendously boring and incredibly obvious in school, but it does work well if you take the time to use it.
Your right to not believe: Americans United for Separation of Church and
You don't get stuff fixed, code is usually an unmaintainable time bomb waiting to go off, and you see constructs that make you wonder if the coders ever went through basic CS. Data storage is rarely thought out and structures resemble rats nests. God help you if you have to deal with C++ code.
In short, the commercial development world is a Mongolian Clusterfuck of immense proportions.
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
Some random thoughts:
Wow, that got long. Hopefully some of it was useful.
Your right to not believe: Americans United for Separation of Church and
I acknowledge many of the problems this guy is talking about, and XP is a promising way of making this work in the real-world of software schedules. Also, some organizations (e.g. Data Connection Ltd, a former employer of mine) have amazing setups. Code review is mandatory - they don't try and tart it up as "objectives" either, it's part of the discipline of working professionally. It confounds me how many paid programmers don't even understand the meaning of the words "professional" and "discipline"!
Incidentally, I don't hold with his notion that a thousand developers adding features willy-nilly is better than a tight team with a coherent design philosophy, but - hey - you can hide all that crap behind a user interface and no-one's any the wiser. Hmm. Didn't Microsoft try something similar?
[The opinions stated in this article are not endorsed in any way by Data Connection Ltd. Just thought I'd make that clear!]
--
It's a
-- Danny Vermin
This is not to say that my own has always been of the very highest quality, but in fact I decided early on to try to come to a fundamental understanding of what was wrong with software development, to get very good at debugging (I say that debugging is a specialty on my homepage) and to learn to write better code.
In my early years I was initially very shocked at what I'd discover in production use at companies. Over the years I just learned that that's standard practice, in commercial software, in-house software, and even in scientific software (where I have become convinced, because of my experiences with high-energy physics data analysis, that many scientific papers are published with erroneous measurements because of software bugs).
Early on I read that something like 90% of software development is spent doing maintenance programming. Some of this is doing legitimate upgrading, but a lot of it is just spent fixing bugs, and even a lot of time spent doing upgrades would be more productive if the code were of better quality.
After reading this figure and having so many experiences with software bugs, both other people's and my own, I decided very early on to get very good at debugging.
One of the first things I did was adopt the regular use of "lint" for checking my C code. I would integrate lint targets into my makefiles and after editing a source file I would type "make lint" before compiling to objects and lint would check all the files that were out of date with the object modules. Pretty quickly I got to where I could write code that was nearly always lint-clean - but the existing code I worked on would make lint scream with hundreds if not thousands of complaints, often serious things like variables being used before they are initialized.
One of the first solid clues I got about software quality came from Robert Ward's book "Debugging C" - now out of print, it predated the common use of source code debuggers and talked about how to write your own stack crawls and other tools.
Ward emphasized the use of the Scientific Method in debugging, and because I was trained in physics, this came very naturally to me; before that I'd mostly floundered and used printf a lot.
I've gotten very good at debugging and have even worked full-time as a debugger at Apple Computer where I was a "Debug Meister" and my business card gave my title as "Cybernetic Entomologist".
I can easily get highly paid consulting work doing debugging for companies desperate to ship a product (and have in the past) but I don't really like to do it for various reasons, some of the same reasons I quit my debugging job at Apple.
One is that if I only do debugging I don't have something to point to at the end of the day and say "I wrote that". I could say "that works because of me", but sadly there's usually lots of bugs left that I didn't have the time to find so I don't really feel proud of the result. The other problem is that the bugs are usually not there because of something interesting, it's not like the code is mostly good but there's some subtle flaws, rather the code is a heap of dung and I can go in with a pitchfork and do debugging wholesale until I get tired of it and the client or manager decides the rate new bugs are being found is low enough they can feel OK about shipping it.
I don't feel good about contributing to such shoddiness. If a company is not good enough to support quality in their corporate culture I don't want to come in and put on a band-aid for them. It would be an entirely different thing if a company hired me to restructure their development process so that quality was a goal that was achieved through direct application of process but gee whiz no one has ever asked me to do that for them.
I do have to say though that the best thing that ever happened to me is that I became a "technology prostitute" as the author of the original article puts it. One benefit of this is that the process is entirely of my own creation, and almost all of the work I've been given has been to write entirely new products from scratch, so I can engineer in quality from the beginning.
Here's a few recommendations I have. Get good tools. Besides a compiler, editor and debugger, you need a static code checker and you also need dynamic testers. The ones I know about are (I haven't used them all yet):
- PC-Lint static code checking for C and C++. It runs on Windows but Flexe-Lint comes as shrouded source code and is highly portable.
- Spotlight dynamic tester for Mac PowerPC - I use this every day and recommend it highly
- BoundsChecker dynamic tester for Windows
- Purify dynamic tester for Unix (but apparently not Linux) and Windows NT
- Optimizeit dynamic tester for Java - do you know many Java programs have memory leaks? Can you understand why? Not just Java but any garbage collected program.
There's also memprof and electric fence, which I think run on Linux. It would be possible to modify gcc so that you program when built with debugging flags and linked with a special library would have all memory references boundschecked before allowing the access. This could work with both read and write access and would catch overruns by as little as one byte - Spotlight does this by patching the executable powerpc code and rewriting the debugging symbol table.Finally, to really come to understand the software quality problem in the industry and what you can do about it, read The Forum on Risks to the Public in Computers and Related Systems also available on the Usenet News as comp.risks. The book The Software Conspiracy exposes the complete disregard the commercial software industry has for serving the consumer by providing quality products - I haven't read it yet but it looks interesting.
A very interesting methodology that emphasizes personal responsibility and puts the fun back into programming as well as maintaining quality from the very start is Extreme Programming. I'm starting to adopt extreme programming (the the extent a one man operation can - I can't work in pairs :-/ ) and find it a tremendous benefit.
-- Could you use my software consulting serv
Oh no! We're doomed!
--
Have fun: Join D.N.A. (National Dyslexics Association)
I've been a professional software developer for nearly 20 years now and everything he writes rings very true. It's actually worse than that in many cases. I can count on the fingers of one hand the number of times I've actually been given a specification for a project, or even a reasonable list of requirements. And the established deadlines are typically even worse, you end up doubling or even tripling your estimates for project length simply because you know that whatever time you ask for will be cut in half (or more). Code reviews are something you read about in software engineering texts, if you actually know someone who is able to do them, you feel privileged. It's no wonder that almost all software projects are over budget and late, the budget and timeframe were unrealistic to begin with and the requirements and specifications never existed!
I've wrestled with reality for 35 years and I'm happy to say, I finally won out - Elwood P. Dowd
The IBM group which writes code for the space shuttle works 9am to 5pm almost exclusively, and they produce very high quality code. There's nothing wrong with coders who work 9-5, theres nothing wrong with coders who have lives outside the office. Sometimes, they're even more productive and predictable than those who put in, and brag about, their 36 hour coding stretches.
In the cathedral Engineering decision are often made by people who have no idea how to develop software, do not care how to develop software and would by no mean consider it appropriate to consult the coders before making decision.
:o)
Thus you end with administrative request about a near impossible feature to be implemented in an astoundingly short delay withouth enough resources. That added with the fact that a lot of programmers in the Cathedral are only waiting 5pm to go back home (and hack in code they really like??) forces people to cut around the corners and take shortcuts. After all the holy priests of the cathedrals do not care if the program is working fine... all that matters is if the feature is implemented and if the software has enough eye-candy to impress other priests in other cathedrals who will pay for our stuff.
Of course it might not be that way in all cathedrals... but the more I move from catehdral to cathedral the more I become atheist
"If liberty means anything at all, it means the right to tell people what they do not want to hear"
A nice piece, but he missing one of ESRs key points: Economic Viability.
His discussion of peer approval being crucial reinforces ESRs insight that a "gift economy" lies at the heart of Open Source culture, but the author gives no hint of how his (or anyone elses) open source coding would be compatible with long term economic self-interest
ESR does so in the Magic Cauldron
"one treats others with courtesy not because they are gentlemen or gentlewomen, but because you are" --G. Henrichs
Now I know that people will want to jump in and say "But look at all the bugs! Look at the poor quality! 2.4 will be released as better software because of the delay!" but the fact remains that the Cathedral gets products out the door while most OSSers live in the near future. "Just wait till the next version," we say, "when it will have [X] feature."
To a large degree, when we buy a piece of software, like Word 2.0, we don't just buy Word 2.0. We also buy into the idea that we bought a piece of software that works decently now, and the creator is committed to eventually upgrading the software until it's the word processor to end all word processors. And so we're at Word 9 today, and the cycle continues. When software is looked at this way, getting a product out the door becomes more important than waiting for people to finish it in their free time. That's why Cathedrals (read: corporations) will make money and that's why they'll stay in business.
I'll add two caveats to the above:
1: This doesn't necessarily apply to security products. Thank God Microsoft didn't introduce us to sockets.
2: Another reason why the Cathedral will always exist is because of competition. If you have highly sensitive secrets it doesn't make sense to publish your source. Security through obscurity might not be a good thing, but that doesn't mean I go around giving everyone copies of my door key and daring them to find out where I live.
--
Have fun: Join D.N.A. (National Dyslexics Association)