Slashdot Mirror
Ask Slashdot: On Good Software Design Processes
Marko Rauhamaa asks: "I'm a software manager in a medium-size technology company. Today I ran into a professional argument with my superiors about our company's software design process, which, I suppose, is fairly standard: The software team is to write one or more MS Word documents that have predefined section headings. The documents should describe all aspects of the coding phase that is about to begin. Then the documents are peer-reviewed, polished and submitted under document control. Questions: What kind of design process do the successful free-software projects have (Linux, gcc, Apache, XFree86, etc)?
In your experience, how often are design documents revisited after
the project? Have design documents helped in technology transfers (that is, have they been more helpful than the source code alone)?
Are engineers good at writing design documents? Have you been able
to read design documents written by other engineers? Have old design documents been kept up to date with the changes in the implementation?
Has the quality of your products been better because of design
documentation?"
I am a 'senior developer' at a 'technology company'. Would you like to know what our design process is? I'll tell you. A manager expresses a desire for a new program. Someone bangs out the desired program over a couple of weeks, that's our 'prototype'. Then we debug the prototype until it doesn't crash, most of the time. That's our 'beta'. Then we encounter tons of unexpected problems, corrupted data, and the program has to be scrapped.
In conclusion, it could be a lot worse than at your company.
ps. Anyone looking to hire an experienced C/C++ Solaris developer?
Just my limited experience... All standard disclaimers apply. Also, the rules vary a lot depending on the personalities on the team, the type of project, etc. It's impossible to summarize a general methodology in a slashdot message. I'd recommend picking up a few good books on software engineering.
- pmitros at mit.edu
This sounds like the philosophy exposed by Kent Beck and his XP crew.
The basic philosophy of XP is to write code until you have something that works. Write a unit test for it. Listen to what the users think of it. Then write it again. Refactor mercilessly.
It's a good philosophy, partly because it cuts out all the crap about up front design and UML and allows the shape of the code to take shape naturally. At the same time, the refactoring ensures that the project doesn't become a mess of unfinished prototypes, and the tests ensure that everything works when you refactor (or at least tell you when you broke something.)
here and here.
Strong second on Steve McConnell's books, though my personal favorite is Code Complete. Steve continues to write on this topic, and is using his current position as editor of IEEE Software to continue spreading the gospel^H^H^H^H^H^Hmessage.
He's been getting curious about the open source movement and the Linux phenomenon -- note Software's Jan/Feb 1999 Linux edition, and the editor's column and response in the current (Jul/Aug 1999) issue. Access is limited to IEEE and ACM members, but editor's columns tend to show up after a month or so at the Construx website.
Among Steve's criticisms of OSS are that design and architecture documentation are sorely lacking. As others have noted here, there are many instances where free software has set sights on existing functionality -- implementing systems the way they should have been in the first place, often closer to the design documentation than the proprietary application.
What part of "gestalt" don't you understand?
[just venting...]
...end of transmission...
An open source package that does this would be nice so that I could add any features that are missing, but I could settle for something closed. It would also be nice if it were accessible from a web browser, but it would be OK if it were not web based as long as it runs on Linux.
-----
Free P2P Backup, Windows & Linux
Fred also has said,
"My opinions are my own, and I've got *lots* of them!"
The most successful plan, is to fail...~ ^~
^~~^~^^~~^~^~^~^^~^^~^~^~~^^^~^^~~^~~~^~
I take a notebook to the bathroom and write brief design notes while doing business. The most difficult problems have to be solved that way. Some solutions are individual steps a procedure must take to do something. Some solutions are block diagrams. It doesn't have to be typewritten but it must be fast.
Detailed design is only necessary for the most difficult problems. For real world problems, databases, you don't need to document in extreme detail. Since most open source projects are simple utilities and desk accessories, there isn't much designing to be done.
I just got back from a training course on RUP, and I have to say that in my opinion it's a pretty good match to the "right way" to do software development. It certainly sounds better than what you currently have.
That said, I think Open Source projects have different goals and objectives, and this can get in the way of following traditional development processes.
In general, the more successfull OS projects have followed a lowest common denominator approach. They've general lowered the bar of entry for a "developer" to get in on the project as low as reasonably possible. This is a good strategy to follow, as the key part of open source is to get as many people involved as possible.
sigs are a waste of space
Try Extreme Programming ... a very light process that encourages coding, testing, refactoring and communication. Suited mainly to object-oriented apps (i.e. user level vs. system level).
-Stu
First of all, Microsoft Word is not a suitable software package for writing professional documentation. I don't know what your superiors mean by ``document control'', but it sure is difficult to do version control on binary files. Resist Microsoft Word.
I do Windows NT development, yet I run LaTeX (on NT) for documentation. Pepole want MS Word around here too, but they can stuff it. If I'm going to document, it will have to be with the right tools that save time and energy. LaTeX also allows me to easily carve up the document into multiple files which are checked into the version control system separately. Since our stupid version control system has strict locking (should be using CVS, doh!) this helps when two or more people need to modify the document. I can also use keyword expansion to label checked out copies of the document automatically so it's clear where they come from in the revision control system, and what version they are.
My second point is this: your superiors clearly don't know how to manage the risks of software development, and are trying to fall back on a naive ``waterfall model''. In this model, a carefully managed paper trail is supposed to lead up to the working product. One begins with the Exctracted Requirements, then moves on to the Requirements Specification, followed by the High Level Design, and the Detailed Design. CASE tools may be involved. Only at the end of the paper trail does the coding begin.
This model is unrealistic, difficult to follow to the letter, and increases your time to delivery by wasting everyone's time with paperwork. It's also unrealistic to assume that the requirements will never change, or that the design won't have to be modified after the coding has begun. It's a huge waste of time to polish documents that may turn out later to be wrong or obsolete.
The activities of designing, docuemnting and coding should be done in parallel with development. Often, it's acceptable to produce a design document after the coding is already done, as a way of gathering recording the intellectual work done by the developers. The design document can use elements from e-mails among developers, and their scrapbooks and whatnot. (It's not like nothing is recorded as the software is developed, it's just that it's not in a polished document!)
What you should do is settle on the requirements specification first, and then run with it. If your project is split up into teams whose software components have to communicate using tight interfaces, then design and document those interfaces, but don't worry about the internals too much. Write only those documents that are essential for everyone to be unblocked so they can go on with the business of writing the software. You need to know what the requirements are, and you need to know the interfaces to software components that other teams are working on, so it's good to have these captured early on. Worry about the design documentation later; wait until at least you are past the prototype stage and are into version 1.0.
Hope this makes some sort of sense.
I've worked at a Large Corporation, a Startup, and on my Own Open Source Project. You can find links to all of them on my web page.
Needs differ. There should be some process, but it differs.
For example, I have just enough documents for my GPL'd game, that I can handle it. They are on my web site for others to look at, but I am the main consumer. When more work is done and others begin to use my code, for example as a library with which to create another game, those documentation needs will clearly outweigh what I have. I will have to produce more documentation.
Finally, search the book reviews for my review of "The Unified Software Development Process" by Jacobson, Booch, and Rumbaugh. It's probably too heavy weight for small/medium sized projects, but the concepts are valid and the ideas intriguing. Reading it didn't hurt me.
--
Marc A. Lepage
Software Developer
I worked at a large corporation with about 75% engineers to 25% computer science. There are remarkable differences.
[I am CS.]
In my experience, the engineers suffer from a fatal flaw: they think computer science is easy. They think programming is just a trivial exercise, nothing compared to say building a bridge and putting your name on the blueprints for liability purposes.
Of course I believe differently. I believe we'd have had a bit more robustness in our designs if the engineers didn't just charge ahead into coding, but rather considered the theoretical aspects of what they were doing.
Of course this is a generalization, there were good and not so good individuals on both sides of the fence. But I found the generalization strange and peculiar.
--
Marc A. Lepage
Software Developer
Just one question.... have you ever tried to read a lot of the Linux source code? :)
Originally a quote about the NSA but works as well for MS in the above context.
--
"L'IT c'est moi!"
First, there's a big difference between coding FOR yourself and BY yourself. Doc for anything done just for your own use doesn't matter much to the rest of the world. Assuming it's not really important to you either, don't do any doc. Fine!
If you're programming BY yourself for others they (and the guy who eventually has to maintain your wonderful creation) could use some clues as to how it works, the caveats to using it, why you did this or that, and more importantly why you DIDN'T do this or that. (A little note saying "I wasted three days investigating this other neat -looking alternate design that didn't pan out" can save you or someone else another three days later in life.)
You may even find that mapping it out a little can save you a ton of time and effort later, or assure that it'll also work with XYZ package/file format/etc. When you're working on a big project composed of hundreds of different programs, you'll be screaming for up-to-date doc on them. Given your attitude though, I don't think there's much chance you'll get hired for anything like that in the near future.
The revolution will NOT be televised.
Most places you work, there will be some formal "process" that is perceived to let the factory model of production be applied to software development. There are two parties that have a vested interest in this: those that push a particular design process (typically CASE tool vendors), and those that want close estimates and the ability to interchange designers and programmers at will (managers).
Unfortunately, ANY such process, while it may have it's benefits, isn't perfect. True wisdom is knowing when "the Process" doesn't apply.
A few thoughts:
1) You can't think of everything up front, no mater HOW GOOD you think you are. This only works for small systems and systems with few well-defined intercomponent interfaces. You'll struve for the latter, but you won't get it first time around (or even second), but you'll get closer every time.
2) Anything which divorces implementation from design runs the usual risks associated with multiple views of the same think. Code often gets out of sync with UML, for example, though this can be audited for, and incrementally corrected. I do this CONSTANTLY. It is a form of overhead, though, and must be taken into account.
3) While metrics are great for predicting project side and time, collecting them often brings up a Heisenberg Uncertantly Principle effect: the mere act of measurement distorts that being measured. In some cases this can slow down the productivity of your best people to that of the mean -- that's expensive because you're best are likely an order of magnitude more productive than your worst, and so generate far more metrics to collect, even though many of them are useless. This flies directly in the face of rapid prototyping where the goal is to make things with the intent of throwing them away.
4)Don't waste your time auditing for mistakes that you are very unlikely to make. Of course, you need to collect some metrics to find out what mistakes each individual DOES make, but, in the absense of these, most programmers and designers have a good idea of where the mine fields are and ALREADY take extra care.
5)Don't come up with a "one size fits all" audit procedure. Some things are hard to audit for, and not all reviewers will have the skill to do this.
6)Do try to get the machine to do most of the work. If there is a "coding style standard", invest in a prettyprinter program instead of having people count spaces. Reading other's code isn't the same as iteratively developing your own. For example, I like my code to be fairly dense, so more of it fits into an editor window. This way, I can review the last thing I coded for correctness while I code the next one.
7)If something is too error-prone for some to use, but a great productivity booster to others, do let those others use it with the caveat that with this freedom comnes responsibility.
8)Remember that programmers and designers are not interchangable - the difference between the best and worst will be on about one order of magnitude (i.e. a factor of 10). Do not fall into the trap of hindering your best because the average has difficulty understanding a particular technique - if it is well documented (say anything in "Design Patterns", or even "The Art of Computer Programming" (I'm dating myself here), it should be fair game, and the onus on the neophyte to learn from it.
9)Remember to not etch process in stone. Third-party code you license will likely NOT follow your coding standards, and woe unto him who changes it when it comes time to integrate a source upgrade. Neither is it perfect. Don't trust the documentation -- read the code. If I had a dollar for everytime I say code not match documentation, I'd be rich.
10)Document what matters, and keep it short. The basic idea on a design or sub-design should be expressable in no more than a page or two. Don't linger on unnecessary details unless it is a code API.
11)Be flexible. While your process may be best 95% of the time, you will be burned when you try to apply it the other 5% of the time.
12)Know when to violate process, and how to weigh the risks of doing so. There is certainly risk here, but it is not infinite, and it might be far less than the risk of not violating process. This usually happens when (a) schedules are tightened unexpectedly, and/or (b) you run into the 5% scenario that your process handles badly.
13)Don't get bogged down in overhead. If you make a one line change that corrects a clear oversight, don't fill in a three page form, unless the oversight looks like it may be a systemic problem.
14)Finally, never assume anything. You're always playing the odds. Learn to manage uncertainty.
In Liberty, Rene
I've always hated uncommented code to the point that I over comment a bit. I always love the ability to pick up a random page of code and be able to understand it just by reading and not having to look up all the function calls and interpreting each line. And I can have a hard time remembering my logic in choosing a particular implementation, so I try to leave notes too.
One reason (if not the only reason) that people hate documenting is that it doesn't directly affect the final product - the compiler preprocesses documentation out of the code stream, docs only are the source code communication between programmers. In short, it feels like a waste of time because we like short term effectiveness and productivity over long term maintainability, programmers feel like they are making a difference in writing things that end up in the compiled program structer. It's all psychological.
I don't have the best 'root' document stating everything a code tree does, but it always has something to start from, and a generalized layout and plan of action.
One thing that might help is write everything in pseudocode, and go back to make real code, leaving the pseudo code as comments. It never hurts to think something through twice rather than coding on the fly and forgetting the details of why you did something, saving much debug and rewrite time.
I like to test and compile my code often, every time I finish even a function or a complex loop / conditional.
I suppose some of you might think I am anal, but I've never had porting problems with my code - not a line changes when compiling for 5 different platforms (NT, Solaris, SunOS, Linux Alpha and Intel/AMD).
While I am not against artistic coding, the scientific method can help a lot, as well as communicable clarity in code writing. The compiler can tell you wether your code is legal according to the given set of rules, it can't tell you anything about future maintainability or readability by you or other people.
Your attitude is flamebait for many of us who care about the quality of software. It won't be long before the lawyers start to have a field day with our profession because of attitudes like yours.
Here's some clues:
The size of project affects the software process you should use. You may be a good carpenter and can build a doghouse with your eyes closed, but your skills won't get you very far if you need to build a domed stadium.
The sooner that mistakes are caught, the cheaper it is to fix them. For example, it's much better to catch a missed requirement before the code is written than after, and much better than after the customer has bought and installed the software.
The initial code is a very small part of a "product". Users need to know how to use it. Tech writers need to know how to document it. Testers need to know what they are supposed to test. Product managers and sales people need to know what features there are and what features may be added to it in the future. Other programmers need to know how to enhance and maintain it. Etc. etc. And where do you suppose that all of these players are to get their information?
"You can find old programmers
And you can find bold programmers
But you can't find old, bold programmers."
That's because old programmers have too many scars from running boldly onto the battlefield shooting at anything that moves. Other formerly bold programmers are now dead.
>This requirement was needed since most military at the lower ranks had no high school or dropped out of school. This is not the case so much anymore.
Actually, This is not the case at all. Entrance to the military (even the Marines!) requires a high school diploma or GED (most branches won't even take a GED).
FWIW, anyways.
Personally, I greatly prefer programming alone,
as it eliminates problems with group communication
and lets me pursue a vision as to how things
should work. IMO, if you have 10 programmers
and 9 programs to be written, it'd be really
great if you assign 1 to making a library with
stuff that the other programmers think are likely
to be useful to many people, and assign the other
9 to one project each.
For every problem, there is at least one solution that is simple, neat, and wrong.
There's nothing wrong with C... it's still a very viable language to solve problems with. I do agree with on the hungarian notation bit.. though it's not limited to C.. I've seen it used in C++ and even Perl.
I've done some larger projects (aka "The Death March") and consulting work..
....
Usually documentation can only speak to 1 audience. If it's for the lusers.. I mean Users the techies say their isn't enough detail. If you include enough detail the users get bored to death, come back at the end and say I didn't know it said that I thought it said
Here are some of the thoughs I have on project management.. Not really a process though.
1. All users are morons. Untrained monkeys can understand the system but our users can't.
2. Users never say what they mean. They expect you to be psychic.
3. Talk to the Users NOT the managers of the users. Standard biz philosophy says "Managers don't need to know anything about what they are managing." And true to the philosophy they usually don't.
4. Make the involved parties sign off on what the project is. The document is probably wrong but at least you have something to change. (Verbal/Mental designs mutate with no controls.)
5. A project schedule that is highly detailed and inflexible is always wrong after a short number of days. Stick to a functional points in a sequence and an estimate of days. Be willing to change it when it hits the fan.
6. Break up the project into manageable bits.. 5 days for a person at most. Have a done, work in progress and to be started list. Watch your "critical path" for changes because of slippages/changes and adapt as needed. (if it's more than 5 days then you haven't broken it up enough.. I used to not believe that but it's true)
7. Project Management is kinda like when they had to fire the rocket on Apollo 13 with no guidance computer. Keep the earth (end of project) in the window and hang on.
As for the crux of the original question..
Any documentation is better than no documentation.
Find something that works and use it.
Source Control is an assumed. Those that do without it are just ignorant of it. Teach them and they will use it. (I had to do that once.. After a year they said I don't know how we ever got by without it)
Force all changes to documentation to go through a review and sign off. If someone can goto an individual (programmer/manager/etc) and change something with no review you are doomed.
-Jerry (and now back to our regularly scheduled interruptions)
If they don't pay attention to stuff like this, they will *always* be a small company.
If you don't document things, you'll waste time when you have to pick it back up in five months to do phase II or a vital bug fix.
If you don't have a process, your hackish methods of "getting it done" won't scale beyond about 5 people in the company.
If you *do* have a process, your senior people can usually write specs, document interfaces, explore architectures, and then turn *all* of that over to the juniors to get 80% of the coding done. Then they can do the really hard stuff that they know how to do best (because they're seniors).
My company is still working on this, and we'll probably keep working it out until we feel we "get it". But we know that we want to go from 7 coders => 20, and we can't do it unless we get a process in place.
It's a strange world -- let's keep it that way
That said, this question touches some deep issues that I'd like to try to comment on.
The main thing is that different projects have different needs in the design process. In most of the projects that I've worked on, learning how to do "X" is more of the thing than actually writing the code to do "X." This is probably because I tend to choose "fun" projects. If I were banging out yet another goddamn middleware transaction thingy, I'm sure that more formalized "processes" would make more sense.
As it is, the usual processes that get applied assume that it's already understood what is needed and how to accomplish that. One common thread to many of the horror stories I've read is the churn-mill of changing requirements and specifications. To me, this is a natural consequence of not knowing what you're doing to start with.
And not knowing what you're doing when you start is necessarily a problem, especially if learning is a part of the process. This is perhaps one of the greatest things about free software - it's a learning-centric process.
I think good design is just as important in the free software world as elsewhere, but it manifests itself in different ways. Usually, there is more than one competing project in a given space. Ideally, the one with the better design will win more developer hearts and minds. This is of course not always the case - often, a program will succeed in spite of its bad design, or perhaps because of it (a certain popular mail transfer agent certainly comes to mind :).
And the other thing about free software design is that it's often a whole lot simpler than comparable designs from the commercial world. Free software rarely comes with chrome-plated tail fins. In my opinion, the mark of a truly outstanding design is that you don't even notice it's there. Take the sockets API for example; compare it with any other networking API out there.
I guess my point is "different strokes for different folks." The fact that free software turns out such good work even though there's no formal design process certainly proves that the formal techniques are not necessary, and suggests that the formal people are missing a big piece of the puzzle. But for successfully executing needlessly complex projects by an army of more-or-less competent programmers, I'm sure they are an indispensable communications tool.
Maybe some day the software engineering crowd will work out tools and processes that work so well everyone will use them, even in free software. But I'm not holding my breath.
LILO boot: linux init=/usr/bin/emacs
In summary, while by no means am I suggesting that doing quality design work before the software is written is a bad idea - I'm just pointing out that in my experience, the only projects that can be designed to the last detail are projects you'd most likely find exceedingly uninteresting. Most software I've written that's worked out well has had a highly iterative quality to it - you design a bit, argue in front of a whiteboard a bit, code a bit, demo a bit then repeat. Certainly a good rule to remember is this: The longer development goes on for without the client/customer seeing the product (even as a demo/prototype) the larger the potential for the project to diverge significantly from customer expectations becomes.
there are two kinds of people in this world - those who divide people into two groups and those who don't
You're starting from the wrong end entirely: good software documentation has almost nothing to do with word processing, it has to do with creating the tightest possible association between the software elements and the descriptive elements so that the descriptions are actually USEFUL and get USED. If you start from the basis that it's going to be word-processed and that it'll have headings x/y/z then I can guarantee you that you're about to create yet another total waste of time and effort which will just drain resource and return almost nothing of value other than to satisy the paper pushers. If I earned a penny for each fancy document that never gets looked at even once after release, my fortune would make Bill Gates look like a pauper. There has to be a reason for that, and there is: such "documentation" is worthless because (i) it's too verbose, (ii) not verbose enough, (iii) too technical, (iv) not technical enough, (v) not suited for its audience, (vi) not wanted by its audience, (vii) not integrated into the development tools, (viii) at the wrong level of granularity, (ix) static, and (x) always out of date. Not surprisingly, 99% of the time it's a worthless pile of junk. Sigh .... If you really must put time and money into documentation, at least try to do it in a useful way rather than in the "standard" way: hire a developer with a research bias and get him to create an integrated information system for your developers. A system that updates itself from source code automatically and which allows structured annotations to be added with zero hassle and with automatic version control hidden behind the scenes. Make sure access to it is just a mouse click or keystroke away and directly driven from within (or alongside) the programming medium, and provide commandline access to the infobase because power users will leave the capabilities of any other interface way behind. And make it a living, dynamic info system, because if you don't then it'll be a very dead one.
"The question of whether machines can think is no more interesting than [] whether submarines can swim" - Dijkstra
"you'll get about half way into the project, realize the way you are solving your problem is all wrong and you have to scrap your work"
There's nothing wrong with recoding things. In fact, you should always plan on throwing away a lot of what you start doing.
The way I usually design programs (which seems to work well) is by hacking together a half-working solution (very buggy, missing features, command-line, but actually doing something) and then looking at the code and deducing where the separate pieces are, what problems will I face, and then creating a design document which outlines a good design to the problem.
Until you start coding, you really don't know what you're messing with. The best idea is to plan to throw away your first prototype, but use it as a place to design from.
Also, a great book to read about the software design process is "the mythical man-month". It's old, but the ideas are still useful.
Engineering and the Ultimate
I think your attitude will change about documentation when you inherit a project with a poorly written spec sheet, uncommented code, and hard-coded values in the program (my situation when I started in my current job).
Sure, it's no big deal when it's just you doing the coding, etc. Now try looking through some of the programs you wrote 5 years ago (in high school?) and figure out what you were doing. Even better, have someone look through your recent code and have THEM try to add a feature to it just from their looking at your code.
That's when good documentation becomes invaluable.
What your teachers are trying to do is have you get used to this process early in YOUR development as a good engineer, so that it's a mere force of habit once you start doing this professionally. It will not only help you, but also your co-workers and those who will eventually inherit your code.
My company engages in the same type of development cycle. We follow the published Software Engineering Institute (SEI) guidelines (we aim for level 3, although we don't always achieve it). Basically, the SEI requirements are a method of successfully managing a large software project. The positive things that come out of the requirements are a way for management to know the technical staff's ability (through metrics gathering), a way to grease the wheels of communication between developers, peer reviews, and effective client management. Many of these controls and processes are necessary for a corporate culture so that the pointy haired types can go and find work for us programmers. It also helps the pointy haired types to tell the paying customer that if they want to change the software requirements for the 12th time that it will cost x amount of dollars. By using the quality controls that have been learned from academia (peer reviews, open communication, and written design), the SEI has proven to management that the software sold will be better and cost less to develop.
With open source projects, many of those controls are unnecessary because the pointy haired boss is removed from the picture. So what is left? All the good points from the engineer's view. We have a large number of peer-reviews (more than any company could produce). The main difference here is that unlike a formal peer-review, you don't generate a list of defects for the programmer to go back and address--the reviewer has the ability to make direct changes. You have open communication. In fact, a lot of corporate entities could learn from the openness of communication. The development mailing lists are excellent tools, and it helps newbies get up to speed quicker. Granted there is a fair amount of noise on some of the lists, but the gains far outway the annoyance. In fact, the only thing that open source software may be able to learn from corporate life is the pre-written design/requirements documents. Many projects start without a specific direction in mind, or don't effectively communicate that direction, and consequently run into problems. What helps them, and what replaces the written documentation, is the open communication. In fact, the development lists is, many times, the 'white board' so to speek where the design is discussed before implementation. So your dev list archives become the replacement for your peer-reviewed documentation. In some ways it is better, because you get all the reasoning and ideas behind why they are going in a specific direction. In other ways it is not as good, because you have to fish through all the reasons and justifications to arrive at what is going on. All in all, the average developer has a better idea of the design and why we didn't use algorithm xyz that you thought (in theory) was so great.
-------------------------------------------------
Well, I kept telling my boss this was the route we should follow, but he just wouldn't listen.
Using HTML in email is like putting sound effects on your phone calls. Just say <strong>no</strong>.
Meanwhile, back in TRW...
I used to be a games developer, now I'm a website developer. If your game crashes on the MD (CEO in the States?) two days before delivery date, you'd better forget about seeing your bed anytime soon. And if your website loses customer orders for as little as a hundred quid (tr: bucks), you've got a lot of explaining to do.
The simple fact is that badly or undesigned code is a pain. Ever tried debugging 1000 lines of somebody else's uncommented 8086 assembler knowing that the man with the money for this month's wages is calling back in half an hour?
(And BTW, that's another downside to working for a small company where nobody is organised - it probably means the boss isn't either, and he's got to get the cash coming in.)
Using HTML in email is like putting sound effects on your phone calls. Just say <strong>no</strong>.
Questions:
> What kind of design process do the successful free-software projects
> have (Linux, gcc, Apache, XFree86, etc)?
None, I think.
> In your experience, how often are design documents revisited after
> the project?
Less than what they should.
> Have design documents helped in technology transfers (that is, have
> they been more helpful than the source code alone)?
Yes.
> Are engineers good at writing design documents?
No.
> Have you been able to read design documents written by other
> engineers?
Yes.
> Have old design documents been kept up to date with the changes in
> the implementation?
Yes.
> Has the quality of your products been better because of design
> documentation?"
Yes.
I think that I learned this "the hard way". At my first programming
job there was a "software supervisor" who forced all the
software-writing employees to submit design documents. Everybody
agreed that he was a pain in the ass. But everyone did his
duties. Design submitted, talked over with the guy, and software
written. 3 weeks, and that includes designing and building the
hardware, and learning the processor involved (8051).
Next they assigned me to a new project, in a different building. So
that software supervisor wasn't there to bug me. That project was
about of the same complexity, but it became a mess. Simply because we
didn't have a design document, we started adding features along the
road.
I think it pays to spend say 10% of the estimated development time
with your feet in the air thinking about the problem that is to be
solved. To give the bosses something tangible, you should write
something down in your design document.
Oh, another trap to watch out for: I've been reading some the specs
for the binary format for Excel sheets. Those are really well
designed. The specs that is. The actual implementation of what Excel
writes into a spreadsheet file is complete rubbish. So they have a
"design team" which did a reasonable job at designing the file format
and it turned out to be unusable for the guys actually implementing
the thing.
So you should not have a "design team" and then an "implementation
team". The implementors need a say in the design. Officially there is
a bell-curve in the number of developpers involved with a project. If
your project is small enough (say around 5 programmers), start the
design process with all programmers involved. Possibly you'll send
half the crew back to what they were doing during the next phase (say
if you have around 10 programmers), but have them in on the design.
If the project is even larger, you may have to design the project with
a small team, but then you should have a catching up session with the
new programmers once they get involved.
-- Roger.
Exactly!
My job is to make my users happy!
They don't need to know about 3NF, relation databases, parent-child relationships, objects, inheritance or any other techno terms. I design what they want and do it using the best design techniques at my command. The only time I "counsel" them is when they propose logical contradictions. In fact, they have supplied some of the best ideas towards solving some problems.
Running with Linux for over 20 years!
In a nutshell: peer review.
The projects i have worked with spend a great deal of energy on reviewing ideas. They are all quite egalitarian about the review process, and it mostly takes place on email lists. Large pieces of work tend to get adopted by small groups, who may work off-line and then publish the result. A good example of this is the evolution of apache's mod_jserv, housed at java.apache.org. Thus, requirements and high-level design are covered. The mail list bantering, change logs, and source comments are all that is needed to keep many OSS projects afloat. Of course, this is amongst some very experienced and dedicated code-grinders. It should, however, serve as a point of reference. Oh, and BTW: not all of these developers use English as their primary language, yet the projects are expressed in English.
How can they feel the rain but not know of the flood?
Well, think about it, maybe that is because you never really thought about the design until you totally screwed up version 1;)
I think it's best to make a rough sketch of your program first and only start filling in the details when you've coded the sketch (you can code a sketch you know, it won't work, but you'll learn from it anyway). This has the advantage that interfaces you code are usually pretty good...
just my 2 cents..
AUGH! Back evil lazy coder!
1) Have you been programming for more than a few years?
2) You've programmed, but have you ever really designed and implemented software?
3) Have you ever had to work on a team?
4) Have you ever had to work on something someone else worked on over a year ago-- or better yet, something you did a year ago?
I'm assuming your answer to all of these is no, or maybe the answer to this question:
5) Are you insane, a masochist, or both?
...is yes.
If you would have LISTENED in some of those classes, if they were anything like mine, you might have heard somewhere along the line that 80% of the software dev cycle is MAINTENANCE.
That's important in at least 2 ways:
A) If you have a battle plan laid out before you even start laying down a single byte of code, it's much more likely that later, you won't have as many issues to resolve because you thought about it and kept yourself from screwing it up.
B) You don't remember as much as you think. You will look at things you've written not more than 6 months ago and not remember a damn thing about what you were thinking. Let alone trying to divine what someone else was thinking if you're inheriting someone else's work.
So even when I'm just 'coding for myself', I still load my code with comments and at least write a readme in the header of the program or in a text file somewhere.
Please, please, please... do us all a favor-- don't go work for Microsoft, go back your classes and LISTEN for goodness sake.
The biggest misconception I find in the whole process is that once the design documents are reviewed, they turn read-only.
I'm a firm believer in the proof of concept prototype and the investigative coding but I also feel that you need to go back and make the design document reflect the final outcome. Remember, this won't be a release one project forever and, god forbid you should move on to a new project, they're still going to come to you for the sticky problem that they can't figure out and it's nice to have a quick refresher doc available when all eyes are on you. Besides, if the doc is complete enough, they won't have to!
P.S. Don't skip Step 2 8^)
I agree that most design documents end up badly out of date, inconsistent with the final product, and occasionally misleading through incompleteness.
However, I still insist on creating them - because they help me think before writing code. I happen to need all the help I can get when coding, and writing down what I intend to build before building it works best for me (peer reviews being a close second). If you and your team are all capable of working without these crutches - good for you ! But most of the code I have seen over the years would clearly have benefited from a bit of thought up-front - and I would assert that it is easier to deal with poor quality code than with a poor quality conceptual design if you are the poor sod who is trying to fix a bug long after the original developer left...
The purpose of design is not to keep management happy - most managers wouldn't know a good design from a hole in the ground, and the others haven't got time to read and understand a useful design document. The purpose is to make the developers think before they code.
Oh, and Steve McConnell has said all this far more clearly than I could...
It's all very well in practice, but it will never work in theory.
I hate it, personally. I'm a very linear thinker, and I don't like just throwing stuff at the wall and seeing what sticks. But the business reality is that for some people you don't really have the option to spend gobs of time on design.
I should say here that my client is quite reasonable when it comes time to fix things that might have been avoided had there been time to do proper planning up front -- so they're at least willing to admit that at least part of the problem is their own impatience.
DFL
Never send a human to do a machine's job.
I've walked both sides of the fence. My last employer was from the "And thou shall always wear a tie, yea even to code" school of thought. More documentation quality control systems than you can shake a stick at, let me tell you.
Where I am now, there's almost no formal documentation. My current project is going much better than any of the 'documented' ones, even though it's approximately 40 times bigger. How? Let me tell you a tale...
Good design documentation is focused. It serves a particular purpose. The person writing it knows what they're writing, and the person reading it know what they're looking for. If the prescribed section headings don't entirely fit, then they'll be thrown out and better headings will be used. Nothing gets in the way of the outcome that the document is being written to satisfy. Nothing.
Most design documentation, however, is done by bored programmers who would rather be coding. It's done to a template they didn't design, with no knowledge of why. It's done too early, by the wrong people, for the wrong reasons.
Fundamentally, there's a difference between documentation as a Tool, and as a Chore.
If the people aren't excited about writing the spec, then don't do it. Hmmm. 'Excited' might not be the right word. Ah, I know. 'Frantic'. That's a much better word. You should be frantic to make sure the project's not going to fail because you missed something. That's the reason for a spec, and don't ever forget it.
Management likes specs because it gives them that 'warm inner glow' that (a) work is happenning, (that they can understand) and (b) they've got something to cover their asses with later.
Orinoco's laws of Software Specifications.
1. The best spec is 1 page long, and a high-school student can read it. This goal is impossible, but worth aiming for.
2. End users can't write specs. If they could, they'd be system architects, and not order-entry clerks. Compiling a wish list of every user is like fulfilling the combined christmas list of a room full of 5-year olds. What they really need is new socks and underpants, but takes a deep knowledge of human nature to understand this.
3. Moreover, End users can't READ technical specs either. Don't show it to them, it will just cause confusion and worry. This causes more documentation. An endless cycle.
4. Management can't read specs either, though they pretend to and like to have a copy. Pander to this need, otherwise they will make your life a living hell.
5. The only people, therefore, who read and will understand the spec are your co-workers and yourself. So keep them in mind. If there's nobody on the project but you, you're only writing for yourself.
6. The full functional spec is only ever used during a billing dispute. If you've hit that point, you've failed anyway.
7. If you haven't already done it at least once, it shouldn't go into the spec. This implies that the spec should be written after the prototype.
8. Once work starts, a spec should be constantly evolving, but not too much. New things should trickle in by everyone's agreement. Major changes are cause enough for stopping and asking "Why wasn't it in there in the first place!" because there's bound to be more. An unchanging spec is dead and useless.
9. Writing code is inherently creative. Any creative endeavor is unpredictable. Never forget that.
10. Specs are like Backups. No-one understands the value of a good spec. until a project has crashed and burned around them for lack of one.
Far better than specs are what I call 'principle documents'. Something nice to nail to the PM's door. Nothing specific, just all the important generalities like "Nothing should take more than 3 seconds to process" for a help-desk system, or "Easy learning curve" for a system used only rarely.
And lastly, all specs should contain some elements of process re-engineering. A modern buzzword which boils down to "Why the hell are you doing it THAT way?!?!"
A good project is just like a good hack. Half of it is social engineering.
Hmm. As usual I spouted off and this is far too long. Apologies.
Jeremy Lee | Orinoco
Before I start, my qualifications: I've been a programmer, dba, software engineer, technical
architect, configuration manager, and sysadmin, on mainframes, pcs, *and* UNIX for coming
up on 19 years. I've worked for companies from 6 people, including the boss and his wife, to Ameritech.
You wrote:
> The documents should describe all aspects of the coding phase that is about to begin.
Define "describe all aspects". I have worked from design docs that I think Boing did, in the
early eighties, that were pseudocode, and I have worked from vaguely-worded memos, and
I have worked from verbal descriptions. Personally, when I do it *right*, I prefer flow charts.
So, just how detailed will these documents *be*? If they're down to the pseudocode level,
they'll take as long as the coding phase.
> Then the documents are peer-reviewed, polished and submitted under document control.
Define peer review. Most places I've worked, *if* they had peer review, it was dumped on
someone already overworked, who then maybe skimmed it, and said, "looks fine to me".
> Questions:
> In your experience, how often are design documents revisited after the project?
Never, except for *possibly* some poor programmer trying to figure out what this thing *does*,
so they can fix what's broken, or enhance it, years after everyone else involved is long gone.
> Have design documents helped in technology transfers (that is, have they been more helpful than the
> source code alone)?
That depends on the level of the design documents. You might consider that they define what has to
go in, what has to come out, what *must* be stored (not what they *think* they want to store, for
some possibility that they might look at it in the future), and conditions *must* be looked for,
along with some style standards, and a *real* architecture, appropriate to the project (not
ridiculously overengineered, nor kludged together as they go), and leave the lower level details to
the folks actually doing the work.
> Are engineers good at writing design documents?
Few (I'm an exception). Unfortunately, competant tech writers are few and far between, and
even more rarely *hired* by management.
> Have you been able to read design documents written by other engineers?
Some.
> Have old design documents been kept up to date with the changes in the implementation?
Never. Ever. Not in nonprofits, not in colleges, not in mortgage co's....
> Has the quality of your products been better because of design documentation?
Depended on the documentation. Note that 80% of the time, btw, that documentation is the
*last* thing done.... (B-{)}
mark roth-whitworth
Basically, you script out how a user interacts with the system and try to cover all standard and anomalous cases.
If the system is large, it is usually productive to phase the project in terms of iterations, as opposed to doing -all- the design up front a-la the waterfall model. This is because a tremendous amount of discovery happens during the implementation process, no matter how thorough you might have thought you were in the design process. Both the customer and the implementer will end up discovering new and changed requirements.
By doing a top-to-bottom iteration- a reusable prototype so to speak, you and the customer can get a better sense of what you will encounter during the rest of the project. You will also have a preliminary unit effort measurement with which you can guage the rest of the project by.
I recommend putting the use cases in a database that preserves revisions, and composing the design docs on the fly. This is because you will want to view them from multiple points of view, such the users point of view, a workflow point of view, or a system component point of view. Databases are much more amenable to this than documents are.
On the other side of the coin are infrastructure type projects where the users are higher level components. The primary design doc for this kind of project is an API spec. With languages like Java or C++, you can deliver this in code, that both the users and implementers can compile against. You can write stubs and drivers that permit both the implementer and the user to work independently. Code is the language of choice for 'design docs' targeted at technical people. If you cannot express a design clearly in a layer of code, the language and API set your are basing your project on may be either at too high or too low a level for what you are trying to do.
IMHO, a chunk of infrastructure is best designed by a single good architect and reviewed by peers. This ensures some degree of consistency in the outcome. If an infrastructure project is over-divided among several architects, the 'whole' may end up being a suboptimal compromise.
I have personally experienced such a situation. Because many designers wanted to have a say in the project, there was a lot of political horse trading over who would implement what capability in their part of the system. The end result was an execution path that had way too many interface layers and representation conversions to be efficient. There were good arguments for each individual design, but since there was never 'proof' of the superiority of any the approaches, management forced a compromise rather than a choice.
I think that a group needs to elect a leader for a project and then submit themselves to the choices that this leader makes. The leader should not compromise architecture in order to appease different opinions in the group.
Well, that's my 2 cents. I hope it helps.
Mike
How true it is that version 2 is always the first
good release. Always plan to throw one away.
Procedures like this are important for proprietary software development, but not for open-source projects. Bear that in mind when you read peoples responses here, because /. is obviously going to be biased toward open-source style advice.
With free software and open-source projects, people coming on board the project have probably already reviewed the source code in detail. They've had it for awhile, and probably are joining your team because they've begun tinkering with it on their own.
With a proprietary project, new developers have never seen the code before and have no idea how it might work, because it's proprietary. Therefore you need to spend resources helping them learn about it, and documents are one way.
With an open-source project, all of the users can peer through the code and search out the bugs when something doesn't work. This in my opinion is the biggest "10x" advantage that open-source projects have. Also, a saying I like is that "every problem has an obvious solution, if enough eyes see it". A bug that just mystifies you will be totally obvious to one of your users.
With a proprietary project, your users cannot read the code, and only you and the people on your team can debug it. Therefore you must devote time and energy toward tracking down bugs--to make up for not having hundreds or thousands of eager users doing that for you. And the bugs may not be obvious to you at first (not enough eyes) so you have to have additional procedures to methodically track them down.
Thus a properietary project needs peer-reviews, code-walkthroughs, and so forth.
In a nutshell, all that "software engineering institute" advice really is valuable, and you really should do it. Though people here will tell you that they have almost no procedure and things seem to work--it's likely they're talking about either one or two person projects, or else about open-source projects.
In a proprietary software environment, you really do need to do a few things by the book.
Of course if those documents aren't reliably maintained, or there isn't a way to mark them as "out of date", or whatever, then they may not be very much good for knowledge transfer; they should still be of some "design review" benefit though.
My biggest problem with this is...
WHY ON EARTH ARE THEY MSWORD FILES?! For gods sake, put those things in HTML and post them on your internal web. They're much more accessible that way.
A corrollary to this is that if you do build one to throw away, you will end up throwing away the second one, too.
Your best bet is to look at the RUP
;-)
Try the Rational Unified Process
It's about the best of the current processes.
(Even though it is a Rational product
Has a lot of recommendation for documentation and
process cycles
(BTW, I'm a Java consultant and I use RUP)
Asmo
Having a plan in the form of some written specs is very valuable when embarking on a coding project. I was managing a decent sized project some years ago and we went through the process of a preliminary design followed by a review, a critical design, again, followed by a review, etc. If this sounds too rigid, well, we did it to satisfy the people who were paying the bills (we were contractors to the Army). Since our system had to interact with flight critical equipment, we pretty much had to do it their way. We actually got off pretty easy. If we had had to adopt their methodologies in toto we would have had to hire a boatload of subcontractors.
For many of the people on the team, it was the first time they'd had to sit down and actually plan what they were going to do before they started doing it. It was painful for these folks but, you know what, the hardware and software that we produced worked the first time. When additional software modules were introduced for testing... they worked the first time. I suppose that much of that might have happened without the highly structured design work that was done up front. But I don't think that the coding and testing phases would have gone anywhere nearly as smoothly if we hadn't done it this way. We would have spent way too much time rewriting code that didn't allow additional modules to be added later. Our data structures would have been modified too many times and adding each additional function would have been a separate nightmare.
Maybe I was lucky and had great programmers (I'd like to think that I was able to put together a crack team.)
As another example, take a look at the way the basic building blocks of the internet came to be. I don't think the IETF documents for the various protocols were written after the fact.
An interesting study was done some years ago. I think it was at McGill Univ. Teams of engineering students were assembled to build the same thing. One team built the project and wrote the documentation and the user manual afterward; the other team did all the documentation first. The team that did the documentation beforehand produced a product that worked better and more closely met the design goals. Not proof that this is the best way; just another example showing that it seems to work better when you do it that way.
Let's face it, programmers hate doing documentation. Just look at the spotty documentation for some important areas of Linux. The manpages and the info pages are often out of sync (why must this be?). The LDP project needs to be better supported and, dammit, the developers need to be major contributors. Maybe they shouldn't write the prose (they're often much, much better at C/C++ than English) but the technical input needs to be there. Pass the documentation back and forth a couple of times until it's correct and is readable by people of different levels of UNIX experience. This is the stuff that the new users are told to read in order to get Linux to work. If the documentation sucks, then that's the impression they're going to have of Linux as a whole. If Linux works but they can't figure out how then it's all going to look like Black Magic and its adoption will slow or decline. Do any developers out there look for assistance from the user community to help write the documentation, manpages, etc.
I realize that this got slightly off the original topic but I feel that the two go hand-in-hand. Now that I got that off my chest...
CUR ALLOC 20195.....5804M
Ah, youth!
I can remember having that attitude when I started programming.
BZZZZZZTTT! Wrong!
I have many old floppies with code from those days on them that I would just as soon rewrite from scratch than attempt to figure how they worked. I'm sure some of these old programs might be worth modifying to be useful again or contain some code snippets that could be re-used. But without the comments and accompanying documentation, it just isn't worth the time anymore to have to put on the ``code archeologist's'' hat in order to find the useful stuff.
Go ahead and take that attitude now; you'll come to regret it when you get older. Or, perhaps, when you take that attitude into a job interview.
CUR ALLOC 20195.....5804M
btw, your questions seem to indicate that you already expect a certain answer. why do you do that?
3.The goat is sacrificed in the middle of an inverted pentagram while the PM chants "CTHULHU FNAGN" (this step is optional)
That process will give you... unpredictable... results. Are you prepared to cope with the following?
1). Do you really want to write a library function to see if your source code is infested with the minions of Nyogtha?
2). Are your users sufficienly trained to cope with the non-Euclidean madness that will appear to form somewhere above their keyboards?
3). You won't be able to stop gazing into the Shining Trapezohedron.
4). Thursday morning staff meetings will now be used to contemplate the enactments of ruin.
You should weigh all of the pros and cons involved before taking a step this drastic.
->Dan
->Dan
Absolutely correct. This boils down to the difference between being an engineer and being a cowboy.
The products of engineering are design documents. ChemicaL engineers produce designs for chemical plants, and then construct, or supervise the construction of, those plants according to the designs. Software engineers produce designs for software, and then construct, or supervise the construction of, that software according to the designs. If that is not what you are doing then you are not a software engineer.
Anyone who cannot write a good design document is not an engineer. A programmer, perhaps, but never an engineer.
I also highly recommend this book. I have found the contents invaluable on numerous occasions.
One important thing that I got from the book is that there is no "one true way" of software development. Each project has its own nuances that may or may not warrant a specific approach. As a contractor, I have now gotten to use almost all of the major software life cycles.
Something to also consider is that the team itself will work differently under different life cycles. I personally have started to favor the "jelly roll" model. It's directed chaos can be adapted to many situations and its high output and use of numerous prototypes keeps both management/users and developers interested. However, there are some teams where it will break down rapidly or become too much work to keep it from breaking down.
Poor communication and/or poor team dynamic can be overcome, but it's often easier to choose a more classic life cycle that requires more stringent design and documentation. Some programmers really can't seem to produce quality stuff unless you've micromanaged their portion down into bite-sized chunks.
P.S.: In documentating a design, never underestimate the power of use-cases. They are both useful in making sure the design is complete/robust and imparting the ideas onto the other developers in the team. When you are presenting a "well understood" idea to a team of competent developers and every one of them goes, "Oh, I get it now." when they turn to the use-cases then you'll know what I'm talking about.
P.P.S.: Never overestimate the value of whiteboarding. If someone doesn't document it for review it's as good as useless. As an experiment, doodle a design on the whiteboard for you team. Then, have each one of them go back to their offices and put together a design document. They will be extremely different in ways that will surprise and possibly shock you.
Edu. sig-line: Choose rhymes with lose. Chose rhymes with goes. Loose rhymes with goose.
Comparing? THEN use THAN.
I have recently started to develop open source applications and I suppose I could say that most of them have no use for the documentation you describe.
I have not seen any of such documents accompanying any of the most popular open source applications. I wonder if there is something like that for stuff like GCC, Linux, Emacs, WM, XWindows, Apache and the like. As I said, I have yet to see it. Umm, at most I could mention the GNU Standards document, but that does not document an architecture or anything like that.
While some applications (GNOME comes to mind) do have some sort of Manifesto describing the architecture behind them, I believe that is not really important in the development of open source applications.
Most open source applications begin because the original author has a problem and needs to get it fixed. They usually start with something small, not very ambitious... just a little set of hacks to solve the problems the authors have. Over the course of time others will play with the code and begin to improve it and improve it and improve it. And suddenly the set of patches and hacks become something like Apache or Linux.
I believe `Start small, grow fast' would be the best way to go in open source and, in some situations, closed source applications.
The original author usually does not have a clear idea on how he'll implement all of the things he knows he will have to. As ESR points out in C&B, he will usually realize of new solutions to the problem only after he has implemented things.
The time spent documenting the architecture behind the applications is time not spent actually coding them. The architecture is constantly changing as other persons are always suggesting new ideas and sending code back to the original author. The latter has to be willing to throw lots of code away and begin again continuously. And I don't think stoping to write documents explaining the framework will really help.
I'm not saying the design process should be taken lightly; it is very important and can really save time. But once the author knows how to do the thing, he will just go and code it, not document it.
On the other hand, most open source applications have a developers mailing list behind them. It is not uncommon to see discussion about the framework on the lists. I believe the messages passing through it can play the same role of the design documentation you mention.
What is often important to provide is documentation on how the application *currently* works, documentation on the latest implementation. For example, you can go and find documents stating how to create Apache modules: explaining the structs and the functions you can use to get things done. But their purpose is not to document how Apache should be built; they are just provided to help persons interested in using Apache (creating a module). And they are written after the application is written, not before. They follow the implementation, not otherwise.
Alejo.
... nd then parcel the work out into manageable bits. Let the
implementation details be documented by the individual coders as comments within the actual code!
The only thing I would add to this is that, at the design level, I find way too many people get WAY too into the details of a project. As the post to which I am responding said, algorithms first, details later. You never know exactly how you're going to need to fit parts X, Y, and Z together once part AA rears its ugly head. It's much more efficient to expect and plan for an evolution of model ahead of time than to expect to be able to plan for every contingency.
--
blue
i browse at -1 because they're funnier than you are.
Correct. Documentation takes about 40% of the time.
Design takes about 50%.
And implementation takes about 60%.
These are not performed concurrently, which is why software is always late. Except when you leave out the design phase... But we all know what kind of software that spawns.
-- What you do today will cost you a day of your life.
What you describe doesn't sound like a development process at all. It sounds like a documentation burr... A development process is a pretty ethereal animal (those that may or may not exist, depending on whom you ask).
I've personally not worked on open source projects (yet) but I imagine that they are vastly different than any commercial effort. Seems to me that managing gratis developers is like herding cats - if you try to control them, they'll simply leave.
But, I would strongly recommend Steve McConnell's book on Rapid Development, and Code Complete while you're at it.
The RD book - well, eat it. Read it cover to cover twice, and with that knowledge in your head, use what fits your project and developers.
Your people may like to do thorough design up front, and follow the traditional 'waterfall' process, but that doesn't stand up well to changing specs.
Incremental development seems to work well where I work. We have a small team, and in-house users, so feedback and even design changes can happen pretty quickly..
You'll need to look at the risks your project is facing as well as a number of other factors - i.e. do you subcontract, buy COTS stuff, use strict CM and are you subject to stringent V&V?? Who are your users, how skilled are your people? Look where you fall on the Capability Maturity Model (1.1 release) hierarchy and how you rank per ISO 9000-3. If nothing else, you'll get some ideas.
As you can guess, there's a huge number of variables that go into defining a successful process. The Software Engineering Institute at Carnegie-Mellon University may prove helpful, but I would recommend the aforementioned book (RD) first.
-- What you do today will cost you a day of your life.
Don't bury innovative ideas by making the "process" of software-development water tight. Though the project will accomplish it's goal, it is usually in the project's best intrest to be open to development of primary ideas after the launch. Software devlopment can be seen like the making of a Jim Carrey movie. Sure, you can force Jim to follow the script to the letter, but if you keep him from doing his last-minute improvisational work, you loose the lifeblood of the movie (and wind up with shit like "The Cable Guy").
:)
Maybe I just need to get some sleep
Agreed. This is a huge question, and honestly I think I might be difficult to get an balanced answer from this type of forum. Most /. readers are highly technical, and tend to focus on the "gears" of a problem instead of the whole "system".
In general, the answer I like to use in bracked in these types of situations is "whatever works for you." How you and your team approach a problem will decide how you solve it. Forcing youself to an artifical system that no one understands WHY you are doing it will get you nowhere fast.
I know programmers that like to just hack stuff out just to get it out, even if they have to redo it all later. Some like everything pre-modeled and conceptualized to the nth degree. Most do something in between.
My personal preferences includes something I call the "metaphor" system. Whatever I do, whatever I try to come up with an accurate metaphorial equivalent. During the lifetime of the project I create whatever documents are required to demostrate the functioning of the software. The advantages to this are:
= Both technical and non-technical people can read your documentation and get a clear understanding what is going on based on their respective levels of understanding.
= The documentation and code should if correct make clear rashional sense.
= The metaphoral system can be used to give you a clear understanding of what tools you use in order to accomplish your goals most effectively, for example if you metaphors include references to finite physical objects, then a OO design might make sense.
= In my experience, the metaphoral system can lead to more effective and innovative designs, and eases troubleshooting.
With a large group of people with different skill levels, a metaphoral system is always the way I go.
Most of the open source has man pages, info on a web site, a README, and then the source code. Being someone who has developed a few products both corporate and open source, design documents are not always done first. Sometimes the application comes first and then the design.
In the open source apps I have developed, there were 'idea' documents. Basically a few sheets of paper that had what I wanted it to do, and what I wanted it to look like on it. Other than that there was nothing that I'd call an actual document.
In the workplace however there have been cases where there have been design documents, and there have not been. In the case that there is a design, it always seems to be a changing design. Either the system cannot do what the design says, or after an initial implementation the design is changed. Sometimes the design document is done as the code is written. This is a working design.
On a few occasions there was no design. My notes on a whiteboard, and the the code. The white board gets errased, and so does the design.
This is not to say that designes are useless, they are good to have, and theu do make writing programs easier, if they are well thought out.
The thing to remember is that just because you have a design does not mean that you should stick to the design like it was carved in stone. The design should be flexable and changeing, until the applicatin is complete. Also a good design makes redesigning a system later a bit easier. You may write it in Java this year, but in 2 years there may be a new language. You will want your design in those 2 or 3 years, especially if your application does Financial Calculations, like an account package.
In the system I am currently working on, we do not have the design documents, so we are looking at the code and reverse engineering it. It does ledger updates, and number tables updates. There are literally thousands of programs that make up this system. It is a real pain. Right aboutnow I wish I had the designes from 20 years ago.
Only 'flamers' flame!
Everyone above was too hard on the new programmer & to top it off, you posted anonymously. If you are going to give advice like that, have the guts to post who you are. I am not disagreeing about documentation & stuff, I have been a software developer for 5 years in the industry. Not much time I know. And I do planning & documentation as much as I can. However, the places I have been employed just want it done & done yesterday. Also, the development teams were small, too small to complete the project efficiently. The groups I worked in had to improvise, planning & documenting only to the point where it benefited us & those maintaining the code base. No overkill. All that is great if you have the time. But you usually never have the time to do the job the way it should be done. Business people just don't care. They just want something that meets the requirements. Documentation is only important when the changes are needed. Then they yell at you for not documenting the project that well ;). Anyway, I am familiar w/ all the training one receives in college about good development methodologies & tools. Alot of it isn't practical, atleast in my experience. Writing code takes time--and that is the one thing you won't get when you develop. Well, all I can say is learn the development techniques your profs are emphasizing-you will find them useful in projects. Remember though, none of it is an exact science-anyone who says so is full of bs. You use the tools & document the project so that it meets your immediate needs. Then try to document the project keeping in mind that someday, someone might have to change it. Most of the jobs out their in programming are altering someone else's code--odds are, they did not follow many of the methodologies you learned in college (they didn't exist, the programmers didn't care/know, etc.). So all of the people that have posted earlier probably have jobs fixing the problems. Just like 10yrs+ down the road from now--programmers will be fixing their code & wondering why they didn't document it very well ;). That is the way it goes. What is adequate now is inadequate later. Welcome to computer science.
I think this comment is not completely correct. I work on free sw mostly because I get to do process and documentation. In fact, I think that free sw benefits even more from process than commercial sw. If you only have a few hours per week to work on a project, you can't waste time. Wasting 5 hours might be half a day on a commercial project, and an entire week on a free sw project.
I am currently having a small debate with my boss over whether to have any design documentation or peer review at all. It's small because she just tells me what she thinks, and won't listen to anything else. She even publically humiliated me last week because I wrote requirements and design documentation for a tool I wrote some months back. She is of the belief that getting a prototype working and shipped is more important than getting something which works. (Of course, we also had a big shouting match about how to round numbers up to the next highest multiple of eight. She got some expression from some Microsoft sample program which didn't work, but we had to do the calculation for all 8 possible remainders before she believed me.) Of course this all means I will be looking for a new job in the next few weeks.
So, I in my spare time I am working on a program to do the very prosaic job of backing up a file system to a CDROM burner, using mkisofs and cdrecord. Since it's my program I did it my way. I started with a requirements document, did a detailed design, and only then started coding. I started with RCS, but now I'm using CVS. It seems to have worked pretty well, but I've had to do some backtracking to fix some technical details. Some of the stuff I wanted to implement turns out to be too time consuming to do with the 5 hours/wk I have available. I believe the process helped me a great deal.
I also believe that requirements documentation is even more important than the design document. The design document tells you how to solve a problem, but the requirements documentation tells you what the problem is. Of course, both of them are in some sense conjectures until you get to the end of the project. You don't often know what the problem is until you have tried to state and then solve it a few times. But if you don't try to state the problem in natural language terms, you will never succeed in solving it. Just my 0.02 Euros.
(Anybody need a CS Ph.D. with 15 years of experience who both designs and codes? Boston area, or my DSL? No Win, W2K or NT please. Resume on request.)
I have always thought this was caused by history, as are all things I suppose. Computing was very expensive 20 years ago, and even 10 years ago it was not cheap. The computing community learned to optimize their scarce resources, and they carefully crafted memory and time efficient solutions to problems. This is why people in the 80s would never use something as inefficient as the C++ standard template libraries, for example.
To try to draw out the EE analogy, a CPU designer undoubtedly wastes many transistors in order to get some other sort of efficiency, say regularity, or to use something like a standard register definition. These other efficiencies might be manufacturing or design efficiencies which don't occur in the actual device, but which represent a cost savings overall. (I don't know if this is true, and I would be happy to be told I am wrong.)
Now that large scale computational resources are almost freely available, the environment has all changed. It's now possible and probably necessary to design large systems, each of whose components are perhaps not the most efficient possible, but which fit together nicely, and which work, whatever "work" means.
This is an extremely rude remark, and in very poor taste. This is the kind of thing which gives /. readers a bad name in the Linux community. Your ill bred language and foolish, foolish threats do you no credit. You should be soundly ashamed of yourself.
I've spent some years learning The Way To Do Software Engineering through university for some years now, I've done some reading and I've worked on quite a few projects. In essence I agree with pretty much everything that has been said so far.
/and/ code the majority of the system. Other people perform support tasks. Larger projects require a more rigourous/formal approach but the intent is the same. Do read "The Mythical Man Month" as was referred to earlier as it explains why this sort of approach is a Good Thing.
I have a couple of things to clarify/add:
- Assume you will throw your first one away. I think I got that from the Camel book, but it has proven very correct in the real world. If you are doing something that has not been done before or is not substantially similar, you will get it wrong the first time. This is a Good Thing. The first try (sometimes called a prototype, or the alpha version, or whatever; pick a label) will teach you a lot about what it is you're trying to do. You will discover holes in the requirements. You will discover that what seemed easy is actually hard, and vice versa. You can then take this newly gained experience and start again. A caveat: Don't try to hack bits together to make the first try work. Scavenge the good bits and write the thing again, or you'll end up with an unmaintainable kludge that doesn't work properly and is difficult to debug. Clean code is good code.
- Don't adhere to one design methodology all the time. Just as sometimes it's good to use Perl and other times you need C or assembler or whatever, you need to choose the correct tools for the job. Informal methods may work well for you, or you may prefer a structured approach. Project size will affect your decision. I generally use an informal approach on small to medium projects with 1 or 2 system architects who design
Ultimately, experience is the best teacher. Learn how to run a project by participating in one. Note what works well and what doesn't. Constantly review what you've learned (albeit subconsciously). Everyone has their own style and their own preferences. Learn from others and develop your own. You'll know you've got it right when it works well.
Justin
Just because you're paranoid doesn't mean they're NOT after you.
I worked at a small software company (this one was VERY small) and we obviously had nothing that rigorous.
You know this, but I'll just go ahead and say it anyway: Unless you're writing "Hello world" you ALWAYS do a thorough design first. With my company, that meant everyone standing around a white board armed with markers, arguing about how to design it. Once we had our design up on the white board we would leave it there and divide it up into pieces and hand out the pieces.
Things were always informal with a small group, and we never had to worry about miscommunication. With a big team the plan you described sounds like a good one to me.
Of course, I think they should always start with a couple hours around a white board.
Vidi, Vici, Veni
If you are serious about implementing good software design you need a helluva lot more that an Ask Slashdot.
There are a multitude of books written on the subject of software design including a whole field of software engineering. The fact that the documents are written in Word but Word is not available on Linux seemed to be a major focus of your question. As many will tell you, the document format you use is of course irrelevant, your goal should be to insure that those documents have the required information so that not only will any future employees be able to understand what the author has done but the author himself will also be able to review them to re-focus himself. As anyone who has ever worked on a large software project will tell you, you WILL lose track of your original goal, you WILL get distracted by relatively minor issues. Hell it even happens on small projects.
I personally lean towards Object Oriented Analysis and Design (probably because a certain professor pounded it into my head). There is a whole series of books cropping up around the Unified Modeling Language which you may want to read if you are serious. It provides a way for you to represent you system because let's face it, words are not sufficient and MS Clip Art while better than some OOAD packages *cough* Paradigm Minus *cough* just won't do.
I'd learn the concepts before worrying about what packages are out there and what platforms they happen to run on. In the end it won't be nearly as important as you think.
This works for small projects but only for small projects. This is the kind of shit my buddies and I did for our CS courses where the longest projects lasted a month or two.
This is not an appropriate forum for this question. The idea that someone could convey how to implement good software design in a 100 word post is ludicrous. This is something that someone who is serious would have to invest a great deal of time into.
Oh and as far as algorithmic proofs, they may not be right for you but there are some projects where a simple "Uhhh geee well it looks like it should werk" won't cut it. Sorry that plane just crashed, I guess the logic in that algorithm was flawed....
Ah, I take it Microsoft hasn't gotten around to throwing anything out yet
www.atacomm.com - The Leader in VoIP Product Distributi
I think most programmers (even those of us who call themselves software engineers):
a) agree on the need for good documentation
b)...provided we don't have to spend too much time writing it.
From looking at KDE, it seems the requirements for a successful open source project are:
a) Version control of all software (CVS/PVCS/Clearcase whatever)
b) A good centralised mailing list/ UseNet group where all can comment and criticise
c) A bug progress tracking system
d) Documentation. In KDE, I don't think there are very tight design documents written at the start of the project. Standards have been agreed on and documented, but this appears to have risen organically during the project.
The design process appears to consist of a number of people making a sales pitch for what a program should do, and whether that feature appears or not depends on how much support it gets in the development team, or the single minded dedication of the proposer. A successful Open Source project seems to allow both engineers and programmers to contribute [ I'm using the term engineer in the sense of taking a more formal approach to software design than a programmer]
Donte Alistair Anderson Roberts - hi son!
Karma: Chameleon
This isn't the best way to design software, but it seems to be a common method.
--Shoeboy
Check out this O'Reilly book. There are a few good essays in there about software engineering, in general and specifically dealing with open source. Like "Software Engineering" by Paul Vixie. Hope this helps.
Or maybe Microsofts code is all the code everyone else threw out...
I don't actually exist.
The phrase "predefined headers" is what set off the alarm bells for me. It suggests an exercise in filling out the required tonnage of paper. I suggest filling in the predefined headers with random paragraphs selected from Slashdot discussions, and see if anyone in management notices.
I don't know about open-source development, but what I've found -- even on 1-man projects -- is that the process of writing out a functional spec of design is valuable because it forces me (or my team) to think about the design issues early. That's the real value.
> In your experience, how often are > design documents revisited after the
> project?
Rarely. The main value is in the thinking you have to do in order to produce the document.
> Have design documents helped in > technology transfers (that
> is, have they been more helpful than the > source code alone)?
You mean to other engineers? Definitely. Assuming the document is written well, and with care, and is read carefully.
> Are engineers good at writing design documents?
The absolute best engineers I've worked with *all* produce excellent design documents.
> Have you been able to read design > documents written by other
> engineers?
Yes. I think the best approach is to do brainstorming and then record the important points. Everyone gets to do some writing, and everyone reads what everyone else writes. The brainstorming and documentation can be achieved simultaneously in email, (if the engineers are comfortable with this); or you can brainstorm in front of a whiteboard and write stuff up later.
> Have old design documents been kept up to > date > with the changes in the implementation?
Usually not. But on the other hand, if the original design is good, the distance between it and the implementation will be small.
> Has the quality of your products been better > because of design documentation?
Yes. If I skip design and start hacking, (always tempting), I find that I suffer for discovering important design issues too late.
All that said, the documentation effort has to come from the engineers. If imposed from above it won't work.
I will agree that the Requirements Document is extremely important. It's real nice to solve a problem, but it's much better to solve the right problem. As for the design document, I generally only used a high-level document, and defined interfaces where multiple modules were to communicate, to allow for multiple programers. Also, I maintained coding standards, which included commenting standards. With all of those implemented, I haven't had any real problems debugging (especially with assert standards).
(I was a C/C++, mostly UNIX, developer for 9 years, before switching to UNIX Administration).
-- Keith Moore
This sig is the express property of someone.
I agree that a brilliant programmer or comedian, though I personally don't care much for Jim Carey films, needs a free rein. In my 18 years in S/W development, I've only worked with a handful that I'd trust. For the rest of 'em, give 'em enough rope and they'll hang you.
One of the biggest problems is people solving non-existant problems 'cause they're interesting. Usually writing their own middleware, reinventing STL or some other junk. Control is necessary.
As a Software QA engineer, I can say that design documents are critical. How am I to know that the software is working correctly if I don't know how it's supposed to work? For those that think this is trivial, think back to last weeks "life or death software" article.
Developers don't like doing design documents. I don't blame them for not wanting to write the specs until after they're coded in. But it's important. They keep everyone on the same page, and prevent developers from putting in the app what they think should be in it.
Design documents allow more than one person to code an application at the same time. The smaller the project, the less "formal" the docs need to be, but they need to be there. Even if the app is just a simple text editor, if one developer is thinking "vi", and another "wordpad", you're going to have problems.
A story: Two days after a major application is released, customers are screaming that the autosave feature is overwriting unrelated files. Someone traces the cause down to a combination of autosave and the autospell. PHB calls the tester into his office and demands to know why the bug wasn't found previously. Tester explains that the autospell was never in the specs, and had no idea it existed, and so could not test it. The PHB, the idiot that he is, fires the tester, and gives a bonus to the developer that wrote the autospell.
A Government Is a Body of People, Usually Notably Ungoverned
It's people's failure to admit that they can't plan everything that gets them into trouble. If unexpected things happen in every project, then you should start planning for unexpected things in the next project. Managers who demand that everything is totally specified get to keep their jobs, because their managers think the programmers just didn't do what they were told. Managers who learn from experience and develop their projects incrementally get usable product out the door.
Just don't get religous about it. Brooks' book is pretty ancient, and goes on at length about stuff which should be an absolute baseline for development today. You could get away with reading the summary at the back to make sure you're not completely clueless, then read something more up-to-date.
Most importantly, because it's so old, it talks about monolithic projects in which the developers and managers have no contact with the end-users. People expect much more these days.
However, TMMM is useful for saying to does-not-compute conservative managers 'Look, we're not doing this, and it says in TMMM that we should.'.
First, a word of advice: when a techie argues with a boss, the techie is always right but the boss always gets his way. Appealing to a third party for support is both unnecessary (because the techie is always right to start with) and unhelpful (because the boss always gets his way to end with).
Beyond that, you're right: in most cases following such an anal retentive method is likely to prove to be an exercise in GIFMO (Garbage In, Flying Monkeys Out (or "Gerbals In", if you prefer)).
For most software projects, the best approach is to throw down some code that lets the users get some benefit now, and then fall into a cycle of revisions that follows the constraint that each revision sucks less than the previous.
Admittedly, that scheme isn't the best for special cases such as launch control or life support software -- the revision cycle in such cases would be prohibitively expensive. But notice that even the most rigorous software engineering discipline is no guarantee here either -- think Arianne V. I'll concede that SE can hedge your bets, but overall the result is only as reliable as its weakest link, and the cost of attempting "no weak links" is prohibitive for most development projects.
Also, the throw-down-and-revise strategy is probably not apropos for commercial hypeware, since the goals there are not what most of us would put under the heading of "Good Software".
But for the workaday world, I think the throw-down-and-revise strategy is best. Here are some reasons -
1) I've worked at corporations where most of the day-to-day software that employees actually used to get their jobs done was unauthorized, throw-down solutions provided by some engineer or low-level IS techie at the departmental level.
2) Similarly, I've seen where the most effective big software solutions were simple accretions and evolutions of things arising as in (1). These often attained official status and accrued a support staff, simply because they were so darn useful.
3) But some very-high-level PHB always has a better idea, so ground-up custom solutions tend to be mandated from above. These may be purchased or developed in-house using "good methodology", but the infallible result is that they make the grunts less productive by being difficult to use and generally imposing an electronic bureaucracy (e.g., requires input of information that the intended users do not have, or refuses to divulge information that they desparately need).
All that to the side, let's look at what typically(?) happens when organizations do try the "documentation approach".
4) After some unbounded time in committee, the documents are released and the programmers go to work. Usually the software is already late by this point, at least from the p.o.v. of the people who need to use it.
5) While the programmers are busy at their work, the whole hierarchy of PHBs are off attending meetings of some Revision of the Week club that they've joined, so that the developers are flooded with a steady stream of non-negotiable, undocumented revisions to the official requirements.
6) The program is finally rolled out, but even ignoring (5) it is full of bugs (as all software is), and thus differs from the specs by yet another increment.
7) The program hits the real world as if it were a fan, and customer requests start flowing in. At this point, one of two things can happen: (a) the requests are ignored and the company looses personal productivity and goes broke, in which case no benefit was obtained from working from the design documents, or else (b) the customer requests are satisfied, which means that the software drifts even further from the design documents.
8) The changes in (5..7) are never systematically integrated into the documentation.
9) You end up with a wheelbarrow full of spaghetti code that does not match the design documents and may or may not match the users' requirements.
Which is exactly what you would have had if you had skipped the paperwork and followed an informal release-and-revise strategy to begin with, except that you spent more money and took longer to get there.
YMMV.
AMMCFOOMB.
...which is why I call it a GIFMO system.
Sheesh, evil *and* a jerk. -- Jade
Check out the whitepapers on developer.gnome.org and developer.kde.org. Yup, obviously a whitepaper is quite different from thorough design description, but a few of these docs, especially those related to KOM (KDE Object Model) lay out the costs/benefits and architecture of the chosen solution. As far as the topic of whether engineers can write good design documents, I'd say that many can't, but a few are actually VERY good at laying out all the ideas ahead of time. In those cases, the documents can be quite helpful to others. The most important part, however, is that the document should be a useful roadmap to the programmer who writes it. If you think all the project's aspects through before committing a line of code, sure, some things will have to be changed as the project progresses. But at least you won't end up with a poor hack that needs to be redone from scratch. The final barrier that can exist is that many engineers I've worked with don't have English as their first language. Worse still, some are C programmers and a few are even stuck on Hungarian notation. Ah, I see, "struct _qXrtsh" has a field called "lpszTbsdx", yes, that makes great intuitive sense to me. --JZ
Well, why don't we try to improve on this, then?
> Im more than serious about designing things properly...
>as for the original poster, If you dont design you deserve what you get (and if you worked for me it would be the sack)
I didn't see that he said anything about not doing design; I interpreted the question as "Is it worth it to spend so much time writing it out in prose and polishing the form of the document?".
In my experience, the answer is "no". The documents are frequently copied-and-pasted to fit the form and meet the [the letter of] the document requirement, but rarely get updated.
And just because I don't write it in prose doesn't mean that I don't do design.
I agree with an earlier comment that diagrams on a whiteboard are likely to be far more useful than a formal design document.
This is a poor attitude.
Users *do* know what they want - they're just too used to being *told* what they want, and then having to *ask* if they can do something. Computing and software have reached a point where it is almost unreasonable to say "sorry, we can't do that".
ask users what they want. get a clear picture. make sure everyone understands. !!!!Get it in writing!!!! commit only to what's in writing, and then do the work.
and never, ever take the attitude that "i'm the coder, you're the user. i know everything, you know diddly"
my $0.02
I read slashdot every day, and a linux fan i may well be, but Im more than serious about designing things properly as I expect many people who read slashdot are.
as for the original poster, If you dont design
you deserve what you get (and if you worked for me it would be the sack)
meeting ~ A real alternative to serious work
On my first big project (as an intern mind you, but I think this gives me a bit of "beginners insight"), I learned that documentation takes up about 40% of the allotted time on a major project. We started with UML diagrams, the whole thing taking four coders and a manager the better part of a day. by the end of the week 90% of the UML stuff had been rewritten, resubmitted to managers, and integrated into the project. Each coder was responsible for logging docs on functions/subroutines/algorithms/objects/etc. that they had written (along with a brief description of usage and functionality) into a central database, and everything we wrote was peer reviewed at least once...
Come to think of it, the whole thing seems a bit tedious now, but at the time it was really no big deal, and it got the job done.
... I'm currently involved in a very slow moving project (school affiliated) where the ONLY documentation we have, outside the *extremely* well commented code, is a napkin I picked up at a cafeteria on a DoE site while travelling through Washington state which is currently holding my place about 2/3 the way through Hofstadter's G.E.B.
--mind you, this project isn't progressing very quickly, but, as far as I know, poor documentation has never been a setback.
=+P
Java also comes with a tool called "JavaDoc" that does this very thing - it takes header comments for classes/methods/fields and generates a really useful set of HTML pages documenting the code. I've used it extensivley for a few years now, and it's really about the most useful thing for documenting code I've ever run across.
For one thing, the comments are right in the code so if they start to get out of date you can update them as you work on the code. Also, since the comments are so visible in the HTML docs, it's a lot easier to spot out of date sections than comments that are just in the code. By having one source of information that is viewed in two different contexts, it makes it a lot easier to keep everything up to date.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
I haven't done a lot of team development, but in my experience, there is no way you are going to be able to think of ALL the possible functions/problems in a large project at the start. The "get around a whiteboard for a few hours/days" suggestion was a good one, then write out the plan you come up with. As you make changes, keep a changelog file somewhere, and then when you are all done, draw straws to see who the unlucky bastard is who has to go back and write the new documentation based on the original document and the changelog. :)
Anyways, just a suggestion. There are companies out there who sell system design philosophies and the tools to keep to them, you might want to do a search on "CASE tools".
Design documents and formal processes have both good points and bad.
Sometimes the documentation and rigorous peer-review proves
invaluable. Other times, it simply gets in the way of achieving the
primary goal, that is, to solve a problem with the computer.
Now, I currently work at a large software development company and have
seen quite a number of developers come and go. This is one case in
which documentation of the design is invaluable. Keeping up-to-date
design documents allows new developers--and even QA staff--to come up
to speed quickly. It flattens the learning curve and allows them to
begin contributing to the project much sooner than if they had to
fight for another developer's time or dig through mountains of source
code.
The peer review process helps here as well when the developers coming
on board are fairly green. There are many mistakes that developers
make (and yes, we all made them at some point) which can be
easily caught and rectified with a thorough peer-review and follow-up
instruction.
That said, I have to admit that religious adherence to such processes
does tend to get in the way of producing a product. At my previous
employer, I worked on a government contract where we had to deliver a
formal requirements analysis, design documentation, and subsequent
detailed code reference. Due to the constantly expanding scope of the
project (When does this ever not happen?), the documentation
was never quite up-to-date and was useless to the development staff.
We had our heads in the code and relied on reading the code and
associated comments to figure out what was really going on. The
documentation we generated in the end would definitely prove useful to
later developers who would maintain the code, but for those working on
the first version, it was seen as time wasted.
At my current job, I see the problems that come from constantly
changing requirements. No matter how much we try to plan ahead and
generate design documentation for a new product feature, we inevitably
get caught up in the implementation and the documentation suffers.
With constantly impending deadlines and never enough resources, it
becomes a struggle to keep design documentation up to date. More
often than not, new developers and QA staff end up hunting down the
person responsible for developing a feature in order to find the
answers to questions. Most of the design documents we do have are
horribly out of date or don't have enough detail to answer the
questions at hand. Ultimately, developers still dive into the code to
figure out what is going on.
There are pros and cons to rigorous engineering practices. What I
have learned is that you must find a balance between free-form
creating and hacking and disciplined engineering practices. Where
that balance point lies depends greatly on the people involved as much
as the project itself. Take a good long look at your developers and
how they have reacted to processes in the past. Take a look at the
software you need to develop and figure out how rigorous you need to
be in developing it. Look at your schedule and see how flexible you
need to be to get things done. Pick a point somewhere in the middle
and give it a shot.
If you've taken a degree either in engineering or computers then you've bound to come across courses in software engineering. Personally I find such courses very enlightening, especially from a software management point of view. I don't see why someone who's qualified in say history who haven't taken such courses be eligible to use the term "software engineer". I wouldn't call myself a historian if I read history or watched X hours of the Discovery channel.
At the end of the day who'll takes responsibility when something goes wrong? This is especially of concern when you're writing mission critical applications...not web servers but e.g. software for medical equipment, vehicle control systems etc. I personally hope that professional groups would properly endorse qualified software engineers (as they do qualified Electrical and Mechanical engineers etc.) and protect the term "engineer" in general. The last thing you want is people misusing words like doctors, surgeons etc. so why not add "engineer" to the list.
I mean come on.. something's funny here.. I've been reading /. for almost 2 years now and in the last month or two I've seen somewhat random Linux bashing comments?
.. stop it!! hehe) That would be funny.. some poor co-op student at M$ being forced to flame and spread FUD the web for a whole term .. hhehe .. that would suck !
Is this the product of the MS anti-linux team (heheh hahah lol!! ohh
I've never done open source development, so I can't speak to that question. But that hasn't stopped anyone else here from giving advice, so I won't let it stop me either. :-)
First of all, and most importantly: Don't use MS Word. I'm serious, it's not just random M$ bashing. MS Word is not cross platform and can't be read on any machine that doesn't have M$ Office installed. You don't want to have to leave the server room to check a document spec. (Even NT servers ususally don't have M$Word installed.) Besides, it seems like whatever version of Word that you use is the version that the client doesn't have yet. HTML is the best document format that I've used. It's cross platform, degrades nicely when people have different tools and versions, and best of all can be posted to a shared webserver for ubiquitous access.
Second, know what purpose each piece of documentation has. Generally, I've found three types of documents although there is some overlap.
Just MHO.
From least important to most important:
(1) code
(2) data
(3) how you will tell if it works
(4) interfaces
(5) motherhood
Motherhood is things like: is it supposed to be fast? Or small? Or intuitive to radio astronomers? Or prototyped by Friday? When push comes to shove, what goals win? This is the crucial stuff.
Interfaces are the next most important, because chunks that use well chosen interfaces can be reimplemented without causing a disturbance in the force.
If you want to successfully implement a decent sized project, you have to spend some up front time writing design docs. I don't care how talented you are, it won't get you around this.
I'd rather lose some brainpower on my team rather than have to scrap the whole project because johnny doesn't want to tell everyone what he's going to write.
I think that software companies should do what FSF and the Linux did all along, write code to simple and readable that anyone can change it if they need, and a README.
Most commercial code is unreadable compared to free code because free code was designed so that any programmer will understand it fairly quick.
Ofcourse most free software (atlist gnu's) isn't object oriented which makes it much much more readable as you don't have to read lots of classes before you understand what's going on.
Ballerinas have fins that you'll never find
Here's what we did at my last job:
:-)*
- Write a short design document, pass it around to everybody on the project (I was one of two software/electronics guys, the rest were machanical, marketing, and the president (small company)). Everybody adds their $0.02. President gets final word and says, "go with it."
- Design, build, and test first prototype (hardware and/or software).
- Update design document for any variations or additions.
- Show off prototype at trade show.
- Repeat for each prototype (we only did prototypes, it's up to our clients to do full production, etc).
- Tack on test results (they let me use GNUPlot, yay!), pictures of test apparatii, etc.
- Frequent trips to Radio Shack/Electronic Center/my house to get that part or tool we don't want to wait two days/weeks for.
- Finish workterm and go back to school
That was pretty much it. I hated the design documents because it was coding time/electronic design time wasted. However, clients don't like seeing nothing but the product. Design documents are necessary for a company's viability.
I disagree that the question is too big for /.
Translating user desires into working systems is one of the major challenges of this information society. Its been difficult for about 50 years now, and innumerable books have been written on this topic. However, it all boils down to this:
1) Find out what your users _really_ want, and _why_ they want it. (The latter can allow you to modify the former, if you really understand it.)y
2) Prototype internally, and get your coders to commit to the feasibility of what you've planned.
3) Show users a prototype (story-board, expected screens with process flow, even detailed design). Get the users' feedback and reiterate 'til done.
4) Freeze the design! Don't let the users ask for "just a few more features" they'd forgotten to include in their review of the prototype spec. Tell them these are good, but will be in Rel. 2.
5) Deliver what you prototyped and promised. If you do this, you can't be reasonably criticized. If you deliver more than is expected, better yet.
WRT (5), its always best to underpromise, then overdeliver. This is how to be a software hero.
I have over 25 years experience with many roles in commercial data processing, including Systems Programming and large-firm Management Consulting. This is the best advice I can offer succinctly.
Programmers cannot write good English
Maclir's second rule of software development:
Documentation never gets written
Maclir's third rule of software development:
Any design documentation you find will be (a) Out of date; (b) Plain wrong; (c) Missing the five crucial pages that explain the really tricky bit; (d) All of the above
Seriously - A project will fail if the initial design and requirements is not documented. This does not meant a 400 page perfect bound thesis, completet with charts, diagrams, pictures and the whole shooting match. But - unless you and the client can express the requirements of the application such that it can be written down, and understood by someone else - then you are in big trouble.
But - and this is the big but - in general, the person / people who write the code are not the people to do the design and document it. You are looking at a different set of skills, and a completely different approach to the task.
Experience has taught me this - and experience is a brutal teacher.
maclir
Extreme Programming is a sort of minimalist approach to software development, but that doesn't mean it requires no discipline.
At the heart of Extreme Programming are the following practices (from Wiki):
For some great resources related to Extreme Programming, visit Ron Jeffries' website.
> But who needs more than that?
It depends what you are doing. But having a good documentation of your software is really handy, when you have to fix the program some 5 years later. Happened too often to me.
Pseudo-code and flowcharts are just crutches. They are are real big help when writing complex parser, but you don't need them for the easy stuff. (Most stuff you can do in a week comes under easy). The same goes for UML, Jackson- and other diagrams. You don't need them for you stamp collection database, but when the diagrams are big and complex enough to make a nice wallpaper, they are indispensable.
By the way, the toilet is a real useful place to hang them. Best at eye level.
(Does anyone even read the 229th comment?)
I had some thoughts on software development models I put together
here
NOT!!
I am a leaf on the wind
i'd recommend a few high level UML diagrams to describe the program flow etc. Use TeX or soemthing for templates...Lyx is very good at producing printable postscript and keeping a standard template. Dont go overboard...less than 20 pages of high level diagrams is good enough for projects containing 500,000 lines of code. Any more and theres a serious drawback/tradeoff. If you have several million lines of code then scale the number up keeping the ratio the same. As a rule of thumb, communication is more important than any high level design details. Keep standard templates for function interaction (i.e. standard data passing) and standard code documentation style (a short paragraph of comments before each function is fine - dont comment each line...developers arent stupid) ..In general try to be sensible and dont reach for perfection..you'll never get there. Also use CVS or some sort of collaborative method of data sharing...couple of in house mailing lists and bugzilla style things also help. have a look at mozilla - great way of organising a difficult project.
And if you have this documented and follow it, then you are ISO Compliant, and you will have plenty of work because customers will be impressed with the ISO Certification.
Steven Rostedt
-- Nevermind
I've worked on dozens of s/w projects ranging from solo assignments to enormous multi-headed monsters involving hundreds of people doing coding and validation. I've noticed a method which seems to work fairly well for a variety of project sizes. Of course, YMMV.
1. Don't get tied up in writing a bunch of specs up-front. The spec-it-all-first method almost always results in sub-optimal implementations and reams of out-of-date documentation.
2. Get a handful of senior coders (and one or two customers of the product, if possible) in a room and have them architect a skeleton of the project on the whiteboard. Get a junior coder to take notes. These minutes are your first "design document".
3. Figure out how to partition all of the pieces from #2 into nice libraries and data structure definitions. Define the basic functionality of the libraries without specifying the exact interfaces (which will be wrong for the first two iterations).
4. Prototype and re-prototype the implementation, allowing the developers to work out the interfaces between themselves. Get customers to try out the prototypes.
5. Schedule a "get well" phase of the project where code is cleaned up and interfaces become mostly frozen. This is the ideal time to document the "final" interfaces, data structures, and algorithms. Hold peer reviews of code and documentation. Discipline is vital here if you want a documented design.
6. Productize.
7. Maintenance mode. If you can find a way to keep the documentation up to date from this point on, please let me know how you do it.
While any document which isn't auto-generated directly from the code will become out-of-date, getting most implementation details figured out by trial and error before writing an official document will make it more accurate with less maintenance overhead.
I've seen a lot of projects fail miserably when management spent too much time trying to spec everything up front. One group in my company has an offical flow which goes something like this:
1. Write an Market Requirements Document (MRD)
2. Review the MRD with a lot of people, many of whom have no idea what the market really needs but have a lot of opinions about how the MRD should look.
3. Revise the MRD based on the review.
4. Write an External Product Specification (EPS). This defines every feature which is to go into the product that is visible to the user (as well as how it is visible to the user).
5. Review the EPS with a lot of people.
6. Revise the EPS and get "buy-in" from all of the
"stakeholders".
7. Write an Internal Product Specification (IPS). This defines all internal modules and most of the data structures and algorithms. Note that usually this is published before any code has been written.
8. Review the IPS with a lot of people.
9. Revise the IPS until everybody is happy.
10. Start writing code (usually about 4 months after the process has started).
11. Find out that the customer has already started using another product from somebody who has followed the first method above.
12. If the other product is from another group in the same company, try to kill that product via political means.
13. Finish coding up something which is grossly inefficient and doesn't meet the customers' real needs, but that meets most or all of the items in the MRD, EPS, and IPS.
Ok, maybe some of that isn't in the offical methodology...
Finally, there are many factors besides design documentation which help to make or break a project (having talented people seems to be most important); so pick your battles if you're going up against management edict.
I think the benefit of a formal design methodology is a function of two things:
1) team size / size of project
On a large team, the diagrams & documents provide a roadmap for the team to follow, and they are essential for the purpose of keeping everybody on the team gainfully occupied.
On a small team, the full "process" may be serious overkill, and it should only be followed to the extent that the team considers it helpful and not a waste of time. For example, if some UML will help hash out the key design issues then by all means do it.
2) project complexity and team's amount of experience with the problems/tools/technologies involved
If there are a lot of difficult design issues on the project and/or the team lacks experience in some key areas then you'll need to be more rigid in following the process. This will take some extra time but it will reduce the risk.
On the other hand, if the design problems are very well understood or you're able to re-use a lot of code then you can save considerable time by avoiding a lot of the process.
***
Unfortunately, the process often becomes an end in itself, and it's followed on every project regardless of whether it's helpful. It's important to remember that good code is the #1 goal, and everything else you spend time on is in support of that goal.
I'm employed as a software engineer at a major international telecommunications firm. In order to keep my job, I'm not going to say which -- but if you live in America, it's decent odds that we're your long-distance carrier.
... well, office politics ensues.
Around here, software development is an extremely procedural thing; it's almost as if there's an algorithm in place for software development. Unfortunately, this algorithm is buggy as all hell. I'm detailing it here so that (hopefully) others can avoid the same pitfalls.
1. An MBA type has a meeting with a client, at which point the client tells the MBA type what they want. The MBA type makes a judgment as to whether or not it's feasible, and how much it will cost.
2. The MBA types up a Requirements document, which (in theory) outlines only what the client requires from the software package. In practice, every MBA thinks they're qualified to make technical decisions, so at least one or two bits of brain damage pop into the project here.
3. The MBA sits down with representatives from Development (who writes the code) and QA (who verifies the code) and goes over the requirements document. Theoretically, Development or QA can veto the project at that point (if it's impractical to code, or impractical to verify). In reality, the MBA is above them on the food chain, and if the Developer or QA representatives veto it, someone's ego will get hurt and
4. Development is required to write a Design document which outlines how they're going to write the software project. This is a lot more than an 8x11 sheet scrawled with a few sketches and diagrams; design documents run fifty pages at the minimum. A recent project a friend was working on ran to 400 pages and took up two three-ring binders. Everything must be specified in these Design documents; if it's not in Design, then it doesn't exist.
5. Development meets with QA and the MBA-type to go over the design document. QA can veto the Design document if QA feels that it's designed in a way which would be, well, hazardous. In reality, this never happens. The developers and QA people are generally pretty cool (a lot of them are hackers), but the problem is by this time the deadline is approaching and there isn't time to come up with a new design.
6. While Development is writing the code, QA is writing a Test Plan. The Test Plan is a bridge between the Design and Requirements document; it explains how the Design is going to be tested to verify that it meets Requirements.
7. In theory, the Test Plan is finished a few days before the coding is. THIS IS AN ENORMOUS MISTAKE. There is no good metric to use to plan how long a software project will take. The rule of thumb is 100 lines of code per coder per day, but there's so much variation there that the rule of thumb is about useless. In practice, the Test Plan is usually finished considerably before the coding is -- and the few otherwise occasions, the coding is finished weeks before the Test Plan is.
8. QA sits down with the MBA-type and the Developer representative to go over the Test Plan. Development can veto the Test Plan if they feel that it's not adequately testing the program. This, of course, never happens because by this time the deadline is looming.
9. Development hands off the code to QA. QA gets to spend a week getting the damn code to run. (No, I'm not kidding. More than half the time the code QA gets will not execute.) QA bounces the code back to Development. Development fixes it and bounces it back to QA. Repeat this dance a few times until the deadline is in your face.
10. QA rubberstamps the project. QA never gets to look at the source; QA never gets to check to see that every malloc() is free()d, that every pointer is accounted for. All that matters is (a) deadline gets met and (b) Requirement gets fulfilled. Good code is purely optional.
11. The code gets shipped out the door.
Actually, I generally find that when I try to develop using this model, it takes one more iteration of throwing the code and design away and redoing it before I have something I'm reasonably happy with.
GNU and Linux -- Oh no, Mr. Bill!
No, no, no. We don't write about "What kind of design process do the successful free-software projects have (Linux, gcc, Apache, XFree86, etc)" in M$Word format. You could at least start off on the right foot by using WordPerfect (the free download version, of course), or anything BUT Word. :)
~REZ~ #43301. Who'd fake being me anyway?
Because of the way my job has turned out (insane web startup-turned-public), I have on two or three occasions had to write fairly large applications (Java with 100-200+ classes) mostly by myself, which later have other programming resources added to them. I have observed that my own design process is like the bunch of engineers getting together in a room to hash it out on the board, except there arent any other engineers around, so the same design process occurs in my head. If there is pressure to get something accomplished in the next 4-6 weeks, I will get very deep about it, think about it all the time, drift off into space at social engagements, etc. Typically I will prototype the difficult/critical/hard-to-visualize sections, which brings hundreds of more issues to light (im not sure how large design processes work without bunches of little prototype/pseudo code to keep the ideas grounded in reality).
On the occasions that I have tried to write complete design documents ahead of time, pure abstract interfaces with tons of comments, etc., I find that for me many important logistical issues fail to come to prominence at that stage, so when the implementation begins the interface has to change massively anyway. I have yet to see how *the* design can really be worked out by 10 people and a whiteboard with no prototype/pseudo code whatsoever...at least something that could be working in 8-12 weeks. Most designs seem to have some degree of iteration in my experience.
My own (obviously flawed) process, i.e. "iterative brainstorming", causes the most headaches when other resources are brought in to contribute and the documentation for the code is incomplete, and whatever documentation exists is very out of date since it has been snowed over by 1 or 2 iterations. I have done it with groups of 4-5 people, with one person maintaining class diagrams, a DBA handling table schemas, and lots of verbal explaining in conference rooms. In these situations its good if the programmer can bite the bullet and write some comprehensive docs, or at least an overview of key data structures and methodologies. The act of actually writing it forces the "thinker" to verbalize his or her thoughts much more clearly than just speaking them to a small group. I'd love to someday have most or all of those docs written ahead of the code, but it seems like you need a team of extremely smart people (all experienced coders) and a few months of lead time for this to be feasable.
My current design process involves many "mini whiteboard" sessions with 2 or more people to build sections of an app or perform partial reworkings of sections to allow for a new feature, while trying to keep internal and external docs current for what is currently working. Internal code documentation is always done as simultaneously as possible to actual code, although higher level "per-class" documentation lags behind more than "per-method" or "per-line" code, which I focus on the most while coding since it is the most difficult to revisit later. I fortunately have usually had a resource to help build external design documents.
Programmers can certainly write their own design docs, but the constant friction with this issue is caused by our habit of thinking ahead of the design far too quickly...the natural instinct to race against time and bring forth yet the next iteration of fantastic new ideas that obsoletes all the docs you've been plodding away at is huge, particularly when your company's business model is based on superfast turnaround of code.
Once apon a time, I used to teach this stuff. Doing it is much more fun.
/. as many other respondents have previously noted.
Design Documents are just part of the whole theme of a well engineered software system. Much depends on the system being constructed and what it needs to interface with. I've written and used extremely detailed design documents (this BIT goes here, that BIT goes there) for a project where all the pieces were being implemented in different geographic locations. When the thing was assembled it even worked! On the other hand, I've been gratefull for a very brief, high-level 'how this fits into the big picture' design statement that described a report system that just sucked from zillions of different data sets that were previously defined elsewhere. So it depends on what the goals of the project are.
In 'open' software, I'd much rather have sections of documentation inline with the code that say things like, "At this point we know XXX and we're now going to YYYY." Tell me the high level story as it is less apt to rot over time compared to the minutia.
Finally, Design Documents that are used will be updated. DD that are just worthless trash from the outset, no one will care if they further get bit rot.
If you have not read Fred Brook's book, "The Mythical Man Month", now would be a good time.
Your question certainly is bigger than a ask
P.S. Revision control and MS Word documents? Doesn't the constant change of format of MS doc's cause you to worry that your documents will not be readable later in the products life? I'd be using "Plain Old Text" as ISO-Latin1 is not apt to go out of style anytime soon.
I'd suggest two different words, ``Literate Programming''.
Dr. Knuth's work on this is wonderful, and if concepts like it had been more prevalent in the computer science course I took, I'd'be likely managed to've gotten a second major in CS in addition to my degree in art/graphic design.
William
--
Lettering Art in Modern Use
http://members.aol.com/willadams
Sphinx of black quartz, judge my vow.
Quit your job and go to work for a small company where they don't waste time with crap like this.
-- $SIGNATURE
I've worked on projects with documentation out the wazoo (a B3-targeted trusted operating system base ), with no docs but good comments, and with no docs and useless comments. (Anyone who comments "x++;" with "/* add one to x */" needs to be severely beaten about the head, shoulders, and typing fingers with a copy of the ANSI C specs.) Too much documentation is almost as bad as none at all, and poorly written docs can be worse! Based on my experience, my advice is:
- Lose the MSWord!!!!!!! No data of any relevance should ever be stored in a proprietary format, especially one that can introduce macro viruses and leak unintended data. Use plain text and HTML (LaTeX can also work well, if you're Unix-geeky enough, and I look forward to seeing what XML may offer here) and GIFs or Postscript for pretty pictures if need be, and put your docs on a local web server. This also makes you platform independent.
- Source control is essential for both code and docs. So is some kind of bug/change tracking. I'd avoid "integrated" solutions and pick the best individual components for your needs. I like CVS and GNATS.
- There's basically four things that need to be documented for a software component: requirements, high level design, low level design, and interface.
- Code reviews. Formally or informally, have developers look at each other's code to check for problems and make sure that it is documented and maintainable.
Someday, you will have to explain what you code. It's much more accurate, useful, and pleasant to do it up front and get it out of the way then to have people come to you with annoying questions later on when you've moved on to other, more interesting things. (A friend of mine once compared the design and documentation process to foreplay...it helps get you excited about getting down to the coding.)Don't make either system hard to use. Give senior developers the access rights needed to fix problems - don't limit it to one or two designated source control people. A bad source control setup can waste days of programmer time very quickly.
- Requirements. For the system as a whole, and perhaps for large components, we ask - what does it have to do? Document reqs up front in a collabrative process that includes developers.
- High level design. From an outside perpective, what is this thing? What does it do? What does it model? What is the abstraction that it implements? This should also be documented by developers before coding begins, in separate documents. The design may change as development progresses and the problem becomes better understood, and these docs must be kept current. If you're heavy into OOD, this is where your object diagrams and such go.
- Low level design. How does it do what it does? What other components does it use or talk to? The key question is, what does someone developing or maintaining this module need to know about it?
- Interface. APIs, message structures, data types, etcetera. What does someone using this module need to know about it? I think that the definitive version of this information is also best kept in function and module headers, and extracted into documents at stable points in development.
Having a tech writer around to make the grammar pretty and deal with the formatting is a very, very good thing.Once requirements are frozen, they are frozen - that should be clear in the contract, if applicable. The only exception should be clarification of existing reqs. Avoid creeping featuritis. (IMHO, developers really need to start standing up on this point. Sometimes, we have to tell management "No, you can't have this." If speaking the truth and standing up for good development principals gets you in hot water, maybe it's time to walk.)
HLDs are very important for new people coming on to a project, or for developers taking over a new component, to get up to speed. Creating them also helps develop encapsulation and data hiding.
Because this changes rapidly in the course of development, and shouldn't have to be known to other modules, I think this is best kept with the code in function and module headers (structured comment blocks).
If you can't explain your code, you shouldn't create it in the first place.
Tom Swiss | the infamous tms | my blog
You cannot wash away blood with blood
Get the book The Mythical Man-Month by Frederick P. Brooks - the one in the brown cover with La Brea tar pits on the front. This timeless classic offers pragmatic advice about how to organise a software development project, and is excellent refutation for the now all too common PHB school of thought that project management consists of using MS-Project to produce a ton of Gantt charts.
The book is based on the author's experience of running the (world's) first operating system development project at IBM. It has a few things to say about documentation which may surprise you.
I guess my situtation is slightly different than yours, but when I'm coding for myself, screw documentation.
Adam Berlinsky-Schine
Requirements Documents. Can't live with them, can't live without them, can't ignite them and dance around their burning ashes.
::Loud screams and coders jumping out windows::
The requirements document of the project I just spent 80 hours a week working on for the entire summer was a source of so much grief that I get queasy just thinking about it.
Basically, the customer handed us over 1000 pages of documentation and said, "This is the best requirement document that has ever been written. You shouldn't need to ask us anything about it, because it's so damn PERFECT!"
Needless to say, it was NOT perfect.
It looked all fine and dandy on paper or on design flowcharts, but actually trying to write compliant code was a nightmare and a half.
But wait! Here comes the frustrating part!
Every time we tried to talk to the customer about something we needed to implement that wasn't in the requirements document, the response would be...
"Read the requirements document!"
-
The biggest problem I find in projects I have worked on is developing with reference to original requirements and design docs. Once you "parcel out" everything, people go on their merry way, and as long aws a component has the right polish (superficially appears to do what it needs to), some team members figure the work is finished. Or there's the "works right twice in a row" criterion for finishing.
Here's where the whiteboard-only approach is a BIG problem. Who is doing what? when are they considered finished? what criteria are there for completion? what dependencies have you introduced between sub-projects? If the only reference you have is the whiteboard and a few e-mails, you are making the accountability and completion part of your process potentially more difficult than it needs to be. This is a difficult bit (and often overlooked) even if you have a details project plan and lots of supporting documentation.
One reason flowcharting, etc., seems really anal to a lot of developers is because they have always been lucky. Once their streak of luck runs out, they may not be around any more to warn others. Another reason a lot of people object to rigorous design planning is because they have seen process used a a political tool in their organization. This doesn't invalidate the process, it just means that your organization has too many jerks!
A concept that has worked very well for me is using "non-functional" prototypes -- in other words: demos.
;-)
The problem I had to solve was an Intranet database project. Just "drawing" a front-end that looked and feeled as if it worked (using a graphical HTML editor like Netscape's or MS Frontpage) helped a lot. On the demo you can simulate how the thing will feel for the user.
I wouldn't recommend MS Frontpage for any serious web page design, but for those quick hacks that don't have to be compatible with older browsers it works very well.
AFAIK there are unfortunately not too many open source graphical design tools that offer similar functionality, either for web programming or "serious" coding.
Of course this approach only works if your program is going to have a GUI
... and some pre-prototype paper-based design still is recommended.
I recently began coding a module for Apache for the first time, and thus got to find out how well the API is documented. I'd say about half of what a programmer needs to know is formally documented, and the rest you have to get by looking through the include directories and at other people's code.
The HTML manual pages in the distribution include a section about the module API, which is quite thorough in some areas, but some of the sections have so far not got past the author's outline. Naturally, some of those incomplete sections cover topics I needed to know more about over the course of my project. I'd say this document is a pretty common specimen in software engineering: The author probably spent as much time as he could with it, and did a very good job in some parts but didn't have time to finish the rest, and it had to get distributed with the software in this incomplete state, or else the distribution never would have happened at all.
There's also a mod_example.c with illustrations of the various handlers. Working sample code like this is a great help, obviously, but mod_example.c isn't perfect either, particularly because it doesn't seem to have been updated for API version 1.3.
All of this helps you understand the general architecture and philosophy of Apache server configuration and request handling, but you'll never get a module written without looking at other module code and perusing include/*.h. For one thing, most of the really useful explanations I've found are programmer comments in the source code. Also, this is the only way you can find out about the useful and tested function calls in the package. Apache has a rich supply of useful code, for everything from MD5 hashing to output formatting to iterating over the request headers and who knows what all, and I've often caught myself trying to re-invent some wheel or another before I realized that another Apache developer had already got the job done.
So what can we conclude about this? The Apache documentation for programmers could clearly be much better, but that has evidently not prevented it from being a remarkably successful project -- and the module API is one of the principal reasons for its success. I guess this is just a fact of life. Good software cannot be completely undocumented, but in all likelihood the documentation will be spotty, very good in some areas, incomplete and out of date in others. The project programmers will always view documentation as a secondary priority. If you really need to know exacly what's going on, there is no substitute for reading the source code.
Always keep a sapphire in your mind
Step 1: Screw design, charge right in. Start coding immediately. Throw it all against the wall and see what sticks.
/dev/null before I even think about real design. The designs I've been most proud of were always version 2.
Step 2: Type "rm *"
Step 3: Write the design document.
Seriously, I've been amazed what you can find out by simply trying to code something up. I find that I can avoid so many horrible errors by throwing up some code that I know will go to
The cake is a pie
In my experience when organisations get religion about documentation, they tend to produce lots of useless documents. The key documentation to produce is the maintenance manual and a users' guide. Keep both short and up to date.
Historical documents are useless eg those produced by various project phases, except for butt-covering in bureaucracies.
At the program level, document the interfaces to the programs and the expectations and assumptions, again quite concisely.
Some great reading on development is at c2.com/cgi/wiki?ExtremeProgrammingRoadmap.
Is formal algorithmic proof actually a waste of time? I spent a lot of time studying at degree level and although I have never found any use for it in a commercial situation (I have never worked on safety critical systems, for example) but I am sure it has some use... doesn't it?
From my partisan position as an interaction designer, I have to say that the creation of a seperate interaction design team is the best thing that you can do to solve these kinds of design problems. The interaction design team should contain only a handful of people (no more than four, and preferably less), should do its work before coding begins, and should have a mandate to create a behavioural spec.
That should help resolve the problems raised above:
1. The interaction design team can really focus on users' needs and perceptions.
2. Good interaction designers know that users ask for features when they are in pain, but that users don't need the candy they asked for -- they need a nourishing meal.
3. Amen. The design team needs to look at the end users, not the purchasers of the system.
4. The design team's behavioural spec gives you something to sign off on, to control the project.
5. Too many projects have arbitrary schedules because managers have such blunt tools for trying to keep control over the project. After you have a spec, then it's possible to create a realistic schedule.
6. Again, the spec is a good guide to how you can break things up into chewable pieces.
7. Again, the spec is the earth that you keep in the window!
I think a lot of the free software products don't actualy have design criteria that are stringent and as such don't require documents.
To a certain degree I believe this is a good thing. Commercial software is written for a specific purpose and the product must conform to that purpose.
In the free software world as long as it performs _a_ function it should be enough. If that function conforms to what you want then you use it, otherwise you find somthing else.
The free software need not live under the fear of inadequate programs. If somebody wants it, there is quite often a programmer who wants it too. If the program that that person writes doesn't fit the bill there will still be other programmers out there who are inclined to do it themselves.
Consider someone writing a ICQ clone, some of my friends want something that does just the who-is-online and messaging. Others want chat, file transfers and any number of weird features. I'm sure sooner or later, of the many clones in development, there will be programs to suit both types, and a bunch of crappy things that hardly anyone uses.
One of the most important features of programs with no fixed design is that they can become something different to their intended purpose. I'm not talking a complete genre shift but a change to fit a slightly different field.
As an example, if someone was writing a paint package intending to do something gimpy and they provided some powerful cut and paste featurs that were really easy to use. They could mid-development change tack and use the powerful features as the central basis for the application
I feel the heart of good design comes out of one or a few people doing something that they wanted to have Mark Kilgard(glut), Linus(some os thing), K+R(stuff) and John Carmak(commander keen). Once they had what they wanted others went hey, that's what I want too.
For all those people there are plenty of others who have made things that no-one wants (my turing machine emulator for one).
-- That which does not kill us has made its last mistake.
Just out of curiosity, why do you think CMM is totally anal? In your opinion?
Saluton Marko,
While Marko has reason to believe that I may not be a completely typical programmer in terms of literacy, I will assert that some programmers can write. There are two very important skills to consider for a designer. The first is the ability to actually produce a good design. Nothing will eliminate the need for it. The second is the ability to communicate it.
Now let me suggest how you can achieve useful results. First, set the goals for the design. It serves two purposes. The first is to communicate the information that the programmers writing the code need. The second is to specify how the requirements are being met. That indicates two different sets of reviewers. Write to both audiences, not necessarily in every paragraph. The high level portions of a design document must indicate what the requirements are and which portions of the code meet each one. The lower level portions need to specify other things that programmers will need. The one thing that should always go into formal documentation is the interfaces, both between major components and with the outside world.
My final piece of advice, and I admit that I have been terse, is that no design document should be considered final until it has been reviewed again when the code is complete. It should reflect last minute additions and changes because the information in it should serve as a resource for maintenance and support.
- Dale Gulledge
The net will not be what we demand, but what we make it. Build it well.
I object to your first rule. Programmers CAN write good english. Programmers HATE to write. Let me clarify a bit. All programmers hate to write. Some have the ability to write good english.
:)
I agree with your second and third rules though.
The last company I was at was big into design, they went through a serious process and sometimes the people who wrote the design were the ones that wrote the code.
This company has a different design process. It goes something like this:
"We need the software to do X."
That usually completes the requirements phase.
The design phase is more strenuous, and usually involves a meeting, and sometimes a document. I don't usually write the document.
I don't know which approach is better, but I know I am capable of writing documentation, I just don't want to.
The other difference is that the previous company I worked for produced software for a client. I agree that good documentation is a must in this case. The company I work for now is producing software to run our product, which means that the software has to meet the needs of the instrument we hook it up to, and gives the software a baseline set of requirements to work from.
>>>>>>>>>> Kvort
-Don't mind me, I'm personality-deficient and mentally-impaired.
>>I don't know which approach is better, but I know I am capable of writing :)
:)
>>documentation, I just don't want to.
>Those who can read but don't are no different from the illiterate.
> --Mark Twain (badly paraphrased)
Good quote. I have a copy of Clemens' autobiography at home, I should see if I can find that.
Unfortunately, I don't feel that it is applicable. Not only do I read a great deal (3-4 books a week) but I occasionally write as well (not very well, as was pointed out earlier). The discussion revolved around technical documentation, which is a far cry from literature.
The major difference between the two being, in my estimation, that technical writing contains only information (for the most part) while literature contains more, such as meaning, wisdom, and knowledge. (Oh, and humor. We must not forget humor.)
I really wish I had a good comeback quote.
>>>>>>>>> Kvort
-Don't mind me, I'm personality-deficient and mentally-impaired.
"A yeer ago I cudent' speel engineer, now I are won."
I do hope someday to be able to return to the halls of knowledge, so that I may learn to properly command the english language, and prove myself an asshole by correcting faulty english written by programmers.
No offense intended, I have proven myself an asshole in many other ways and wear the name as a badge of dishonour.
To speak the truth, I actually do want to return to acadamia someday and learn many of the things I have missed in the time while I was trying to get my degree. English being one of the major subjects which I neglected.
I apologise to those who offended by my profanity.
>>>>>>>>>> Kvort
-Don't mind me, I'm personality-deficient and mentally-impaired.
Linux, gcc and apache all had clearly visible targets
You got it. Cloning exercises are easy in this regard because the requirements are already defined.
Some advice from an old dog: if it's not a clone project then a clear, agreed description of the requirements is the single most important piece of documentation. It'll be a living document so you'll need an agreed proces for churning the requirements, too.
The next most important doc is the HLD. It should point at the requirements and show how the propose d design will satisfy the requirements.
The actual design documentation is relatively unimportant. One of the tools which generates it from the source comments would be appropriate.
No programmer likes to do documentation. I hate it myself. However it is a crucial step in order to upgrade the product after more than one iteration.
I get typically called in after budgets (Time and monney) are gone way over. It gets very hard to get productive if There is no documentation. Now don't get me wrong: I don't need a lot! Get me a Use case scenario, an UML diagram and a list of all functions, procedures and types; what they do and what values are expected in return and I'm set.
The other extreme from no documentation at all that I've run into is the one where people ran over budget because all they did was document! I've run only into this once and it was much easier to remedy than no documentation at all.
Hajo Monogamy: Belief so strong that millions of people end perfectly good relationships in order to start a new one.
Better than it could be...
Running on less hardware than it might have...
Where I work we are at the end of a project,
and are just starting to make use of a program called 'doc++' (see freshmeat.net).
There must be other tools like it (I believe that the GNOME people have something similar that creates SGML docbook documents). It takes inline comments in c/c++ code, above each declaration, and produces a design document (HTML or latex). This could be useful in the design process if used from the beginning (as suggested above) with a change log, and everything stays together in the code.