Slashdot Mirror
MIT Studies Software Development Processes
IsoQuantic writes "A new MIT study (pdf) looked at SW development processes around the world. One striking difference that the researchers found for U.S. developers is the relatively small use of specifications before development begins. I can already hear my EP-zealot colleague chuckling in the cube next to me. (sigh)"
Specs!? Specs!? I don't need no stinking specificiations!
I used to work in a very loose development shop. The only specifications that were ever written down were protocol specs - and even those were often "documented" in the form of a header file.
I found some parts of this method usefull in that the specs were often written as as pseudo-code comments, and the actual code would be filled in later.
However, eventually the development pool grew, and we got a few folks who couldn't follow this method, and we lost several weeks of work. After that standardized specificiation paperwork was produced for every project from that point on.
Perhaps it's a lesson everyone will eventually learn. Perhaps I'm being an idiot.
Kinetic stupidity has a new brand leader: Allen Zadr.
Outsourcing these jobs should fix all these problems.
I Am My Own Worst Enemy
In my software engineering class, my teacher vehemently states that Requirements are the Enemy of Design. You need to have an idea of what you are doing for the project, but you honestly cannot know how much space it will take, how fast it will be, etc. Its sheer folly. And who isnt to say that a customer may realize that they want it differently as the process is going along. Design is dynamic, always growing and changing. And the Open Source Community best represents this, because a project never ends, but continues to develop in a myriad of directions.
je suis parce que j'aime
when working in R&D, the specs is always a step behind the cutting edge...
Our managers don't GIVE us development specs, or keep changing the specs every 5 minutes so that a formal document is worthless.
Or, in my personal experience, we stick to a formal document for 3/4 of the product then get hit with feature creep for the last 1/4 which makes the product late, buggy, over budget, etc, etc, etc
Sure, I've tried instituting "processes" and management's alwasy keen on the idea. But when push comes to shove, >poof.
The only time management ever stuck with a process was the medical company that, by law, required governmental oversight that demanded process. And you don't want to know how much we skirted process anyway. (Most of the times we built the product first, then wrote the "planning" documentation second.)
MIT used to be so creative... what happened...
I have learned the same lesson. I hate bureaucracy as much as the next guy, but I really like having a good specification. Most of the programming I do is related to various forms of messaging, and having a detailed spec containing
/var/log/myapp.log' :-)
a) The purpose of the integration (business concepts)
b) The protocol
c) Examples. Lots of examples
makes the whole process a lot easier.
Especially the business concepts are important. They allow me to foresee where changes and extensions may occur, and I then put more work into those parts.
If you're fortunate enough to have a good project leader, use him to communicate with the other parts involved and make him also document all the changes in the protocol. That will save you a lot of time on the phone and quite a few 'tail -f
Interestingly, Cusumano and Selby observed a decade ago that Microsoft programmers in general did not write detailed designs, but went straight from a functional specification to coding in order to save time and not waste effort writing specs for features that teams might later delete [9, 14].
Cue a lot of M$-related jokes and M$ bashing!
http://efil.blogspot.com/
In a small team - say 3-8 people, you can get a hell of a lot more done without formal specs. and it is usually exactly what the customer wants.
Of course, this never works in real life because managers want documents to mark the progress against their gannt charts so they always interfere with the "make sure you spec it properly", and the manager on the other team will say "dont do a damn thing until you see a signed off spec"
Shit - its no wonder commercial software costs so much.
You can't expect to wield supreme executive power, just because some watery tart threw a sword at you
And of all the praise they lavish on Japan and Indian the conclusion brings it back to reality with:
Politicus
Shouldn't that be colleagues? :)
I prefer a void in conversation to a vacuous one.
Quite frankly this study is worthless. As a business owner, here is what I really want to know. Who is best at producing a product that meets my customer's needs the quickest and cheapest, has great uptime and the fewest bugs. I could give a rats ass how you did it, as long as it meets the above and it is documented.
-Subotai
"The only way to catch tiger cubs is to go into the tiger's den."
Hah! I'm halfway through my current project, as indicated by the constantly shrinking schedule, and I haven't even had to have ANY specifications or requirements to get to this point!
However, they HAVE managed to change the name of the project on me at least three times, and our last two-hour meeting was consumed by a lively debate on what to call a particular form, so it's not like these critical planning issues are being neglected.
When you use specification - you are making the "customer" the deisgn engineer.
I highly doubt the "customer" is going to deisgn great software.
for example.
A thousand customers could say I need a tool that lets me input the various factors of my budget and look at what the sums will be.
But how many could say - I need a program with a grid of flexible cells which can hold a value or a formula?
AIK
well now that I've read the conclusion, the author seems to agree that Bender is from India. :)
page 20 conclusion
"It is important to remember, as well, that no Indian or Japanese company has yet to make any real global mark in widely-recognized software innovation, which has long been the province of U.S. and a few European software firms."
The presence of detailed development specifications is arguably directly related to the size of a design team.
If your development team is two guys sitting next to each other all day long, there isn't much need for very detailed specs or a set structure. You tell then what your project must have, and they deliver (if they're good).
On the other hand, the larger the team, the more structure is required; you don't want one person breaking what another person took four weeks to complete.
I think in the US, the relative lack of specs is probably because most US firms are in one location where the developers are in close proximity, making communcation quite easy (you don't even have to take your eyes off of your screen to yell over a cubicle).
It would have been nice if they had included smaller companies in their sample. Probably just as well that they didn't, though, because I suspect those would make the US numbers look even worse.
Slashdot - News for Herds. Stuff that Splatters.
Having worked in America, UK and India, I can certainly relate to the findings of the report. Here are some of my personal experiences:
:-)
:-)
- In America, they are in a tearing hurry to produce a prototype/model/proof of concept, which of course forms the basis of initial release ("it works, doesn't it? so why re-write it?")
- Gathering requirements is a pain in the wrong end - I've seen all sorts of instances - CFO disagreeing with CTO and re-writing everything, PO Manager introducing something new that CFO promptly over-rules. At the end of the allocated gatherings period you have a hash of what each stakeholder at the company wants. Needless to say, there is a lot of fun in ensuing months - Most American customers do not like to spend time on discussing/analysing progress or answering questions during the development (yes, I agree they shouldn't crop but, but most times they do).
- As a natural progression, the aspirations change over time. By the time it's written and demo-ed they want it to do 10 different things and pipe-up about how they had "already" mentioned they wanted this cool new feature. The signed copy of requirements specifications sure helps
- In Britain though, they pore over every single email the PM/Team Lead sends. Tremendous emphasis at every single aspect ("We are bloody payin you for this, aren't we?")
In India though, life is Sh*t. The documentation team, the process team and the internal audit teams sort of join hands to drown you in sh*tload of paperwork. It's good to have processes, but IMHO, they just get carried away and want copious detailed documents about everything and anything...Unless they develop tools to automate and regulate the processes, it is gruelling and something no-one looks forward to...err, except the audit team
http://efil.blogspot.com/
There's a difference between a spec and a design..
SPec documents are the "what" the customer wants.
Design documents are the "how"...
Slashdot is like Playboy: I read it for the articles
Kinetic stupidity has a new brand leader: Allen Zadr.
Apart from the classic Software Engineering advantages of a proper design document, it can also save you the problems which can arise if the customer and the supplier have different ideas about the eventual product.
I've seen it happen way too often; the expectations of customers can be very unrealistic simply because they have no knowledge about software engineering.
Having a complete design document with two signatures can prevent 'just add this one little feature'-type problems.
When I work by myself or with one or two developers
who I know well and get along with(and talk a lot with) We can get by with practicly no specifications even with a coding which lasts several months.
You can split up the work by writing header files first and that usually does the trick.Obviously this
cuts down on development time.
However I had on several ocasions needed to join in on a project which has been going on for sevral years, and I found it much much easier to start working quickly on the projects with more specs.
I am currently on a project run by a man who is quite anal about specs and standards and documentation, and organized testing. The result is that I spend more time dealing with standards then programing but It greatly increases the quality of the code, it makes the throwing out a week of work for incompatebilty impossible, And perhaps most importantly it makes getting aquainted with a diffrent part of the project very easy.
As for EP, I have seen it work well and I have seen it fail miserably. I have not yet gathered enogh expirience with EP to identify what makes it work.
Me.
P.s I am not a US resident, nor did I study in the US.
While I'm not disagreeing that U.S. Developers may not spend enough time in design before starting to build, I don't think that the blame always lies with the developers. My personal experience has been that, for every missing design doc or spec, there is someone in sales who shaved a week off of the estimate the developers gave them for building the solution. If MIT is studying the Software Development Processes, maybe Harvard should do a study on the Software Development Sales process, and on why US companies have developed the mentality that it is ok to give developers less time than they need to complete development, and even better to base the timeline of a project on developers working 65 hour weeks when they know damn well that they are salaried employees and thus get no overtime.
What about the twinkie? - Dr. Peter Venkman, PHD
But if I look at those that aren't suffering much right now, those doing well, I see that most of the successful U.S. deveopers and shops also specify things out. But, because we don't live and die by the regulation (those being cradle-to-grave parts of much of the world), we can also be more flexible more quickly. And we can prototype from the seat of our pants quickly.
A down side of "flexibility" though is that we often get called "not team players" if we don't instantly cow to the calls from sales and marketing every time a new feature idea pops up. One of my co-workers calls this "chasing butterflies", the lack of focus that results in never finishing anything. That's the downside that leads to bugs and slow development... and failed companies. But many more successful companies, including most of the ones I've been at, have actual product life cycles. There really are two tiers or classes of development companies that way.
The question is, why are there so many undisciplined shops? I think the answer is the easy money of the past. Suddenly we (developers) weren't being managed by engineers and developers, but rather by CFOs, shareholders and sales people. As the crunch continues, I suspect corporate Darwinism will continue and we'll scale back to more methodical practices again.
So, crack open a cold one and check out my double nested for-loop ollie nose-grind, dude!!
And who says we're zealots?! ;-)
RRR
Stuff that matters: circuitbreakers, vacuum-cleaners coffee makers, calculators generators, matching salt+pepper shakers
With Aegis the baseline code "always" works; it has to pass all the build tests to become baseline.
You can't add a new feature without first defining a test for it to pass, you can't fix a bug without defining a test that the old baseline failed and the new baseline passes.
So marketing can walk up and say "release now" or "add these features" and you can do either. But you can't "release these features now" because the system won't let you.
When marketing say "release now" they can only have the bits that work. And when they say "add these features" they can only get those features when they work.
Sam
blog.sam.liddicott.com
Maybe that's the problem. Too many times we worry about creating the "grid of flexible cells" and forget that the real user just wants to "input the various factors". There's a good usability lesson to be learned here...
Damn, where have they been hiding after school then?
Donte Alistair Anderson Roberts - hi son!
Karma: Chameleon
Lack of specifications is the only thing U.S. programmers have going for them. When there's no specs, the developers have to think and must understand the business requirements, which is difficult to outsource. If someone ever figures out how to automagically create detailed specs, then all our jobs are going overseas. No specs is fine with me.
Still - it is only what the customer THINKS he wants on day one with far less experience in computer techniques.
Customer says he want a report - but in reality he want to understand his business.
An OLAP server can help him understand his business, a crystal report can start him down a long road of report modifications which in the end will lead him to 1. an unmanagable pile of confusing reports or, b. an OLAP cube.
The design process is to understand the need of the customer - not have the customer specify how or what the solution will be.
If you provide the solution the customer specifies - you will be run out of business when he reads the next business journal anout how to do omething better.
You had beter know what he is going to read next if you intend on being a solution provider for very long.
AIK
1. Get an idea and start working on it immediately.
2. Stop work on it to fix a bug, release the bug fix immediately.
3. Work on the idea again, but slightly changed.
4. When about 80% done, switch to new idea or "gotta have" feature.
5. Code like hell on original idea because it was released in its incomplete form and has now killed puppies.
6. Rinse, repeat.
All the while there's very little documentation, most of it being whatever I do.
Great, isn't it? That's how it was. I've taken control and actually have implemented a release schedule and proper bugfix releasing. I've also just gotten a QA guy, things are looking up. The processes before I got here made me wake up at night. Lo and behold, with the new processes, the phones don't ring as much.
Ahh, the joys of a very small company.
Anyone else find it odd that the first page of this supposed 'new' study was marked June 2003? And the second page said it's version 3.1.
Which goes to show -- even if we had specifications for these things, we're just going to gloss over the details, and do whatever the hell we want, anyway. People don't read what's sitting in front of them, unless it's on some blog, it seems. If it were really important, they'd have made a TV show out of it.
[and those of you with moderator points get to vote if you think sarcasm is funny -- you can select 'troll' to vote no, 'funny' to vote yes. 'overrated' if you'd like to abstain, and 'insightful' if you read the first line, and are just trying to burn your moderator points]
Build it, and they will come^Hplain.
So many articles on software development concentrate and talking about what procedures to use (and how to get people to buy into using them, etc. etc.) Very few of them give any good data on the outcomes of using those procedures. Typically we get b-school type "case studies," anecdotes, and proof by repeated assertion.
Typical logic: PREMISE: It is good for software to be of high quality, in time, and under budget. ERGO, do thing my way. COROLLARY: It is important to use an indentation setting of three spaces on everyone's editor, and draw diagrams using the symbology provided in this plastic template.
Here's my first-order analysis.
All over the world, software development uses significantly different formal processes and practices. Yet, to a first approximation, there are no obvious, huge, systematic differences in the quality, cost, or timeliness of software produced in any country versus any other country. Therefore, to a first approximation, the formal process and practices don't matter.
Good people, backed by good management, with a good understanding of requirements and sufficient time and resources, can develop software successfully. Using the waterfall method, Extreme Programming, or no methodology. Mediocre people, given bad management, and inadequate time and resources will fail. The brand name of the methodology that is asserted to be being used has an effect so small it is lost in the noise.
In my software engineering class, my teacher vehemently states that Requirements are the Enemy of Design.
Unfortunately, a lot of college professors are out of touch with reality. Software development is such a diverse area that you really can't generalize.
For example, I worked for a long time in embedded firmware and digital signal processing, both on mega- and micro-projects. You need to design to requirements for these. There can be creativity in how you impelment a design, but the bottom line is the spec. If you don't design to the spec, the satellite falls out of the sky.
Currently, I am working in multimedia, and we don't really use specs. We have high-level goals, but even these are fuzzy. Here, requirements are more of a hindrance, but we still have to draw the line in the sand for some things.
(S(SKK)(SKK))(S(SKK)(SKK))
I think its more like:
Requirements are what the customer wants.
Specifications are what he's actually going to get
Design is how the analyst thinks it should be done.
Bullet proof specs. are very hard to write.
:-)
I spent many years as a low level civil servant; so none of what I am telling about is my fault
To make a pile of money on a government contract, do the following:
1 - Bid on the contract EXACTLY as the tender is written (no matter how stupid the tender is). This is what the government will stick you with. If somebody else bids on the tender the way it should be and comes in cheaper that's ok. The government will stick them with the stupid specs in the tender and they will lose money! They go out of business and you now have one less competitor. **Include a generous hourly rate for 'extras'. Make sure that 'extras' are cost plus.
2 - Build the project EXACTLY as tendered and bid.
3 - (This is the important step) Discover that the project will not work as designed and tendered.
4 - Fix the project at public expense. This is the part where you apply the 'extras'.
5 - Profit! Note the lack of question marks at any stage of this process.
Specs are fine but they are no replacement for wisdom. If you need to cya then use specs. Otherwise, take them for what they're worth.
Maybe I'm misunderstanding your intent here but I'd say the release-early-release-often is a huge strength of open source development.
By showing all the warts, all the problems and solutions as they are worked on, those of us who are chasing the latest nightly CVS release get to help out by bug triaging, raising problem reports and commenting on the very latest change pretty much as it happens. This stops stupid decisions, breakages and irritations early before they are ingrained.
However, for the average user, that is too much. But just because development is pushing on at a rapid rate doesn't mean that the user has to play catch up. This is where the packagers of applications play their part - Mandrake/Debian/RedHat/etc - by providing a tested stable base. If you want the ultimate in stability, you would probably look at Debian Sarge or RHEL. You can pick a more intermediate solution which is closer to the latest releases with Mandrake numbered releases, closer still to the edge with Mandrake community /Fedore core 1 or almost at the bleeding edge with Cooker or Redhat Fedora 2 test 3.
So really, stability of releases is available now. The only scary thing here is that you have a Choice.
Cheers,
Toby Haynes
Anything I post is strictly my own thoughts and doesn't necessarily have anything to do with the opinions of IBM.
What the customer wants and what he needs are different things too.. as illustrated here (no idea where it came originally from, if you have the proper credits, please post them)
There was a little back and forth discussion, but basically the requirements were hammered out in a few days. The development team of about 5 coders took these requirements and over the next 6-8 months we used a "release early, release often" strategy. We called it "Build a little - Test a little", basically we had a GUI designer cranking out do-nothing screens, while the rest filled in the guts one function at a time.
We ended up essentially almost on time and almost on budget. The project was WILDLY successful and still in use 7 years later.
We have been funded now to produce Version 3 of this product. Most of the team has moved on, including the chief architect who drove the implementation philosophy and he's been replaced by someone who has embraced specifications like a baby marmoset clings to it's mother.
Since December, we have done NOTHING but attend meetings and write excruciatingly detailed requirements documents, and crank out UML diagrams. Not a line of code has been written (besides a bit of prototyping).
I'm doing it because I have to. I see a lot of extra labor cost (I estimate at least $250k so far) and I don't think this process has helped us. I think we could have created an entire prototype since December and thrown it all away and been ahead of the game (assuming we'd learned valuable info from our mistakes)
Maybe someone is actually using Evolutionary Programming (not)?
-- Nothing unusual happened today
7. Add 20% (I'm almost there...)
8. Add 20% (Just another two weeks...)
9. Add 20% (Darn these last minute bugs...)
10. Add 20% (Testing takes time, you know...)
11. Add 20% (They want "web based" now...)
12....
Vote in November. You won't regret it.
Made in USA much earlier :-)
My customers don't know what they want. They don't really understand how their own business works and they feel lost because as their business is growing they lose track of the knowledge that was very easy to keep up with when their company only had 20 people.
It depends on your software and your client base, I guess. When I program for hire it tends to be CRM and HR packages for small businesses that have outgrown Quicken, so I guess it's understandable that I see a lot of confused people who honestly just don't understand their own company anymore.
But at least in my case, my customers really don't know what they want because they haven't yet figured out what they know and what they need to know about their own business; to some extent I can help them with that but generally I just point them to some management consulting firms.
All's true that is mistrusted
It is not the easy money. As a programmer who was around before the "bubble", there were many many undisciplined shops before, during and after, at least from where I sit.
The problem is that the business and sales types that run these companies, by and large, dont understand software development, and dont want to understand it. Further, they do not listen to the development managers that they hire, and often overrule them, using their own misunderstanding to guide them.
emt 377 emt 4
As a System Test Engineer, I especially liked this part of the summary
One thing most people don't realize is that 99% of the time, only a small percentage of a program actually gets executed. Much of a program is error checking or handling of unusual events and doesn't get executed if the customer uses the program the way the programmer intended. The only reason that most commercial software is usable at all is because of this. So of course, testing and integration has a negative impact on development time.
Most software companies know this and develop software to be "good enough" since being anywhere near bug-free before shipping the first version requires too much time and effort. Software companies are "gambling" that the bugs that are left aren't bad enough to sour their customers on the product. IMHO, American companies are generally much bigger risk takers than foreign companies. This leads to either a spectacular success or a catastrophic failure
I would like to see the U.S. government start to punish software companies that take large risks with investors capital. I believe a lot of companies die because of poor implementation and not necessarily because of a poor idea. I can't think of anything easier to screw up then software development and it has been considered just an ordinary risk of running a business. I worked for a software company that was run by a man that, from what I saw, didn't give a hoot whether the software worked or not. As long as the IPO produced a lot of money he was happy.
There are many ways to reduce the risk of producing a worthless software program, including certifying programmers, code reuse, and more testing. I know most people don't want to invite the government to enact more regulations but the software industry doesn't seem to be regulating itself. There is a huge crisis brewing on the horizon and nothing seems to be getting better. If the millions lost on government computer systems, like the IRS modernization, isn't enough, we have the potential for a Microsoft virus to wipeout millions of users data.
"Meaningless!, Meaningless!" says the Teacher. "Utterly meaningless!"
You completely missed the point. No one is forcing anything on to the customer. But quite simply, if you have EVER worked in the software industry designing custom systems for customers, you will know that the customer generally doesn't know what they want exactly. They have a vague idea and assume that you have the same idea they do in your head.
The requirements process is where you get your specifications from the customer about what he or she wants. Design is a completely seperate stage. The requirements are something you both agree on, but its not just something the customer sends to you. You give them feedback about the requirements. Perhaps they are contradictory, perhaps there are better ways to do things. This part is crucial in that the more time you put into this, the more information you will fish out of the customer, and you'll be more confident that you and they will know what the software is supposed to exactly do.
The golden rule in software engineering is the requirements are going to change. You just have to accept it. Why do requirements change? Well, usually its because the customer finally realizes that what they got and what they actually wanted were two different things. And thus you make the necessary changes until the next time they come back looking for you...
How many times have you downloaded a program think it has everything you want, until you use it and then realize theres something more it needs?
Think about something like Mozilla. It's be a sufficient browser for a while now. But once people got it, started using it, they thought to themselves "Now I need tabs!". And thus the evolution of software...
I looked through the report somewhat quickly, but I did not see any mention of the peculiar practice of sticking developers into small, noisy cubicles, with cheap, eye-straining fluorescent lighting, and then expecting them to foucs and produce top-quailty software.
I was walking around where I'm currently consulting, and I noticed that everybody had a set of headphones, and it just struck me as odd that software development should require specialized head gear.
Java is the blue pill
Choose the red pill
Not only that.
Usually you try to understand what the client wants, and design the system to provide it. Then you write the detailed specs of *how* you do it, what kind of data you use, etc. Then you present all that to the customer, and have him sign it. It makes things a lot easier as clients sometimes tend to ask for changes in the middle of the development process, when all design has already been done, and half the code has been writtem.
"Luck is my middle name," said Rincewind, indistinctly. "Mind you, my first name is Bad." -- Terry Pratchett
When specfications are turned into an overly formal process, the spec process becomes a time-sucking industry in itself. I have seen cases where some "Spec Police" count how many slots are filled in on the forms, while most of the time it is stuff that is obvious and redundant for the module. Plus, a one-size-fits-all spec form does not work well. Different kinds of projects need different approaches. For example, a report-centric project does not need a "database change results" section because reports usually don't change the database contents. Ideally it is a give-and-take process, not Slot Policing that works best.
Table-ized A.I.
In my experience, one of the most significant and under-recognized challenges in development is the selection of the right tools, operating environment and languages in which to develop/deploy an application.
.ASP, and Perl that really should have been using something different based on the demands of the process.
These days, your typical American developer has a narrow stable of technology that he uses. He often doesn't stop to examine whether or not the application being designed is best suited for the environment in which he plans to build it.
I'm probably going to get flamed for this, but I believe we now have "vanity languages" and platforms that are driven more by marketing than fitness for a particular purpose. In the last several years I've seen lots of programs written in stuff like Java, Cold Fusion,
I would suspect a significant share of development disasters are due to the people involved choosing the wrong tools and then making things ten times harder for themselves later on.
I think it's true, spec writing does often fall by the wayside. However, I think it's far more prevalent at small companies, and non-tech-oriented companies that nevertheless have engineers. It's often a money or time thing, because it takes a lot of time and effort to pre-document things before developing them, and small companies don't often have such luxury. But there is also a lot of laziness involved. I don't know many engineers who like documenting things at all, much less weeks or months before they get to knock out even a line of code. Given the chance, they will generally blow off docs and start hacking.
I think this has very little to do with extreme programming, and everything to do with motivation (or lack thereof). Though I think this phenomenon has one thing in common with EP, in absolute seriousness - laziness with regard to writing specifications.
If you are wasting months on an OLAP cube that is your problem.
I implemented OLAP cube with an excell frontend embbedded in a webpage such that all users in a multistate company could view the OLAP live from the website - and it took LESS time than any crystal report - and it was precisly what the company had been hinting at for years - only they didn't understand what it was they wanted.
Yes - I'm arrogant if it means anything to you.
AIK
This series of comments by AIK is profound. Please get off your apple-pie-and-motherhood methodology soapboxes and pay attention.
Customers do NOT know what they want. Anyone who thinks that they do, hasn't spent enough time architecting software.
What AIK is saying is that you have to dig deeper than the customer requirements. You have to understand the space. You have to look at competitive products. You have to anticipate unstated needs. You should ask, "When I'm all done, and everything is working perfectly, what changes will the customer want IMMEDIATELY?"
I can't stand people who listen for five minutes and start to write "use cases" right away. That works for some dumbass web site, maybe, but certainly not for any involved product design.
Architects need to plot an intercept strategy several YEARS in the future. That's how you build successful software. When the customer puts his Phase N requirements out for bid, and all your competitors run for the hills, your design has anticipated his needs. You've built metatables instead of tables. You've used OLAP where you could have sleazed by with an RDBMS. And so on.
Great thread.
I think Christopher Alexander sums it up beautiful in his Patterns book, in the 'Gradual stiffening of design' pattern.
He observes that only in-experienced craftpeople plan out things down to the minute detail to begin with. This causes them to get lost in the details and not able to recover from an problems during the building stage.
Experienced people all employ processes that may start out with a rough high level design, but the detailed design only gets determined as the construction process matures.
The idea is that a more fluid design allows you to both absorb errors or any problems or new insights that may occur during the actual build process.
The same thing applies for software engineering.
There is a nice balance to be found somewhere between the bondage-and-discipline approach and the XP style design-as-you-go. The type and size of the project also have to be taken into account.
Where I now work I typically build everything in one of the following tools, ASP.NET, VB.NET and MS SQL Server. At the company before that it was all in Lotus Notes/Domino slowly migrating to ASP. And before that it was all Java.
Why did I choose those tools? The answer is, I didn't. I never got a choice of what to use. I had to use what the company already had in place.
I know very well that ASP.NET is not the best place to build a workflow application. But managers have already picked their favorite technology, even if they don't understand them.
In smaller companies the IT manager used to be the network admin. Guess how much network admins know about software design and languages. Try nothing!
So until they start promoting trained software developers to management this problem with persist.
Whoa, whoa, whoa, whoa, whoa, whoa, whoa, whoa, whoa, whoa, whoa, whoa. Lois, this isn't my Batman glass. - Peter
OK, I don't really know how much of Indian work is for US companies ;-)
Honestly, most Dev shops don't have "code librarians" anymore. When they did - this was the person who would make sure that code didn't get duplicated. If a current function could do what needed to be done with very few changes, so be it.
What happened where I was - some people who were not well trained in the code base started re-creating a large number of functions that we already had, and did so in an otherwise incompatible way. Yes, this should have been caught by the team lead... Yes, this should have been caught by their boss -- However, it could have been avoided had they been given a very specific specification of what these folks were expected to do.
I don't care how new the programmer, few of us like having someone over our shoulders - so, most of us assumed that they were each learning about the project as a whole, as opposed to digging in and seeing how to impress everyone with quick output. Human nature prevails again.
Kinetic stupidity has a new brand leader: Allen Zadr.
Based on my experiences of the past 20 years working in Unix OS, Networking and Graphics development organizations. Good practices and team culture can overcome weaknesses in any given development process model.
IMO, no single model works well for all projects. The development model that best fits the project requirements and the team culture will usually produce the best results.
What I've found is that when Key Practices are performed on a daily basis, whatever model is in place can be sucessfully managed.
Key Practices:
Change Control: this includes; source, binary, requirements, decisions, action items, specifications, hardware, firmware, processes, practices, etc...
Communication: the ability to notify and acknowledge change
Assessment: the ability to review change and make corrective adjustments in the schedule and requirements
Planning: the best product bosses work their plans every nite, and publish every morning
On Demand Crank Turn: the ability to turn the crank at least once a day. This includes: build, package, assemble, test, publish, assess, plan, prioritize
GigantanKramePithicus
If the customer has a million expections, but your Lead or Project Manager has not told the customer what to expect from their current "official" requirements, then you will have a disappointed customer. However, if you let the customer know, right from the start that the project will do exactly what they ask, and no more - then "extras" that were designed in through savvy programming, code re-use - or interface standarization, etc. will now be beyond the customer's expectations.
On the other hand, when a customer says, I want it to be like Excel, the Project Manager's first words should be, "why can't you use Excel?" - as opposed to "Sure, what about Excel do you want us to emulate?".
Kinetic stupidity has a new brand leader: Allen Zadr.
In my experience, RUP seems far better. The customer rarely knows exactly what they want. At best they'll have an idea of what they want it to do and can visualize how they think it might work. So you start with a loose set of requirements, do some quick high-level design, and code it. Then show the customer what you've got. They can identify what is different from what they want and see some of the issues they may not have thought about. This defines some lower level requirements and designs. Repeat this process on a regular basis and you will get good, well-written, and efficiently created software.
I've had nightmares with the Waterfall process. I've had to spend 6 months writing requirements and design docs, formally releasing them, then have 2 weeks left to write the code so the software wasn't nearly as good as it should have (or could have) been. In the end, the requirements and design had some inconsistencies that could not be foreseen until you try to code them.
On another project (that I was not involved in) all of the money went into the requirements and design process and the end software was crap, wasn't user friendly, and didn't get used. But it did meet the requirements.
The problem with writing requirements completely before coding is that you have to basically do the design and coding in your head as you write the requirements, otherwise you can end up with things that are inconsistent. Written language doesn't have to be self-consistent, software does.
On the otherhand, writing software without following any guidance from the customer is definitely worse.
As an employee of a US division in a German corporation, I think I can shed some light on why US developers skimp on specifications:
We're too busy coding.
The truth is that commercial development schedules are unrealistic. If it would take two years to do a project correctly, only eighteen months will be allocated. This leaves the developer in a quandary over which part of the process to skimp on. US developers choose to skimp on specifications, while German developers choose to skimp on implementation.
That's my experience anyway. I've seen specs from Germany that are so padded I think the author must have stock in a paper mill. And I've seen the incomplete software that arose from it. In one instance a product was shipped that was completely unusable, whose only sales the first year were to the sales department as demos, but which won a corporate award for adherence to the process.
It's simply a different way of working. To the US developer, if you can't do it right, at least make it work. To the German developer, if you can't do it right, at least go through each step of the waterfall model thoroughly and methodically until your time is up.
Just about every corporation views the process as more important than the product. But their individualistic and rebellious nature means that US developers will work on the product anyway. But German developers will just do what they're paid to do.
Don't blame me, I didn't vote for either of them!
Well put. I noticed in practice both you and the client experience a development process. You develop software, while the customer simultaneously develops an understanding of their own business or problem. Looking at it this way, it is clear immediately why a waterfall development approach won't work.
But still at the development shop where I work, the gospel is to create a detailed requirements document (including quote) beforehand. The customer must sign off, and development starts. Every project I have seen performed so far has ended in sour feelings on both sides, due to the customer wanting to change requirements midstream. And everytime I'll hear other developers complain about customers who do not know what they want. It baffles me that no one (especially management) tries to find a development form that can handle changing requirements in a smooth way.
Extreme Programming promises to take care of this problem, but I'm not quite sure how you'd make an initial quote. The feeling at work is that customers will only accept a quote which promises to deliver a certain set of specifications for a certain price. Apparently, customers themselves do not realise beforehand that their requirements will change over time. Therefor, they will prefer a quote which promises them to deliver what they think they want, rather than a quote which promises them a best effort in a rather open-ended time frame.
Anyone have any experience with quoting XP projects?
Offshore developers are less invested in their users. They rely formal spec's and adherance to "Best Practices" to defend themselves, very politely, of course, against user dissatisfaction with the app as originally (mis) spec'd. No more dollars, sorry... but no recoding...
Name an elegant app that was spec'd right from the start... No? Name any number of awkward apps that meet their spec's but are unusable... No problem!
US users are independent, demanding and coddled. They know what they said they wanted, but now that they see it on the screen what they really want is...RECODING!
These are just the expeiences of a pragmatic old US developer...
"Knowing everything doesn't help..."