Slashdot Mirror
Can Software Schedules Be Estimated?
J.P.Lewis writes "
Is programming like manufacturing, or like physics?
We sometimes hear of enormous software projects that are
canceled after running years behind schedule.
On the other hand, there are software engineering methodologies
(inspired by similar methodologies in manufacturing)
that claim (or hint at) objective estimation
of project complexity and development schedules.
With objective schedule estimates, projects should never run late.
Are these failed software projects not using proper
software engineering, or is there a deeper problem?" Read on for one man's well-argued answer, which casts doubt on most software-delivery predictions, and hits on a few of the famous latecomers.
"A recent academic paper Large Limits to Software Estimation (ACM Software Engineering Notes, 26, no.4 2001) shows how software estimation can be interpreted in algorithmic (Kolmogorov) complexity terms. An algorithmic complexity variant of mathematical (Godel) incompleteness can then easily be interpreted as showing that all claims of purely objective estimation of project complexity, development time, and programmer productivity are incorrect. Software development is like physics: there is no objective way to know how long a program will take to develop."
Lewis also provides a link to this "introduction to incompleteness (a fun subject in itself) and other background material for the paper."
Software development is not a science in the normal sense. Designing large software systems is an art. It cannot be pigeonholed. Stroustrup has a lot to say about this when he describes the 'interchangable morons' concept in the 2nd edition C++ book.
Anyway, read Death march by Ed Yourdon, and the mythical man month by fred brooks, and antipatterns, any time someone asks you for an estimate say 'two weeks' and then bullshit from there on.
That is how it works in the real world. The numbers are essentially meaningless, but the bean counters and suits have to justify their existance somehow :-)
Can you imagine asking Linus when 2.5 will be ready ?
Very large and complex projects do get completed, sometimes even on-time/on-budget. Examples include skyscrapers, nuclear submarines, aircraft carriers, power plants (whether conventional or nuclear), oil refineries, B-747/A-320, etc. And all of these systems nowadays have a software component as well.
So the easy response is that bad management in general, and bad project management in particular, is responsible for software project failures. While this is no doubt true, the next question has to be, why do software projects have such bad project management?
I don't have a good answer, but one thing that occurs to me is the lack of a fixed endpoint. When an oil refinery ships its first load of POL, it is complete. When an aircraft carrier launches its first plane, it is complete. But the amorphous and mallable nature of software means that it is hard to define an exact endpoint, and very hard to avoid changing the definition of the endpoint as the project proceeds. So things keep "creeping" along until disaster occurs.
sPh
My company develops turn-key systems. Sometimes we also develop custom solutions for our customers. Our customer base has increased steadily after the dotcom crash, when we switched from products to services. One of the reasons our customers like us is that we don't bill projects by the hour. We will the project on a fixed price, not to exceed, basis.
The programmers who work with us on a contract basis don't bill us by the hour either. After we have the design and we distribute tasks and prior to submitting the final estimate, we ask contractors to place a fixed bid.
We've done six major projects like this since March, and in all cases we finished within budget and on-schedule, and the systems are currently in production. They are all mission-critical systems running in either robotics environments or high-availability networks.
Our economic motivation is then to do things well and quickly in order to increase our profits. That also enables us to move on to the next project faster than slaving over some customer in order to bill the maximum hours.
As far as development techniques go, we adopted XP earlier on and it's working for us.
Cheers!
Ehttp://eugeneciurana.com | http://ciurana.eu
There are four parameters to a software project:
- Quality
- Quantity
- Deadline
- Costs
In a competitive environment with humans involved, up to three can be specified. Not four. Good examples are:
- Many guidelines for managing software projects tell you to reduce quantity when you get near deadline.
- Some customers have a specified budget but really don't know how much software they can get for that money. They prefer to have costs fixed than to have quantity or deadline fixed.
- Sometimes deadline is so important, that costs may 10-double in order to reach that deadline, and quality and quantity may get reduced a lot in order to finish the project.
It is extremely important to realize the meaning of all four parameters before you can talk about estimating project schedules.
Lars.
The biggest problem I've seen is requirements creep. Most often, you don't have a firm set of requirements to start with. Management and programmers both have a tendancy to view requirements documents and other formal software engineering practices as superflourous. The problem is that without a firm set of fixed requirements, you are always trying to hit a moving target.
Another problem is attitude, mostly on the part of management, but programmers are guilty too. One faulty attitude is that we are conditioned to expect immediate results. There's also a prevaling attitude that there is never enough time to do it right, but there's always enough time to do it over. This leads to undocumented, unmaintainable masses of code that either gets thrown away after a while.
Even worse, you wind up with garbage code that SHOULD be thrown away and re-written from scratch, but winds up getting patched and modified for years. I can't tell you how many times I've had a manager say "there isn't time to rewrite it, just patch it". That would be OK if you are only going to patch it once -- but you wind up patching the same program a half dozen times, and it winds up taking twice as long to do all the as it would have if you had just rewritten it from scratch.
Why is it that the proponents of "one nation under God" are so eager to get rid of "liberty and justice for all"?
The problem is, you don't get paid for coding up a small working prototype in order to do an estimate. So my estimating technique is:
Figure the time to do the parts I understand.
Count the parts I don't understand. Allow a very long time for each of them.
Add it all up, then multiply by 3
Ask a sharp programmer to estimate the time to develop a software solution and he might shrug and look irritated. Ask him if 2 weeks will be enough time, and there is an 80% chance he will say "of course" no matter what the task!
Gung-ho programmers are optimists. Couple optimism with the ennumerable factors involved in programming a non trivial application and you will get what we have today.
By the way. I am a programmer and I have little to no confidence in my time-estimation abilities, or anyone elses. It has taken me 14 years to come to grips with that.
--- -- - -
Give me LIBERTY, or give me a check.
The issue is not physics versus manufacturing, it is scope and cost containment like is done in manufacturing. As a person who has lead multi-million dollar projects, I have grown used to the cliché that goes something like this:
If we built homes like software we would all be living in the street, penniless...
The major issues I have seen revolve around a lack of scope and cost control. In many cases it is because there is little penalty for being late or over budget. In cases where penalties exist it is often beneficial to then over estimate the effort or cost required. Then once the money is approved, using it is becomes easy.
Going back to the analogy consider the following:
Scope
If you were building a house, each piece has a specified cost, known in advance to a very large degree. In addition, altering the scope itself often incurs a penalty, because the work is not done by the owner. You plan a three bedroom, 1.5 bath home. Midway through planning you decide to make it a two bath home instead. The architect will charge the "re-scoping" fee and the builder will add the material fee. Now do the same after construction has begun. The architect gets their fee, the builder adds the material and resource costs, plus a "revision" fee for changing your mind after construction begins.
During a software project, it is common for individuals to approach the developers and ask to expand the scope. This would be analogous to approaching one of the work crew and asking them to just add the extra half a bath. The difference is the work crew would get fired, and the developer gets bonus points for adding the feature, either directly or indirectly.
If the developer chooses not to do it, or pushes them to the project manager, the client may label them uncooperative or difficult to work with. The project manager not wanting to be labeled either may coerce, cajole, or beg the developer to accomplish it, without a scope revision. Failure to do so by the developer results in real financial impact at some point, and offers little incentive to hold the line.
Cost
I call this the "Porsche syndrome".
I go into the Porsche dealership and see a new 911 Carrera Coupe. Smiling the dealer offers to sell it at a deep discount, with options and accessories $84,000 (U.S.). Whewwww baby!!! I cannot afford that. "Look," I tell him, "my wife will never approve that, you need to get it down to $28,500 tops." Would any of us expect to have the price cut down? By half or more?
Okay, how about "Look, what will it take to get it under $30,000? Seriously now, what do I have to give up" As the dealer is escorting me to the door he explains the only way I will get this car under $30k is with a mask and a gun or from a scrap metal dealer.
Yet, daily we go to developers and tell them to do the same. We ask for an estimate and then go back with "This is too much, it needs to be smaller or it won't get approved!" --Insert blank stare here--- The idea that if something cannot be cost justified it should not be done, is often lost in the "request" itself.
To nearly guarantee a project is on budget and time requires things many companies are unwilling to provide. Strict scope control procedures, with oversight by the person responsible for the money. That means each change, regardless of how trivial must be approved by someone above the project management team with business justification. It also means that requests for scope change cannot be made to developers directly, by anyone.
I was very happy with the people who built my home. When speaking to many of my friends and coworkers who built their homes, they describe it as a process akin to having their flesh removed. Everything required such effort and detail that many would not do it again.
Most of them were looking for the relationship to be like one at the office. We all want to get along and help each other out. This is not a commercial arrangement, and when we put the commercial context around it, we see it many offices lack structure.
Internal organizations can be setup like commercial ones, but it is usually unwelcome as the perception is everyone should be working for the greater good of the company and this has the appearance of bureaucracy. Even if inaccurate, everyone "wanting to get along" prevents it from being implemented.
Likewise, I've been developing (C++) for a living for about 12 years now and I've come to some conclusions:
There are estimating techniques/metrics which will work. They depend upon going round a few times to "calibrate" and consistent application. "Task Points" was a good one - basically break your use cases down and down until you have a series of one-line statements about the system. Multiply these by your magic number and that's the estimate. This, like all estimating techniques, is built on sand because:
It depends upon a development team sticking around long enough to do a few projects to calibrate you method.
It depends upon the exact functions of the system being known at the time you do the estimate. This is the killer.
I have never worked on a project where the exact functioning is known at the time coding starts. I have, however, observed that the more analysis/design you do before estimating, the more accurate the estimate is. The problem is, that people always want the answer (estimate) before they've given you the problem (spec).
FWIW On small projects (which are generally better defined), I run through the spec, do a rough n' ready count up of the number of classes, multiply by a factor (decided by the complexity of each class and who I think is going to code it) add a QA+debugging allowance and come up with figures which aren't too wide of the mark.
Oh yeah, and the "who's coding it" is important. Lots of studies show that the difference between "good" and "bad" coders can be a factor of ten. I've been slammed by PMs after estimating how long something would take me, then the PM puts some "cross trained" ex VB dork on it.
To summarise: it is possible if you know who is coding what. Recommendations: 1) read Brooks, 2) keep it small 3) ignore any of the "latest methodologies" that Project Managers try and sell you.
This sig made only from recycled ASCII
The problem with estimating development time lies mostly in the management's concept of software development. I was hired to work on a project that was estimated by management to last two months. My estimate was four months and the actual time it took to complete was over a year. Why could I not meet the project deadline?
The customer claimed it was because I could not seem to fully complete a component of the project. What they really meant was I could not fully complete a component of the project before they would request a change to that component that in some cases required a complete rewrite of the component. They didn't think it was a big deal to add a button here or there in the application after all it was only a button. Never mind the fact that each of those buttons required stored procedures to be written and existing stored procedures to be altered. They would get upset that I could not make their requested changes in a day when they wanted to completely alter the way the interface to the application worked.
The bottom line is most people who don't know anything about software development don't think it is a big deal to add a feature here and there at the end of the development cycle. I try to equate software development to carpentry. Sure I can add another door in the center of those cabinets, but don't expect it not to affect the other doors and their space within.
10: PRINT "Everything old is new again."
20: GOTO 10
One of the greatest criteria for a good programmer, whether it is the quality of the code, or the ability to estimate a schedule, stems from humility. Part of the problem with people when estimating a schedule is that they thing they are Superman. They think that they are so good that the complex task that is in front of them is trivial. These people tend to have very buggy code as well (normally from insuffient testing). All programmers suffer from this to some extent. I've also noticed that these people tend to never use libraries, since they can write one better, but then use up all their scheduled time rewriting libraries and never actually working on the project.
Personally for me, I tend to do the best hourly breakdown I can and then double it before submission. This is normally not too far wrong (say one week on a 3 month project). The double factor allows for inaccuracies, meetings (which really do take time !), and spec changes. I may add more "fudge factor" depending on my feelings for how well the spec is sorted out and the quality of management (i.e. weak management will allow spec changes every week, good management will filter well).
ANdy
PM: How long to do this work ? :) I only want a rough guess.
ME: How about a spec ?
PM: You're kidding
ME: Roughly 6 weeks.
PM: Nah, too long we'll never get that past the customers, lets call it 4 weeks.
ME: Not again remember what happened last time, you chopped my estimate ?
PM: Don't worry I won't hold you too it, this time!
PM: That work finnished ?
ME: NO, two more weeks.
PM: You said 4 weeks, look here it is in the plan.
ME: I said 6, You said 4 weeks, and that you wouldn't hold me to it.
PM: The only thing I can fault you on is your estimates, they aren't very good.
ME: You £$%&* git !!!
And practically every project manager does the same thing.
Why engineer failure into the plan ?
You are forgetting politics: I have been explicitly told Your estimates are unacceptable - they will have to be halved!
Others have mentioned "creeping featureism".
There is also the "event Horizon" - When faced with a project of infinite size, people will tend towards an estimate that is based on their idea of how long it takes to solve an infinite problem. For a salesman, this is a couple of days. For a typical manager, a couple of weeks. For an engineer, a couple of months.
For estimates to be meaningful, the work has to be divided into units which you can guarantee will never exceed your event horizon.
I have managed many successful estimates on large (over one year, more than 5 people) projects, based on the method that it needs an average of two weeks to implement, document and test, any feature of the project you can identify before the project starts.
By "feature" I mean explicit bit of behaviour by the code eg "ack an inbound packet", "echo the character on the serial line". I know any amount of people who can code this in 3 minutes in perl or whatever. That is not the same as developing supportable code. All loops have to be unwound, all nesting flattened. Every level of the heirarchy has to be accounted for serially.
Let me introduce Dr Spin's 2:1 Law: Supportable code needs 2kg of paperwork per byte of executable code. Includes minutes of meetings, sketches on envelopes. (Most of it is binned, but it still has to be created).
Underestimation as a Marketing Tactic
AKA "Vaporware". Even if marketing knew when a product would be shippable, a particularly cinical marketing department may claim it to be earlier, thus freezing competitor's development.
Lack of Feedback (Moving Targets)
Software engineers are particularly bad at estimating because they have never done what they estimated. They are given a large project, give a large estimate, start working on it, and the project changes in the middle in a major way. This is a moving target; the estimate no longer applies. Major law of software development: You cannot change the spec or the development team on the project without impacting the real ship date. If you don't re-assess the estimated ship date, you are simply fooling yourself. Thus, they don't have any clue whether they hit the estimate or not. One way to defend against this is to break the project down into bite-sized pieces and estimate them; a small piece gives you a chance to do precisely what you estimated. Once you have that, you can have somebody track your estimates, and come back saying something like "On average, you go one third over your estimates. Add a third to your estimates from now on, and we'll be accurate".
Management Estimates
Often, engineers don't do the estimate. The management or marketing people tell you what must be done, and how long you have. Sometimes this is done explicitly; other times, management may have a number in mind and shame a software team into agreeing with it by laughing off any number that doesn't match theirs. Business people often negotiate the ship date with the geeks, like any negotiate with any other vendor. To a suit, vendor negotiations are how you determine the "margin", or how much the vendor is making (like when you buy a car, you and the dealer come to a number that determines the dealer's margin). This doesn't work in in-house software develoment because geeks hold back precious little "slack" or "margin" (they don't get paid profits, they get paid salaries); in a decent shop, geeks program at flank speed all the time and always give the project 100%.
See Ed Yourdon's Death March or any of Ward Cunningham's Extreme Programming books for more details, and ways to avoid the above traps. Yourdon suggests that the head geek has to take a hard stand in scheduling to prevent business interests from setting both the project spec and the ship date. He especially tells you never to negotiate schedule, and to help the suits understand why you never do. Whatever number you estimate doesn't affect the actual ship date, so playing with that number is simply fooling yourself.
Extreme Programming actually has a "planning game" (sort of a ritual dance) which places business interests and geeks on the same side of the table. Two big rules are "The geeks may not reject any part of the spec" and "The suits may not reject any part of the estimate". Once the suits set the spec, both teams break it down into pieces-parts, line them up in order of what gets done first and the geeks give their estimates. From there, the suits can choose the ship date (and can instantly see how much product will be ready by then), or can choose a certain amount of project completion (and can instantly see the ship date). The fun part about this method is that the suits can change their minds at any time by changing, adding, or removing pieces-parts, and can instantly see how that affects the ship date. The other fun part is that breaking up the project into pieces-parts allows developers to do a (small) project they estimated. This allows people to track estimated versus real time, and to give developers feedback that lets them make better estimates. Such a team will start off with bad estimates like everybody else, but they will be able to improve rapidly.
--The basis of all love is respect
The truth is, you can somewhat accurately estimate project time. The problem is, few know how.
The thing is, you must get entirely through the design stages first. The design stages should include every screen as well as every possible error message, sub-screen, or whatever can pop up, as well as an outline of how the program flow will go. This takes a lot of time, but not quite as much as it sounds.
Once you have done the complete design, you can accurately make schedules. The problem is, most programmers put all error handling and messaging off as something that doesn't need to be designed. That's where the extra time comes in. If you know _exactly_ how the program flow is supposed to work, estimating time is easy. However, if you haven't finished the design stage, YOU DON'T KNOW WHAT YOU'RE PROGRAMMING, so, obviously, you can't estimate the time. So, with a _complete_ design, including all possible error conditions and actions to be taken, scheduling is not that hard.
Engineering and the Ultimate
"Suffering" from it right now, AAMOF...
1. Programmer comes up with new system in spare time while learning a language. New system, if polished, would actually make a nice application to sell to current clients. Programmer is excited, and shows "product" to highers-ups.
2. Higher-ups are excited, can see it may take a bit more work, and look into what it would take to get it to market. They tell sales and marketing to go see the programmer to have him demo it to them.
3. Programmer is excited, shows it to sales and marketing. Sales and marketing love it.
4. Months pass. Unbeknownst to the programmer, sales and marketing have sold it to a client, as part of the contract, to be a finished package by the end of the year - OR ELSE.
5. More months pass - higher ups finally tell programmer, and others, that this new system is wanted - and oh, BTW, it is wanted in Java - not in the VB it was shown it.
6. Three months are left to complete the project. Original programmer knows little Java. Other Java coders know little Swing. Architecture of app is changed from a simple app to a three-tier client-server system. Only two other coders have sufficient Java experience to code on it. The lead of the project knows no Java, and only takes notes at meetings.
7. Twenty-one days until deadline (ie, it has to be in QA in 21 days) - everyone sweating bullets knowing it can't be done. Oh, and BTW, at every meeting it seems like a new section not planned for is realized...
It was an ad-hoc system, and it is progressing as an ad-hoc system - a system that should have NEVER been shown to marketing and sales. I am not the programmer who originated it, but suffice to say it is a system that will be nice for our clients once it is completed. Fortunately, it sounds like things will be able to be smoothed over if we miss the deadline...
So remember, all you budding coders out there - if you create something in your "learning" time - don't show it to anyone BUT other coders. If marketing and sales come around, have them sign an NDA promising not to sell it or something - you don't want to release a product to market before it is done - quit "selling" vaporware!!!
Reason is the Path to God - Anon
As silly as this paper is, most responses to it are off-topic. What he is trying to show is that there is a good case for saying there is no general, algorithmic way to estimate how long it will take to do a given software project. What he isn't saying is that you can not make reasonable estimates on a given project.
This reminds me of "The New Jersey Method versus the MIT method."
The MIT Method is to take as long as needed to get a task done "right," regardless of cost and schedules.
The New Jersey method calls for solving 80% of the problem, and putting off 20% until later.
The MIT method results in more project failures than the New Jersey method. Microsoft epitomizes the New Jersey method, as does open source. Multics followed the MIT method, and was never actually finished, just killed off years later...
If anyone has a reference for the "MIT vs NJ" in its original form, please post it.
Napster-to-go says "Fill and refill your compatible MP3 player", which is a lie. It's not MP3. It's WMA with DRM.
Yep, there's a deeper problem, and it's very simple. Suppose your manager asks you for an estimate, and you say "six months" because that's how long you think it will take. Your manager works out that the project will not succeed if it takes six months, and asks you if you can do it in four. If you say "Yes", you have just become a statistic.
Saying yes does not mean that you can do it if you couldn't before, it just means that you have lied to management, prevented them from doing their job properly. If your project would take six months, but it will not make money if it takes six months, then you simply should not start that project. Failing to realise that simple fact is the major cause of late/failed projects, IME.
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
Paul Robinson <Postmaster@paul.washington.dc.us>
The lessons of history teach us - if they teach us anything - that nobody learns the lessons that history teaches us.
To take a reductio ad absurdum:
You are given the task of duplicating the functionality of Windows NT. Furthermore, you are given the source code for Windows NT in a .tgz file and the associated development environment within which that source code can be tested. The question now degenerates into "How long does it take me to copy the tgz file?" That can be accurately predicted by measuring how long it takes to copy files on that environment in general, and the estimated schedule can be predicted to absurdly high degrees of accuracy with enough benchmarks of the system's file copying performance.
Here's another reduced complexity angle:
Translate a program written in Visual Basic and convert it to C++ (readably).
You actually can sit down and convert a sampling of the program and get a measure of how long it will take you to do the whole thing -- the more you sample, the more accurate the measure right up to the point where you have converted the whole thing.
Here's another example with a bit less reduction in complexity:
You are given a working program but no source code, and some expert users of that program. Here we are getting into what might be thought of as "function point analysis" but really, it is much easier and more accurate than that since the program exists and works as it is "supposed" to work, you can bang away on it, and the expert users can bang away on your version of it to ensure it meets their needs -- perhaps discovering that some of the features in the old program were not really used thereby simplifying the task.
Each step has been away from the "absurd" position of simply copying a program which was, in a sense, a "spec" for itself.
At the other extreme, we get to the problem of "write a program that will make me as rich as Bill Gates". Note that this specification is not very specific.... it is very far from being source code for a program you can simply copy, isn't it? Guess what that says about the accuracy of the schedule?
So a lot of this hubub about estimating software schedules is really hubub about the nature of the program specificiation process.
Seastead this.
It looks like several people (well, more than several) posted responses without reading beyond the lead-in. If you're one of them, yes, the argument here is in the general ballpark of "software estimation is hard or impossible", but it actually says something more specific than that.
The article does NOT say the following:
The article DOES say
From this, it does NOT conclude either of the points 1,2 above. Instead, it concludes:
Now some of the response posts, paraphrased:
No, it does not say this.
It also does not say this.
No, the article distinguishes subjective and objective estimates, and specifically discusses the case of an objective estimate with bounds in detail.
Ok, but slightly off topic: the article is specifically talking about those who claim objective estimates.
And where did you get an objective estimate of the complexity of a new project? Read the article...
Yes, you are. Your boss is monitoring you, get back to work.
Certainly. The 'manufacturing' camp of software estimators (Humphrey quote in the supplementary material) say or hint that software construction can be made into a repeatable, fairly boring process where projects are always on time and programmers are like factory workers. This may or may not be true (I don't think it is), but regardless: to make this view seem more science than philosophy some of these people have fallen into the trap of cloaking their estimating process with formal notation and claiming or hinting objectivity. This part is wrong.
On the contrary, [conclusions to the article and the supplementary material]: