Can Software Schedules Be Estimated?

J.P.Lewis writes " Is programming like manufacturing, or like physics? We sometimes hear of enormous software projects that are canceled after running years behind schedule. On the other hand, there are software engineering methodologies (inspired by similar methodologies in manufacturing) that claim (or hint at) objective estimation of project complexity and development schedules. With objective schedule estimates, projects should never run late. Are these failed software projects not using proper software engineering, or is there a deeper problem?" Read on for one man's well-argued answer, which casts doubt on most software-delivery predictions, and hits on a few of the famous latecomers.

"A recent academic paper Large Limits to Software Estimation (ACM Software Engineering Notes, 26, no.4 2001) shows how software estimation can be interpreted in algorithmic (Kolmogorov) complexity terms. An algorithmic complexity variant of mathematical (Godel) incompleteness can then easily be interpreted as showing that all claims of purely objective estimation of project complexity, development time, and programmer productivity are incorrect. Software development is like physics: there is no objective way to know how long a program will take to develop."

Lewis also provides a link to this "introduction to incompleteness (a fun subject in itself) and other background material for the paper."

Comments

Of course they can be estimated.

[Anton Anatopopov]

But not with any degree of accuracy. Function point analysis is one method that has had some success. The key to delivering projects on time always has been and always will be RISK MANAGEMENT.

Software development is not a science in the normal sense. Designing large software systems is an art. It cannot be pigeonholed. Stroustrup has a lot to say about this when he describes the 'interchangable morons' concept in the 2nd edition C++ book.

Anyway, read Death march by Ed Yourdon, and the mythical man month by fred brooks, and antipatterns, any time someone asks you for an estimate say 'two weeks' and then bullshit from there on.

That is how it works in the real world. The numbers are essentially meaningless, but the bean counters and suits have to justify their existance somehow :-)

Can you imagine asking Linus when 2.5 will be ready ?

Re:Of course they can be estimated.

[KyleCordes]

This approach applies, more or less, sometimes MUCH less, depending on how well understood the problem domain is, how many times you have done it before.

If you're building your 57th e-commerce web site, which works roughly like the 56 you build before, you can estimate very, very well, and you can reduce coding to nearly data entry.

If you're solving a problem of unknown scope, which your team has not solved before, which the solution is not clear to, and analysis has revealed some but not all of the details, etc., then you are not very right.

Re:Of course they can be estimated.

[xyzzy]

I agree with your risk management comment, and a later poster who mentioned fixing the endpoint, but I'm not sure I agree on your claim that it can't be pinpointed with any degree of accuracy.

After ~15 years in the industry, I've found that one thing that makes a huge difference is the experience of the team, and the familiarity between the actual engineers and the project management.

As you have experience solving a variety of classes of problems, you can predict with increasing accuracy the time it'll take you to solve later problems. And as your management sees you getting increasingly accurate in your estimates (based on past projects) they can create better and better schedules and estimates for the project as a whole, and have a better intuition for the gray areas of development, or the greener developers.

Projects that tend to go off into the weeds have included (in my experience) wholly green teams, wholly green management, or areas of development that are outside the areas of expertise of one or both.

Re:Of course they can be estimated.

[john@iastate.edu]

Writing software is not like building bridges because halfway through the project some dumbass from marketing doesn't come down and tell you that concrete is out and so it needs to be a steel bridge. Oh, and those tacky cables have got to go -- the focus group hated them.

Re:Of course they can be estimated.

[clare-ents]

"
That's exactly the sort of attitude that has caused the sort of spectactular failures of software projects to be accepted as the norm. Software Engineering is *not* "hacking" or "coding" or "programming", it's *engineering*, like building a bridge or a skyscraper. Yes, those projects go over time and budget too sometimes, but they are the exception rather than the rule.
"

But that's simply not true. Writing software of anything that is non-trivial is not the same as straightforward engineering. For a start there is the rate of progress, how many people have 30 years + experience of building 50 story + buildings. How many people have 30 years + experience of dealing with terabyte + sized datasets?

When buildling software previous code can be reused for a very small amount of effort, when building skyscrapers the previous design can be reused for only marginally less effort than the last one.

Compare the difference between building a C compiler from the gcc source and the world trade centre from the blueprints.

Essentially the estimate is

Time = [time to do the bits we know how to do [accurate] ] + [guess for the bits we don't know how to do [inaccurate] ]

With software, the first part of that expression tends towards zero since most things we know how to do we can reuse code, whereas with building it remains a large accurate estimate.

The error here will be of the form

Error = [variance of inaccurate terms] / [total]

For the example of a skyscraper whos construction is mostly a known method this will tend to a small number since the inaccuate term is much smaller than the accurate term, but for software with reuse of all the known methods of coding this will tend to 1 - i.e.. 100% error in the estimate and hence the conclusion that it's worthless to even bother estimating.

In my company we can accurately estimate how long projects will take providing the projects are mostly identical to ones we have done before, and if this is the case it generally costs the client more in programmer time in meetings to dicuss the cost of the job than it does to write it.

Re:Of course they can be estimated.

[Overt Coward]

The key to function points -- or any other -- estimation techniques is relying on historical data to predict future results. This means that they are fairly accurate as long as you collect metrics and stay within the same general project domain and relative project size. The more radical the departure from historical size or domain the new project is, the less accurate an estimate will be.

However, the biggest thing to remember is that no matter what estimation method is used, the simple fact that a methodical approach to analyzing the problem will almost always yield a reasonable estimate.
The main reasons projects go over schedule and budget are:

1. 
   
2. "Feature creep" -- having the requirements change significantly over the course of the project without adding the impact of changes into the schedule.
   
   
3. Rampant optimism -- many engineers (and managers) will typically estimate how long they think it should take to do a specific task but will not add in a buffer in case somthing goes wrong. And something always goes wrong.
   
   
4. Artificial deadlines -- project schedules where the budget (time and money) was set by customer/marketing committments, and not by the technical requirements at all.
   
   
5. Calendar/personnel issues -- people take vacations, there are holidays, and people occasionally fall ill. Plan for it. Also, don't forget any company/department meetings, training, seminars, etc.
   
   
6. Dependencies -- if a required piece of hardware or software won't be available (or is late), it can impact the overall schedule, espeecially if critical path tasks depnd on those materials.
   

Risk management is indeed the key. As the project manager or lead engineer, it is your job to predict what potential risks might be and attempt to mitigate them on a cost-effectiveness basis. You can still be bit by bad luck, but you can minize the chances it will strike.

Re:In all seriousness, this is the wrong place to

[Organic_Info]

True but most experienced S/W engineers or Project managers know that most projects slip because of changes to/deviations from the original project spec.

Fixed specs are much easier to engineer than those that continually change. You wouldn't easily engineer a bridge if the river banks kept moving.

I think experienced project managers know how to control the spec rather than the project. (I could be wrong - It's just what I've seen).

Re:from a Consulting viewpoint..

[markmoss]

To accuratly plan a software release you must have the project, and all it's complexities and nuances down COLD. otherwise you are not giving an estimation, you are giving a guess based upon incomplete knowledge.

The bulk of the work of programming consists of getting all the complexities and nuances down cold. Once you really and completely understand what is required, coding is trivial.

This leads to a thoroughly unrealistic method of estimating software costs:
1) Work for months on the specs.
2) Get the customer to sign on to those incredibly detailed specs, even though he doesn't understand them.
3) Go and code it, no spec changes allowed.

8-)

The article mainly talks about the mathematics of estimating complexity. This is a lot like the proof that you cannot determine when or whether a computer program will end -- it's true for pathological programs, but it has little relevance for the real world. You try to write the code so the conditions for the program to end are clear. If it gets into an endless loop, you probably got a conditional expression backwards and you'll recognize it immediately once you figure out which loop is running endlessly... Likewise, there may be well-defined specifications for which it is impossible to estimate the coding time, but the usual problem is poorly-defined specs, which obviously makes any estimate a guess.

Estimation is very possible.

[Kefaa]

The issue is not physics versus manufacturing, it is scope and cost containment like is done in manufacturing. As a person who has lead multi-million dollar projects, I have grown used to the cliché that goes something like this:
If we built homes like software we would all be living in the street, penniless...

The major issues I have seen revolve around a lack of scope and cost control. In many cases it is because there is little penalty for being late or over budget. In cases where penalties exist it is often beneficial to then over estimate the effort or cost required. Then once the money is approved, using it is becomes easy.

Going back to the analogy consider the following:
Scope
If you were building a house, each piece has a specified cost, known in advance to a very large degree. In addition, altering the scope itself often incurs a penalty, because the work is not done by the owner. You plan a three bedroom, 1.5 bath home. Midway through planning you decide to make it a two bath home instead. The architect will charge the "re-scoping" fee and the builder will add the material fee. Now do the same after construction has begun. The architect gets their fee, the builder adds the material and resource costs, plus a "revision" fee for changing your mind after construction begins.

During a software project, it is common for individuals to approach the developers and ask to expand the scope. This would be analogous to approaching one of the work crew and asking them to just add the extra half a bath. The difference is the work crew would get fired, and the developer gets bonus points for adding the feature, either directly or indirectly.

If the developer chooses not to do it, or pushes them to the project manager, the client may label them uncooperative or difficult to work with. The project manager not wanting to be labeled either may coerce, cajole, or beg the developer to accomplish it, without a scope revision. Failure to do so by the developer results in real financial impact at some point, and offers little incentive to hold the line.

Cost
I call this the "Porsche syndrome".

I go into the Porsche dealership and see a new 911 Carrera Coupe. Smiling the dealer offers to sell it at a deep discount, with options and accessories $84,000 (U.S.). Whewwww baby!!! I cannot afford that. "Look," I tell him, "my wife will never approve that, you need to get it down to $28,500 tops." Would any of us expect to have the price cut down? By half or more?

Okay, how about "Look, what will it take to get it under $30,000? Seriously now, what do I have to give up" As the dealer is escorting me to the door he explains the only way I will get this car under $30k is with a mask and a gun or from a scrap metal dealer.

Yet, daily we go to developers and tell them to do the same. We ask for an estimate and then go back with "This is too much, it needs to be smaller or it won't get approved!" --Insert blank stare here--- The idea that if something cannot be cost justified it should not be done, is often lost in the "request" itself.

To nearly guarantee a project is on budget and time requires things many companies are unwilling to provide. Strict scope control procedures, with oversight by the person responsible for the money. That means each change, regardless of how trivial must be approved by someone above the project management team with business justification. It also means that requests for scope change cannot be made to developers directly, by anyone.

I was very happy with the people who built my home. When speaking to many of my friends and coworkers who built their homes, they describe it as a process akin to having their flesh removed. Everything required such effort and detail that many would not do it again.

Most of them were looking for the relationship to be like one at the office. We all want to get along and help each other out. This is not a commercial arrangement, and when we put the commercial context around it, we see it many offices lack structure.

Internal organizations can be setup like commercial ones, but it is usually unwelcome as the perception is everyone should be working for the greater good of the company and this has the appearance of bureaucracy. Even if inaccurate, everyone "wanting to get along" prevents it from being implemented.

More mundane reasons for underestimations

[remande]

Software development isn't always like physics--often we are boldly going where people have gone before. However, certain factors in software houses cause underestimations:

Underestimation as a Marketing Tactic
AKA "Vaporware". Even if marketing knew when a product would be shippable, a particularly cinical marketing department may claim it to be earlier, thus freezing competitor's development.

Lack of Feedback (Moving Targets)
Software engineers are particularly bad at estimating because they have never done what they estimated. They are given a large project, give a large estimate, start working on it, and the project changes in the middle in a major way. This is a moving target; the estimate no longer applies. Major law of software development: You cannot change the spec or the development team on the project without impacting the real ship date. If you don't re-assess the estimated ship date, you are simply fooling yourself. Thus, they don't have any clue whether they hit the estimate or not. One way to defend against this is to break the project down into bite-sized pieces and estimate them; a small piece gives you a chance to do precisely what you estimated. Once you have that, you can have somebody track your estimates, and come back saying something like "On average, you go one third over your estimates. Add a third to your estimates from now on, and we'll be accurate".

Management Estimates
Often, engineers don't do the estimate. The management or marketing people tell you what must be done, and how long you have. Sometimes this is done explicitly; other times, management may have a number in mind and shame a software team into agreeing with it by laughing off any number that doesn't match theirs. Business people often negotiate the ship date with the geeks, like any negotiate with any other vendor. To a suit, vendor negotiations are how you determine the "margin", or how much the vendor is making (like when you buy a car, you and the dealer come to a number that determines the dealer's margin). This doesn't work in in-house software develoment because geeks hold back precious little "slack" or "margin" (they don't get paid profits, they get paid salaries); in a decent shop, geeks program at flank speed all the time and always give the project 100%.

See Ed Yourdon's Death March or any of Ward Cunningham's Extreme Programming books for more details, and ways to avoid the above traps. Yourdon suggests that the head geek has to take a hard stand in scheduling to prevent business interests from setting both the project spec and the ship date. He especially tells you never to negotiate schedule, and to help the suits understand why you never do. Whatever number you estimate doesn't affect the actual ship date, so playing with that number is simply fooling yourself.

Extreme Programming actually has a "planning game" (sort of a ritual dance) which places business interests and geeks on the same side of the table. Two big rules are "The geeks may not reject any part of the spec" and "The suits may not reject any part of the estimate". Once the suits set the spec, both teams break it down into pieces-parts, line them up in order of what gets done first and the geeks give their estimates. From there, the suits can choose the ship date (and can instantly see how much product will be ready by then), or can choose a certain amount of project completion (and can instantly see the ship date). The fun part about this method is that the suits can change their minds at any time by changing, adding, or removing pieces-parts, and can instantly see how that affects the ship date. The other fun part is that breaking up the project into pieces-parts allows developers to do a (small) project they estimated. This allows people to track estimated versus real time, and to give developers feedback that lets them make better estimates. Such a team will start off with bad estimates like everybody else, but they will be able to improve rapidly.