Can Software Schedules Be Estimated?

J.P.Lewis writes " Is programming like manufacturing, or like physics? We sometimes hear of enormous software projects that are canceled after running years behind schedule. On the other hand, there are software engineering methodologies (inspired by similar methodologies in manufacturing) that claim (or hint at) objective estimation of project complexity and development schedules. With objective schedule estimates, projects should never run late. Are these failed software projects not using proper software engineering, or is there a deeper problem?" Read on for one man's well-argued answer, which casts doubt on most software-delivery predictions, and hits on a few of the famous latecomers.

"A recent academic paper Large Limits to Software Estimation (ACM Software Engineering Notes, 26, no.4 2001) shows how software estimation can be interpreted in algorithmic (Kolmogorov) complexity terms. An algorithmic complexity variant of mathematical (Godel) incompleteness can then easily be interpreted as showing that all claims of purely objective estimation of project complexity, development time, and programmer productivity are incorrect. Software development is like physics: there is no objective way to know how long a program will take to develop."

Lewis also provides a link to this "introduction to incompleteness (a fun subject in itself) and other background material for the paper."

Comments

Of course they can be estimated.

[Anton Anatopopov]

But not with any degree of accuracy. Function point analysis is one method that has had some success. The key to delivering projects on time always has been and always will be RISK MANAGEMENT.

Software development is not a science in the normal sense. Designing large software systems is an art. It cannot be pigeonholed. Stroustrup has a lot to say about this when he describes the 'interchangable morons' concept in the 2nd edition C++ book.

Anyway, read Death march by Ed Yourdon, and the mythical man month by fred brooks, and antipatterns, any time someone asks you for an estimate say 'two weeks' and then bullshit from there on.

That is how it works in the real world. The numbers are essentially meaningless, but the bean counters and suits have to justify their existance somehow :-)

Can you imagine asking Linus when 2.5 will be ready ?

Re:Of course they can be estimated.

[sql*kitten]

Software development is not a science in the normal sense. Designing large software systems is an art. It cannot be pigeonholed

That's exactly the sort of attitude that has caused the sort of spectactular failures of software projects to be accepted as the norm. Software Engineering is *not* "hacking" or "coding" or "programming", it's *engineering*, like building a bridge or a skyscraper. Yes, those projects go over time and budget too sometimes, but they are the exception rather than the rule.

That is how it works in the real world. The numbers are essentially meaningless, but the bean counters and suits have to justify their existance somehow

The problem is endemic in the industry. The other Engineering professions require rigorous accreditation before they let practitioners loose in the world, like the PE (in the US) or the Charter (in the UK). But the software industry hires anyone, and lets them get on with whatever they do, with no real management or oversight or planning.

In a well analyzed and properly planned project, the actual coding stage is little more than data entry.

Re:Of course they can be estimated.

[KyleCordes]

This approach applies, more or less, sometimes MUCH less, depending on how well understood the problem domain is, how many times you have done it before.

If you're building your 57th e-commerce web site, which works roughly like the 56 you build before, you can estimate very, very well, and you can reduce coding to nearly data entry.

If you're solving a problem of unknown scope, which your team has not solved before, which the solution is not clear to, and analysis has revealed some but not all of the details, etc., then you are not very right.

Re:Of course they can be estimated.

[xyzzy]

I agree with your risk management comment, and a later poster who mentioned fixing the endpoint, but I'm not sure I agree on your claim that it can't be pinpointed with any degree of accuracy.

After ~15 years in the industry, I've found that one thing that makes a huge difference is the experience of the team, and the familiarity between the actual engineers and the project management.

As you have experience solving a variety of classes of problems, you can predict with increasing accuracy the time it'll take you to solve later problems. And as your management sees you getting increasingly accurate in your estimates (based on past projects) they can create better and better schedules and estimates for the project as a whole, and have a better intuition for the gray areas of development, or the greener developers.

Projects that tend to go off into the weeds have included (in my experience) wholly green teams, wholly green management, or areas of development that are outside the areas of expertise of one or both.

Re:Of course they can be estimated.

[clare-ents]

"
That's exactly the sort of attitude that has caused the sort of spectactular failures of software projects to be accepted as the norm. Software Engineering is *not* "hacking" or "coding" or "programming", it's *engineering*, like building a bridge or a skyscraper. Yes, those projects go over time and budget too sometimes, but they are the exception rather than the rule.
"

But that's simply not true. Writing software of anything that is non-trivial is not the same as straightforward engineering. For a start there is the rate of progress, how many people have 30 years + experience of building 50 story + buildings. How many people have 30 years + experience of dealing with terabyte + sized datasets?

When buildling software previous code can be reused for a very small amount of effort, when building skyscrapers the previous design can be reused for only marginally less effort than the last one.

Compare the difference between building a C compiler from the gcc source and the world trade centre from the blueprints.

Essentially the estimate is

Time = [time to do the bits we know how to do [accurate] ] + [guess for the bits we don't know how to do [inaccurate] ]

With software, the first part of that expression tends towards zero since most things we know how to do we can reuse code, whereas with building it remains a large accurate estimate.

The error here will be of the form

Error = [variance of inaccurate terms] / [total]

For the example of a skyscraper whos construction is mostly a known method this will tend to a small number since the inaccuate term is much smaller than the accurate term, but for software with reuse of all the known methods of coding this will tend to 1 - i.e.. 100% error in the estimate and hence the conclusion that it's worthless to even bother estimating.

In my company we can accurately estimate how long projects will take providing the projects are mostly identical to ones we have done before, and if this is the case it generally costs the client more in programmer time in meetings to dicuss the cost of the job than it does to write it.

Re:Of course they can be estimated.

[Overt Coward]

The key to function points -- or any other -- estimation techniques is relying on historical data to predict future results. This means that they are fairly accurate as long as you collect metrics and stay within the same general project domain and relative project size. The more radical the departure from historical size or domain the new project is, the less accurate an estimate will be.

However, the biggest thing to remember is that no matter what estimation method is used, the simple fact that a methodical approach to analyzing the problem will almost always yield a reasonable estimate.
The main reasons projects go over schedule and budget are:

1. 
   
2. "Feature creep" -- having the requirements change significantly over the course of the project without adding the impact of changes into the schedule.
   
   
3. Rampant optimism -- many engineers (and managers) will typically estimate how long they think it should take to do a specific task but will not add in a buffer in case somthing goes wrong. And something always goes wrong.
   
   
4. Artificial deadlines -- project schedules where the budget (time and money) was set by customer/marketing committments, and not by the technical requirements at all.
   
   
5. Calendar/personnel issues -- people take vacations, there are holidays, and people occasionally fall ill. Plan for it. Also, don't forget any company/department meetings, training, seminars, etc.
   
   
6. Dependencies -- if a required piece of hardware or software won't be available (or is late), it can impact the overall schedule, espeecially if critical path tasks depnd on those materials.
   

Risk management is indeed the key. As the project manager or lead engineer, it is your job to predict what potential risks might be and attempt to mitigate them on a cost-effectiveness basis. You can still be bit by bad luck, but you can minize the chances it will strike.

Estimates based on motivation

[ciurana]

My company develops turn-key systems. Sometimes we also develop custom solutions for our customers. Our customer base has increased steadily after the dotcom crash, when we switched from products to services. One of the reasons our customers like us is that we don't bill projects by the hour. We will the project on a fixed price, not to exceed, basis.

The programmers who work with us on a contract basis don't bill us by the hour either. After we have the design and we distribute tasks and prior to submitting the final estimate, we ask contractors to place a fixed bid.

We've done six major projects like this since March, and in all cases we finished within budget and on-schedule, and the systems are currently in production. They are all mission-critical systems running in either robotics environments or high-availability networks.

Our economic motivation is then to do things well and quickly in order to increase our profits. That also enables us to move on to the next project faster than slaving over some customer in order to bill the maximum hours.

As far as development techniques go, we adopted XP earlier on and it's working for us.

Cheers!

E

Estimation is very possible.

[Kefaa]

The issue is not physics versus manufacturing, it is scope and cost containment like is done in manufacturing. As a person who has lead multi-million dollar projects, I have grown used to the cliché that goes something like this:
If we built homes like software we would all be living in the street, penniless...

The major issues I have seen revolve around a lack of scope and cost control. In many cases it is because there is little penalty for being late or over budget. In cases where penalties exist it is often beneficial to then over estimate the effort or cost required. Then once the money is approved, using it is becomes easy.

Going back to the analogy consider the following:
Scope
If you were building a house, each piece has a specified cost, known in advance to a very large degree. In addition, altering the scope itself often incurs a penalty, because the work is not done by the owner. You plan a three bedroom, 1.5 bath home. Midway through planning you decide to make it a two bath home instead. The architect will charge the "re-scoping" fee and the builder will add the material fee. Now do the same after construction has begun. The architect gets their fee, the builder adds the material and resource costs, plus a "revision" fee for changing your mind after construction begins.

During a software project, it is common for individuals to approach the developers and ask to expand the scope. This would be analogous to approaching one of the work crew and asking them to just add the extra half a bath. The difference is the work crew would get fired, and the developer gets bonus points for adding the feature, either directly or indirectly.

If the developer chooses not to do it, or pushes them to the project manager, the client may label them uncooperative or difficult to work with. The project manager not wanting to be labeled either may coerce, cajole, or beg the developer to accomplish it, without a scope revision. Failure to do so by the developer results in real financial impact at some point, and offers little incentive to hold the line.

Cost
I call this the "Porsche syndrome".

I go into the Porsche dealership and see a new 911 Carrera Coupe. Smiling the dealer offers to sell it at a deep discount, with options and accessories $84,000 (U.S.). Whewwww baby!!! I cannot afford that. "Look," I tell him, "my wife will never approve that, you need to get it down to $28,500 tops." Would any of us expect to have the price cut down? By half or more?

Okay, how about "Look, what will it take to get it under $30,000? Seriously now, what do I have to give up" As the dealer is escorting me to the door he explains the only way I will get this car under $30k is with a mask and a gun or from a scrap metal dealer.

Yet, daily we go to developers and tell them to do the same. We ask for an estimate and then go back with "This is too much, it needs to be smaller or it won't get approved!" --Insert blank stare here--- The idea that if something cannot be cost justified it should not be done, is often lost in the "request" itself.

To nearly guarantee a project is on budget and time requires things many companies are unwilling to provide. Strict scope control procedures, with oversight by the person responsible for the money. That means each change, regardless of how trivial must be approved by someone above the project management team with business justification. It also means that requests for scope change cannot be made to developers directly, by anyone.

I was very happy with the people who built my home. When speaking to many of my friends and coworkers who built their homes, they describe it as a process akin to having their flesh removed. Everything required such effort and detail that many would not do it again.

Most of them were looking for the relationship to be like one at the office. We all want to get along and help each other out. This is not a commercial arrangement, and when we put the commercial context around it, we see it many offices lack structure.

Internal organizations can be setup like commercial ones, but it is usually unwelcome as the perception is everyone should be working for the greater good of the company and this has the appearance of bureaucracy. Even if inaccurate, everyone "wanting to get along" prevents it from being implemented.

Just like building a skyscraper

[Aceticon]

Let's see:

• At any point in time the ground your skyscraper stands on can crumble into nothingness. [Operating System bugs]
• Your skyscraper can be required to stand on slightly different types of ground. [Operating System types and versions]
• Also the steel, glass and cement you are using have wildly varieing properties. They also might have been imposed by an outside entity (read Company Standarts). [Third Party Components]
• Plus the elevators that you get always do less than their specifications (for example they don't stop on the 5th floor). The next version of the elevator will actually do that but on the other hand it doesn't fit on the elevator shaft.[Third Party Components and Applications]
• Also half-way through building the skyscraper you find out that the plant has been changed and it's now supposed to have a Shopping Mall on the ground floor.[Creeping Requirements]

More mundane reasons for underestimations

[remande]

Software development isn't always like physics--often we are boldly going where people have gone before. However, certain factors in software houses cause underestimations:

Underestimation as a Marketing Tactic
AKA "Vaporware". Even if marketing knew when a product would be shippable, a particularly cinical marketing department may claim it to be earlier, thus freezing competitor's development.

Lack of Feedback (Moving Targets)
Software engineers are particularly bad at estimating because they have never done what they estimated. They are given a large project, give a large estimate, start working on it, and the project changes in the middle in a major way. This is a moving target; the estimate no longer applies. Major law of software development: You cannot change the spec or the development team on the project without impacting the real ship date. If you don't re-assess the estimated ship date, you are simply fooling yourself. Thus, they don't have any clue whether they hit the estimate or not. One way to defend against this is to break the project down into bite-sized pieces and estimate them; a small piece gives you a chance to do precisely what you estimated. Once you have that, you can have somebody track your estimates, and come back saying something like "On average, you go one third over your estimates. Add a third to your estimates from now on, and we'll be accurate".

Management Estimates
Often, engineers don't do the estimate. The management or marketing people tell you what must be done, and how long you have. Sometimes this is done explicitly; other times, management may have a number in mind and shame a software team into agreeing with it by laughing off any number that doesn't match theirs. Business people often negotiate the ship date with the geeks, like any negotiate with any other vendor. To a suit, vendor negotiations are how you determine the "margin", or how much the vendor is making (like when you buy a car, you and the dealer come to a number that determines the dealer's margin). This doesn't work in in-house software develoment because geeks hold back precious little "slack" or "margin" (they don't get paid profits, they get paid salaries); in a decent shop, geeks program at flank speed all the time and always give the project 100%.

See Ed Yourdon's Death March or any of Ward Cunningham's Extreme Programming books for more details, and ways to avoid the above traps. Yourdon suggests that the head geek has to take a hard stand in scheduling to prevent business interests from setting both the project spec and the ship date. He especially tells you never to negotiate schedule, and to help the suits understand why you never do. Whatever number you estimate doesn't affect the actual ship date, so playing with that number is simply fooling yourself.

Extreme Programming actually has a "planning game" (sort of a ritual dance) which places business interests and geeks on the same side of the table. Two big rules are "The geeks may not reject any part of the spec" and "The suits may not reject any part of the estimate". Once the suits set the spec, both teams break it down into pieces-parts, line them up in order of what gets done first and the geeks give their estimates. From there, the suits can choose the ship date (and can instantly see how much product will be ready by then), or can choose a certain amount of project completion (and can instantly see the ship date). The fun part about this method is that the suits can change their minds at any time by changing, adding, or removing pieces-parts, and can instantly see how that affects the ship date. The other fun part is that breaking up the project into pieces-parts allows developers to do a (small) project they estimated. This allows people to track estimated versus real time, and to give developers feedback that lets them make better estimates. Such a team will start off with bad estimates like everybody else, but they will be able to improve rapidly.

Re:Software Schedules

[johnnyb]

The truth is, you can somewhat accurately estimate project time. The problem is, few know how.

The thing is, you must get entirely through the design stages first. The design stages should include every screen as well as every possible error message, sub-screen, or whatever can pop up, as well as an outline of how the program flow will go. This takes a lot of time, but not quite as much as it sounds.

Once you have done the complete design, you can accurately make schedules. The problem is, most programmers put all error handling and messaging off as something that doesn't need to be designed. That's where the extra time comes in. If you know _exactly_ how the program flow is supposed to work, estimating time is easy. However, if you haven't finished the design stage, YOU DON'T KNOW WHAT YOU'RE PROGRAMMING, so, obviously, you can't estimate the time. So, with a _complete_ design, including all possible error conditions and actions to be taken, scheduling is not that hard.

Why almost all the posts are off topic.

[the_great_cornholio]

As silly as this paper is, most responses to it are off-topic. What he is trying to show is that there is a good case for saying there is no general, algorithmic way to estimate how long it will take to do a given software project. What he isn't saying is that you can not make reasonable estimates on a given project.

comment on the posts

[noisebrain]

It looks like several people (well, more than several) posted responses without reading beyond the lead-in. If you're one of them, yes, the argument here is in the general ballpark of "software estimation is hard or impossible", but it actually says something more specific than that.

The article does NOT say the following:

1. software estimation is impossible
2. objective software estimation is impossible therefore software estimation is impossible

The article DOES say

• Various software engineering authorities claim that objective software estimation is possible [paragraph 3, quotes on first page].
• objective software estimation is in fact not possible [body of article]

From this, it does NOT conclude either of the points 1,2 above. Instead, it concludes:

• Software construction is inherently creative and subjective, having more in common with physics than manufacturing; software estimation is inherently subjective [conclusion, Bollinger quote].
• Because software is used in the government, in vehicles, and other places where it can potentially have a negative on people's lives, we (software writers) have an ethical responsibility to not over-represent our ability to estimate (especially when it comes to estimation of software quality- r.e. correctness claim in the supplementary material).

Now some of the response posts, paraphrased:

• "The article says that estimation must be objective rather than subjective"
  No, it does not say this.

• "The article says that subjective software estimation is not useful"
  It also does not say this.

• "The article says that we are looking for exact answers, not estimates" or "the article doesn't understand what `estimate' means"
  No, the article distinguishes subjective and objective estimates, and specifically discusses the case of an objective estimate with bounds in detail.

• "People/organizations can make accurate estimates, I made one last week" or "Estimation is hard, I double my estimates and still miss them".
  Ok, but slightly off topic: the article is specifically talking about those who claim objective estimates.

• "You can do objective estimation, and I did it last week using COCOMO"
  And where did you get an objective estimate of the complexity of a new project? Read the article...

• "I think I'm the only person who has read this far".
  Yes, you are. Your boss is monitoring you, get back to work.

• "Software estimation needs common sense, not advanced mathematics."
  Certainly. The 'manufacturing' camp of software estimators (Humphrey quote in the supplementary material) say or hint that software construction can be made into a repeatable, fairly boring process where projects are always on time and programmers are like factory workers. This may or may not be true (I don't think it is), but regardless: to make this view seem more science than philosophy some of these people have fallen into the trap of cloaking their estimating process with formal notation and claiming or hinting objectivity. This part is wrong.

  On the contrary, [conclusions to the article and the supplementary material]:

  Good estimation therefore requires experience and judgment. The software industry should value human experience, intuition, and wisdom rather than claiming false objectivity and promoting entirely impersonal "processes".