Are There Limits to Software Estimation?

Charles Connell submitted this analysis on software estimation, a topic which keeps coming up because it affects so many many programmers. Read this post about J.P. Lewis's earlier piece as well, if you'd like more background information.

Comments

Unless it's a simple project...

[Nijika]

There are always things you won't consider until something's being developed. If you've done something a thousand times, and have the libraries developed then you can probably estimate the time required very accurately. If the request is something completely new to your team, you'll never be able to accurately estimate the time required until analisys (which takes it's own time as well).

Re:Unless it's a simple project...

[dubl-u]

Yep! The "first pancake" approach is a good one. And, as other posters have noted, a venerable one. Why? as you point out, you discover a lot as you go, both about the user needs and about how to do them well.

But you can go further with this! Any short-cycle iterative model will give you this sort of feedback every couple of weeks, rather than at the end of every version.

For more info on it, take a look at Rapid Development's discussion of evolutionary delivery. Or try a process that's built around feedback, like Coad's Feature Driven Development or, my personal favorite, Beck's Extreme Programming (XP) (dumb name, but a great process).

2x+7

[Lizard_King]

In a software engineering class in college, I remember a professor joking around that the catch-all equation for software estimation is 2x+7, where x (can be in any units like hours, days, weeks, minutes) is your estimate for how long you think the component will take. So for example, If one of your developers estimates that developing some component will take 4 hours (so x = 4), in *reality* it will take them 2x+7 = 15 hours to complete.

After gaining a few years of "real world" experience in software engineering (and I know that the very term real world experience is debatable :-), I'm realizing that this professor wasn't that crazy, and his crude estimation mechanism (which is a joke) isn't any more or any less accurate than a lot of modern techniques I have seen people use in the field.

Accuracy, and doubling a double

[CDWert]

I have been in this industry for what often times seems too long, My father was in from the beggining 1962, When I was younger and he asked me how long I thought it would take to write I blurted out my answer and he said no X , I said noooo thats way too long how did you arrive at that ?

Here was his answer I have ALWAYS found it accuraye to +/- 10% so far on hundreds on small to massive projects.

1. Once you know all , or most of the forseeable estimates take that number. say 10 hours.

This number is an instinctual reaction to a perfect enviroment , a little experience, some ego on your part of what might be accomplishable in a vacum.

2 Take that Number ad double it.
This takes into account all the real world distractions. Events, etc.

3.Take that number and double it again. This takes into account unssen variables and events beond mortal control.

40 Hours.........

I use this on EVERY single estimate I provide, WHY ?? It works, its not too high not too low, just right.

I tell people this and they laugh, then I tell them that there are MANY legacy applications SSI, IRS, FBI, you name it that were qutoed by my father in this EXACT manner.

There is NO practical limit to estimation, As long as you have the information neccesary to determine what the job youre actually doing is.

it develops over time

[foo(foo(foo(bar)))]

I work on a very large software project. 6million+ lines of active source code, with 400,000 new development hours per year and growing. -and- we are on our extimates well over 80% of the time. (if we don't hit it, we are under).

How is this possible? It has evolved over time, some of the same people who started this project 9 years ago are still here and they know the system very well. That knowledge, combined with good project management leads us to several categories. During a requirements phase, designers assign a complexity to the changes for a module, and based on the type an hours extimate is generated.

Now, Lewis is right, no algorithm can be developed to figure out the compleixty, but a human can, and the computer can figure out how many hours should be devoted to documentation, coding, and testing.

My overall point, as a software product matures...esitmates are easy to estimate and project dates are easier to meet. But you already knew that...

Programmers intuitively know this

[Anonymous Coward]

Programmers that I've worked with have almost always intuitively known this to be true, and non-programmers (in particular, product managers responsible for scheduling) have almost never understood this. Hence the frusteration on both sides.

I'm glad there's finally a resource to help the folks who insist on accurate estimates understand why my response to the inevitable inane question is always a cynical "two weeks", regardless of the complexity of the problem.

This is the wrong way round.

[johnburton]

In real life it's rare to be asked for an estimate of the time required.

What usually happens is you get told roughly what to build and the final date by which it needs to be ready. There then takes place a series of negotiations and compremises on the scope of work until everyone is "happy".

I suppose that doesn't really invalidate the point of the article at all, it's just an observation for those who think that estimation is the nice science that it is sometimes presented as being.

My critique

[The Pim]

Very nicely stated! I was going to publish my own response to "Large Limits", but I honestly decided that the paper was too "academic" (in the sense of, "interesting but irrelevant to the practical world") to be worth critiquing. But this is slashdot, and what better place for worthless thoughts, so here goes ...

The glaring flaw of the paper is that the main argument can be applied equally to any human endeavor, not just to programming. The argument is essetially a rigorous version of the statement, "You can't (in general) know how hard (complex) it's going to be, until you do it". The author supports this by pointing out that the purpose of any program is equivalent to generating a string that is a complete, precise description of the problem. Complexity theory tells us you can't predict the length of that program (without a formal system bigger than the program).

But it's not hard to cast any problem into this form. Take baking a cake. The problem can be thought of as generating a precise description of how to turn some inputs into an output within the range of what we consider a cake. In a reductionist sense, that process is incredibly complex (much more than any computer program), involving gazillions of elementary parcticles and their interactions. But nonetheless it's pretty easy to estimate how long it will take to bake a cake.

Complexity theory shows us that complexity is indeed pervasive in general; but everyday experience shows us that it is usually encapsulated within simple abstractions. Most things we plan and do have relatively simple descriptions in terms of objects with those properties we are familiar, and things we have done countless times before. So while estimating complexity may not be possible in general, it is usually not very hard for the things we care about.

In order for the paper to be persuasive, Lewis must show that computer programming is, in practice, more complex than most other activities--that new problems can't be easy stated in terms of already solved problems. (He does begin to address this, but only as a side-note.) I think most practitioners would essentially agree (and I'm not going to argue this, unless someone challenges it). What does this mean for the relevance of complexity theory? It's a deep and difficult question, but I suspect that some insights can be drawn. In particular, I do believe that there are problems that can't be estimated without effectively solving them.

Regardless, there are more obvious, intuitive reasons that complex activities are difficult to estimate. One is that that humans vary wildly in their efficiency at complex tasks. We all know the experience of cracking nut after nut one day, and being stumped the next. Sometimes, to be sure, this is due to misestimation of difficulty, but just as surely it is often purely psychological. Another is that teams working on complex problems are prone to miscommunication and other group disfunctions. A third is simply that the flesh is weak--we often lack the discipline and concentation to plan our projects in sufficient detail.

And this list only considers the difficulties that derive from complexity. Software development faces a host of additional "accidental" challenges, such as bugs in third-party software, clients (and marketers) that change their minds, changing fashions in tools and methodologies, etc. In short, you don't need a fancy theory to conclude that predicting development time is quite hard!

Art VS Engineering

[Srin Tuar]

Programmers that I've worked with have almost always intuitively known this to be true, and non-programmers (in particular, product managers responsible for scheduling) have almost never understood this.

Those in the "Programming is an Art" camp tend to agree that there is no real way to estimate how long doing something new is going to take.

Those who think of programming as simply bulk engineering, repetetive, boring, or just "coding" tend to be frustrated by this seeming fact. It is almost irreconcilable with normal business practices to know how long a job will take until it is actually done. This makes it extremely difficult to make close-ended contracts, and to predict budgets.

Asking how long a particular software job will take is often equivalent to asking how long a research job will take.
Im sure the scientists would be amused if a suit walked down into R&D and asked them when they would be "done" ;)