Slashdot Mirror
Overconfidence: Why You Suck At Making Development Time Estimates
Dan Milstein from Hut 8 Labs has written a lengthy post about why software developers often struggle to estimate the time required to implement their projects. Drawing on lessons from a book called Thinking Fast and Slow by Dan Kahneman, he explains how overconfidence frequently leads to underestimations of a project's complexity. Unfortunately, the nature of overconfidence makes it tough to compensate. Quoting:
"Specifically, in many, many situations, the following three things hold true: 1- 'Expert' predictions about some future event are so completely unreliable as to be basically meaningless 2- Nonetheless, the experts in question are extremely confident about the accuracy of their predictions. 3- And, best of all: absolutely nothing seems to be able to diminish the confidence that experts feel. The last one is truly remarkable: even if experts try to honestly face evidence of their own past failures, even if they deeply understand this flaw in human cognition they will still feel a deep sense of confidence in the accuracy of their predictions. As Kahneman explains it, after telling an amazing story about his own failing on this front: 'The confidence you will experience in your future judgments will not be diminished by what you just read, even if you believe every word.'"
'nuff said.
We've all been under pressure to give our "best" estimates and then some.
Give a realistic estimate? Off to India!
Points 2 and 3 don't seem to apply to me. I know I suck at doing development estimates. When asked for one, I've never been particularly confident about any estimate I give having a good chance of being accurate. I want to estimate conservatively, but project schedules don't allow for that.
File under 'M' for 'Manic ranting'
Predictions on the time it takes for me to do something can be off, but not by much. Most good predictions have contingency plans, etc...
In my experience, the biggest variability in estimates is the reliance on external dependencies. If I were the only person needed to work on something and I estimated 40 hrs of work, I would probably get it done in 30-45 hrs. But when that works requires someone else to do something at a critical point, even if it only takes 1 hr, the ability to acquire that resource in a timely manner ALWAYS messes with the time. Instead, the 30-45 hrs turns into 40-60 hrs. Amazingly, the "wait time" makes my time spent worse as well. I have to go through "ramp up" time again.
You can even schedule out that you will have the person for 1 hr a whole week ahead of time. But I have found it rare that you are able to acquire that resource remotely close to the time you scheduled.
Always tripple all estimates. That way you always look like a miracle worker.
If I were God, wouldn't I protect my churches from acts of me?
It took me a few years for me to discipline myself to including testing and bug fixes in any estimate I made to managers. When ever I would say, "I'll finish coding by X," they would always assume that it would be in release condition by then.
The Moore-Murphy Law: The number of things that will go wrong will double every 2 years.
I often find another problem is management's refusal to accept the estimate of the developer. I am usually pretty good at estimation. Here is what usually transpires for me:
Manager: "How long will it take you?"
Developer: "2 months."
Manager: "You don't have 2 months. You only have 1 month. Redo your estimate. How long now?"
Developer: "2 months."
Manager: "You don't have 2 months. You only have 1 month. Redo your estimate. How long now?"
At this point I feel like saying:
Developer: "Why are you asking for my input? Just write down 1 month. And do you want me to tell you I will be 1 month late right now or in 1 month from now?"
Than a competitor will say it will take two hours and get the job. Ok, finally it will take four hours, but still, he got the job.
Blah, blah, blah. Bad estimates.
Blah, blah, blah. Oh noes! Waterfall!
Blah, blah, blah. Fixed by Agile!
I think they finally blocked the APK posts with a HOSTS file.
La Forge: The Captain wants this spectrographic analysis done by 1300 hours.
Scotty: Starfleet captains are like children. They want everything right now and they want it their way.
But the secret is to give them only what they need, not what they want.
La Forge: Yeah, well, I told the Captain I'd have this analysis done in an hour.
Scotty: How long will it really take?
La Forge: An hour!
Scotty: Oh, you didn't tell him how long it would *really* take, did ya?
La Forge: Well, of course I did.
Scotty: Oh, laddie. You've got a lot to learn if you want people to think of you as a miracle worker.
- TNG 6x04
I hate making estimates. I'm always, ALWAYS wrong. I always know I'm GOING to be wrong.
I've been trying to fix this for 12 years. I thought it was just inexperience talking, but I'm a grown-up programmer now. 'Senior', by some estimates. And yet I still have a hard time estimating the time of getting things up and running. I write one thing, and four things that I couldn't have anticipated crop up. This is particularly true in my industry (video games) where you're often working with an engine that's a few years old, and other people are in the middle of working on it, and specs are changing under everyone all the time. Things that look straightforward end up taking bad detours through networking components that nobody else understands because that part was written years ago and those programmers aren't around anymore.
Man, this story makes me feel a lot better about myself.
Hofstadter's Law:
It always takes longer than you expect, even when you take into account Hofstadter's Law.
People estimate based on how much time they think it -should- take, but you almost never estimate:
1. External factors which grow time
2. Feature/function clarification takes time
3. Outside resource turnaround takes time
4. QA may never be satisfied
5. We're moral and WE make a lot of mistakes along the way
6. Most likely, you don't know all the caveats of developing the piece of work until AFTER the development is over
7. General personal time spent elsewhere (meetings, consulting with co-workers)
Sadly, the best estimate for completion ends up being 1.5-2x longer than my original gut check, so as long as you pad out your estimates, you should be fine.
Bye!
The so called 'experts' are just as much experts at estimating requirements and timing as they are 'expert architects'.
Here is a thread where I argue that J2EE is a crutch given to people labelled as 'architects', turning them into typists while removing any real architectural thought from their activities. If you read through the thread you'll see some AC objecting to that notion and he does not realise that he is arguing my points there when he talks about architects.
He is mistaking what 'enterprise' means, he believes it has to do with some technology, with some instrument, a tool or a set of tools. He does not realise that 'enterprise' really means an approach, a process, set of processes and standards that a company forces itself to adhere to, be it in implementation details or documentation process (all of which are important of-course), but enterprise does not mean just some solution provided by some vendor even if it has the word "enterprise" in its name!
So with that in mind, realise that what we actually have for architects are most of the time not architects at all, they are copy pastors, they are typists, they are managers probably, but they are not actually designers.
Those are the same people that would be considered 'experts', who managers turn to for time estimates. I don't remember myself underestimating projects at all or overestimating by more than a factor of 2, because I have the entire process of what it takes to build a project in mind and I break it down into all the little pieces, put a number that comes from past experience in front of that little piece, then the numbers are added up and there is some adjustment based on the team, the people that are going to be working on this.
AFAIC overestimating is not as a big problem as underestimating but if you are bidding on a job, then it does present a challenge. In case of bidding you are actually not truly estimating a project, you are just trying to get it before the other guys get it, and I think that's where the real problem comes in. Managing client expectations is a serious matter, you better be able to do that and I think the more you way overestimate or way underestimate the less likely the clients are to trust you in the future, so be true to yourself.
But again, how can somebody be true to themselves, when they don't even understand themselves what it is they are doing in the first place?
You can't handle the truth.
Back when Joel spent time on writing, Joel Spolsky of Joel on software had an interesting method for doing time estimates. His point was to go into a deep level of detail. Instead of handwavy "code the GUI" the only way to really get anything remotely close to a real time is to estimate everything down to at least half day, if not lower granularity. It's not the "oh you feel the time better" as much as to think of EVERYthing you need. If you go to a lower level, you may remember that dialog box that you didn't think of at the 25,000 foot level.
It would be interesting to see if anyone ever used it to improve their estimates. Even he "disavows" it now, preferring the method in his software tool. But I like the "the world is a big place, are you sure you're thinking of everything" that the older method pushed you to.
Something my boss has us do when we estimating projects. She has a certainty factor that we set for each task, simple terms, which equate to a percentage in her calculations. The higher our certainty, the less risk that the task is underestimated. The lower the certainty, the larger the margin that the estimate needs to be factored.
Makes a huge difference in ballparking our estimates.
As per my blog post a couple of years ago at http://use-cases.org/2011/06/04/getting-good-estimates/ [use-cases.org] and http://use-cases.org/2011/06/22/updates-on-getting-good-estimates/ [use-cases.org]
Most good estimates have a range - and not a number, or a number with a confidence (both are interchangeable).
If an engineer says it will take two weeks - I push for a range or a confidence. If the range is weird (2-8 weeks), I push for the engineer to tighten their estimate through discussing or raising and discovering the unknowns or the risks that they are aware off. That sort of estimate would usually end up around 3-5 weeks which is a reasonable margin - and a lot better than than underestimating by 50%.
Same with estimates that are too narrow. "2 weeks +/- day". That implies a full level of understanding, no risk and no dependencies. Almost never happens. Work through the same risks/unknowns and the estimates usually become really bad - typically at least double of the "high confidence" estimate - similar to TFA.
There is a lot of reasonably applicable theory behind this (confidence intervals, cone of uncertainty, etc). But we don't necessary focus on mastery of our art...
3 days for bitching, pissing and moaning.
3 days for dicking around on the interwebz
1 day lunch overages.
2 days for "zoning out"
3 days for witty banter.
This sig is not paradoxical or ironic.
I usually take whatever estimate I'm given and change it to the next largest unit and double it. Thus, an estimate of two hours become four days. This is still usually less than the actual time required. And don't even get me started on projecting when some task will be completed as opposed to how much effort will be required. The above alogorithm does a reasonable job at estimating effort actually requied but determining the calendar completion date is a whole different animal.
Cheers,
Dave
They that can give up essential liberty to obtain a little temporary safety deserve neither safety nor liberty.
Ben
When my customer comes to me and asks me to provide an estimate for a job, if I give them a conservative estimate, some of them may think that I am milking them with the extra hours. Specially if they get a competing estimate from an overly aggressive Indian company who is eager to sign the contract but has no clue on how to deliver.
I usually do not fret too much about customer feelings in a case like this. But during slow times I have little choice. Bottom line is, most of us would love to provide conservative estimates, but often times it is not as simple as that.
... to be developed for them in 3 months. I estimated 10 months. So they decided to look around for another developer. A couple years later they came back and asked if I could do it in 6 months. I told them it would take 12 months, now.
now we need to go OSS in diesel cars
My estimates suck because:
Project leader: Ok, so we need to know how long it will take for you to do X
Me: I'm not sure, that's an entirely new API, proprietary to the vendor, there's almost no documentation and their website has a support forum filled with questions and basically no replys to any of them.
Project leader: Well, we need a number.
Me: Why?
Project leader: I have to fill in this box here... see?
Me: Ok fine, 800 hrs
Project leader: Now hold on a minute, this wont take 800hrs
Me: It could, I have no idea. It's already taken the majority of at least one hour and I don't even know what language it's in.
Project leader: Fine, I'll put down 800hrs, but you're the one that's going to look silly.
POST PROJECT REVIEW ....
Project leader: I can see here your original estimate was 800hrs, and your actual billed time was 1265hrs. What causes led to you missing your estimate, and how can we avoid those in the future.
Me: Don't make estimates.
Project leader: Come on now, I need a real answer.
Me: Why?
Project leader: I have to fill in this box here... see?
Me:
I agree. Fortunately for me at least, I happen to be in the happy world where management supports us in realistic timelines and realistic scoping.
Spanning almost seven years now and well over a hundred assorted projects we have been overdue on projects two times total. One of those was during the exceptional case of a co-worker getting in a car accident and breaking 13 ribs, the other was an exceptional case where very serious external forces caused the design to shift mid-development. In no case has it been due to poor estimation.
We have come to learn the development cycle for our small teams:
When I hear about other groups hitting 60% or later in their development cycles and still not getting feature complete, I pity them. They have made the mistakes the original article warned about, and were probably driven to that madness by the poor management you mention.
//TODO: Think of witty sig statement
Actually, most of those things can be predicted. What is harder to predict is the creative aspect of development. Predicting a civil engineering project, like a bridge, is easy because engineers can compute the number of beams, the volume of concrete, the depths of the footings, etc, and they already have a good idea how construction people will behave, so they can add 5% for vacations, 20% for staffing difficulties, 10% for late trucks, etc. Predicting the creation of a new piece of software is less certain, because so many of those pieces are unknown. You make an initial survey of the requirements, and take an educated guess at what the solution might be, but you know that's never the final picture of the real solution.
If you're on a mature Agile team, you trot out your velocity, map your epics to some t-shirt sizes, and do some simplistic multiplication. But you also know to announce the estimates in terms of your team's delivery dates, and you don't overpromise. At this point, either management trusts your team's reputation and you boldly go forward, or you give them estimates with confidence levels around 20%, because you simply can't be more accurate than that.
If you're on a waterfall team, any software development estimate with an accuracy figure of higher than about 50% should be viewed with suspicion, and anyone claiming 90% or higher should be flagged as a true bullshit artist.
John
I would love to see an experiment. Take two groups and give them the same job. Group one would be based on a typical American corporate structure with a Boss, Scheduler, budget person, middle management, supervisor, and finally people doing the work.
The other group would have the same number of people but only those that work. No schedule or budget just work until it's done. I wonder what the results would be?
I love Jesus, except for his foreign policy.
Predicting a civil engineering project, like a bridge, is easy.....
I'm going to stop you there, because civil engineering projects are NOTORIOUS for going over budget. You might have heard of projects like the big dig. Less well know, is that going over budget in less spectacular ways is apparently a fairly common occurrence. I was looking around for a report to link for you that I read awhile back talking about why civil construction projects so frequently go over budget, but alas, I cannot locate it.
Alas!
HA! I just wasted some of your bandwidth with a frivolous sig!
Exactly. If someone asks me for an impossible prediction, I will give them what they asked for with unyielding confidence. When faced with the inaccuracy of my prediction, I will continue, with confidence, in giving equally inaccurate predictions in the future.
My real algorithm is as follows:
Is it fun/interesting to do? If yes, feel the room and give an estimate that will keep the project from being killed. Else, give a long enough estimate that can withstand cross examination that hopefully will kill the project. Regardless of what I answer, management will cross examine my estimate using their own equally inaccurate measurements and assessments, if I deliver with anything less than absolute confidence I will be smashed. You see, the bullshit is layered upon the bullshit, then convoluted by management bullshit, into spectral bullshit that makes for a great power-point.
If you want an accurate assessment of how long something that has never been done will take, you're asking for the impossible. If you want an accurate assessment of how long something that has been done before many, many times will take, either a) you're not in technology in the US, we don't do "competition" here, or b) I'll tell you 75 years because I really don't want to do that anyway.
If you ask any experienced software developer about estimating when the project will be finally completed you will get a blank stare --- for the simple reason that there are always higher mountain to climb, more features to add, more bugs to be squashed, more optimizations to be made, and so on ...
I do not do time estimation --- I do the reverse
I set out a limit on time before I even begin a project
Within that time span I partitioned it into "exploration", "research", "coding", "debugging", "finishing touch" --- and I can terminate the entire project when any part of the partition takes too long, or produce too few result, or both
That's the way I've been using since the late 1970's --- it might not be the best way, but that's my way of accomplishing my projects --- or abandon it before it dragged out way too long
Muchas Gracias, Señor Edward Snowden !
Manager: "We need an application that does X,Y, and Z. When can you have it done?"
Developer: "Well, can you tell me more?"
Manager: "No, time I have a manager's meeting in 5 minutes. Just give a pall park."
Developer:" Ok, umm 3 weeks."
Manager: "THAT LONG?"
Developer: "OK, 2 weeks? Maybe less?"
Manager: "OK"
Later, in the manager's meeting.
Manager:"My developer says he can get it done in less than 5 days."
When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
management/sales: how fast can we get this done?
dev: low 3 weeks, mid 5 weeks, high 9 weeks
management/sales:Great! I was hoping you would say around 2 weeks, because this product is being launched next week, so if we push it, we should be able to get it out the door by tomorrow approval!
5 days later: But you promised!!! Now I'm on the hook for a demo to the VP of International Sales and Marketing!!!
Powerpoint! Those bastards don't know the difference. Just show some slides...
Construction has very real material costs -the beams and concrete you talk about - so every component is drawn and specified. In software development the material costs are virtually zero so you might as well build it twice, once to understand it and once to productionize it, it's the same as writing a detailed spec and then coding it.
Art is the mathematics of emotion