Overconfidence: Why You Suck At Making Development Time Estimates

Dan Milstein from Hut 8 Labs has written a lengthy post about why software developers often struggle to estimate the time required to implement their projects. Drawing on lessons from a book called Thinking Fast and Slow by Dan Kahneman, he explains how overconfidence frequently leads to underestimations of a project's complexity. Unfortunately, the nature of overconfidence makes it tough to compensate. Quoting: "Specifically, in many, many situations, the following three things hold true: 1- 'Expert' predictions about some future event are so completely unreliable as to be basically meaningless 2- Nonetheless, the experts in question are extremely confident about the accuracy of their predictions. 3- And, best of all: absolutely nothing seems to be able to diminish the confidence that experts feel. The last one is truly remarkable: even if experts try to honestly face evidence of their own past failures, even if they deeply understand this flaw in human cognition they will still feel a deep sense of confidence in the accuracy of their predictions. As Kahneman explains it, after telling an amazing story about his own failing on this front: 'The confidence you will experience in your future judgments will not be diminished by what you just read, even if you believe every word.'"

Comments

Scotty knows

[u64]

La Forge: The Captain wants this spectrographic analysis done by 1300 hours.
Scotty: Starfleet captains are like children. They want everything right now and they want it their way.
But the secret is to give them only what they need, not what they want.
La Forge: Yeah, well, I told the Captain I'd have this analysis done in an hour.
Scotty: How long will it really take?
La Forge: An hour!
Scotty: Oh, you didn't tell him how long it would *really* take, did ya?
La Forge: Well, of course I did.
Scotty: Oh, laddie. You've got a lot to learn if you want people to think of you as a miracle worker.

- TNG 6x04

Re:Is that really the problem?

[Platinumrat]

I'm constantly getting this effect at work now. My current manager (who has no technology background or experience) is always challenging my 25+ years experience. I've already felt the pain of optimistic estimates and now include everything, requirements, documentation, design, code, integration, test, more documentation, installation, commissioning and support in an estimate.

He comes out with the following gems:

- "I believe your estimates are too high"

- "I've already committed to a delivery schedule with the CEO and Engineering Manager"

- "Well, we'll just have to challenge your assumption"

- "We'll just have to find ways to work smarter"

- "We'll just need to work extra hours then"

- "You're not showing enough committment", when asked to work on the weekend and holidays. This despite being with the same company for my entire working life

It's like I'm in a Dilbert nightmare now.