The Programmers Who Want To Get Rid of Software Estimates

An anonymous reader writes: This article has a look inside the #NoEstimates movement, which wants to rid the software world of time estimates for projects. Programmers argue that estimates are wrong too often and a waste of time. Other stakeholders believe they need those estimates to plan and to keep programmers accountable. Is there a middle ground? Quoting: "Software project estimates are too often wrong, and the more time we throw at making them, the more we steal from the real work of building software. Also: Managers have a habit of treating developers' back-of-the-envelope estimates as contractual deadlines, then freaking out when they're missed. And wait, there's more: Developers, terrified by that prospect, put more and more energy into obsessive trips down estimation rabbit-holes. Estimation becomes a form of "yak-shaving" — a ritual enacted to put off actual work."

Comments

Simple methodology

[Kjella]

Good managers get my best guess.
Bad managers get my worst case.
Horrible managers get my resignation.

Tag: WORKS4ME

Re:Simple methodology

[Penguinisto]

One would hope that a good manager would have enough practical and direct experience in writing software to at least come up with a half-decent estimate, no?

Most shops I've seen lately have the scrum masters spend a part of a planning session simply asking individual contributors "Here's a rough outline of the proposed project [...] now how long do you think doing that will take", and they come up with an estimate adjustment from there... most of the time, it's fairly close. PMs pad things a little of course, but the results tend to be fairly close.

YMMV of course... depends on who is posting the final estimates - is it devs, or is it the MBAs.

(If it's the latter, run like hell.)

Re:Simple methodology

[jellomizer]

Well a good manager, will work with the developer and give them clear deliverables, adequate milestone achievements even if these milestones may not look like much. Then we can give a good estimate.

A Bad manager will toss out a pie in the sky grand vision that does something. So the programmer will need to go back and rework idea over idea, going back and improving on the design.... So this will take a worse case scenario.

Re:Simple methodology

[lgw]

You can't really give a good estimate up front on most projects, because even if the requirements are well defined (and they rarely are) they will change.

That's all OK. That's the whole point of Agile, if you ignore the Agile BS that consultants sell. You can quickly do coarse-grained task breakdowns (say, to 2-week tasks) to get a reasonable estimate of the work as you currently understand it: that estimate is generally within a factor of 2, if the requirements don't change, so you can at least see if you have 1/3rd the staff you need or something way off.

Then you measure real progress against that first-take estimate. Usually by about 6 weeks in on a team-sized project, you'll have the real multiplier. "We thought the first set of tasks were 20 programmer-weeks, but really as we understood it it was 30", and multiplying that initial "6 months" estimate by that measured 1.5 multiplier is a damn good estimation method. And it only gets better as work goes on. I can reliably give fairly accurate estimates from project completion now, on Agile projects, by the time we're about 1/3rd of the way done (and that's so much better than some teams are used to that it seems like magic).

Selling that all properly to management is a different thing, but that's a whole job skill that any senior engineer just needs to have. Just like showing management the cost of each requirements change, as well as the skill to minimize that cost of change (which is the real point of Agile- the rest is hype).

Re:Simple methodology

[Anonymous Coward]

Yeah. Estimating the time to develop a simple web page is one thing. Estimating the time to design and engineer a new enterprise system is another. It seems that there are "managers" who think the first can be translated into the second. I'm sure Cheops asked his engineers to specify how long it would take to build his pyramid... The answer? "It depends!". Depends upon how many workers we can get, how far we have to go to get the stones, what the weather will be like...
 

Re:Simple methodology

[knightghost]

A well defined project can be estimated. Change Orders estimating needs to be done properly and BILLED - the cost of re-analyzing is a real cost and the business needs to see it. Once you get that across the number of change requests decreases dramatically.

Software engineering is Engineering... some of the costs are inverted, but otherwise it's the same project management as other engineering projects.

Re:Simple methodology

[sjames]

Then something changes and blows the estimates out of the water. But the MBA's think since only one line in the spec changed, the schedule should stay the same.

Re:Simple methodology

[Gr8Apes]

When I have one of those... I just say - well, if it should stay the same, why don't we just change that one line back. There's no difference according to you.

Re:Simple methodology

[Gr8Apes]

Agile is pretty much crap. You don't build a bridge that way, nor any other project, except maybe landscaping. If you have enough to sketch out a plan, you have enough to plan. You can't build something if you don't know what your target is.

Re:Simple methodology

[jlowery]

Got asked about Agile planning estimates (you know: the Fibonacci scale) by our CEO, and why we didn't use time estimates. My answer was the developers a better at estimating complexity than time to completion. And complexity estimation accounts for the fuzzier initial understanding of harder problems. When you start measuring velocity, it stabilizes remarkably well and can be predictive.

But complexity estimation is not a time estimate. If, for my team, a 1 is about 2 hours work, varying between .5 and 4 hours, then a 5 (5x as hard) is going to take between 2.5 and 20 hours. That's a big variance! But it more accurately reflects the uncertainty of the estimate.

Re:Simple methodology

[RabidReindeer]

Then something changes and blows the estimates out of the water. But the MBA's think since only one line in the spec changed, the schedule should stay the same.

More often in my experience the managers and the users refuse to believe the estimates and force new "more realistic" estimates to be submitted.

Because "It's Simple! All You Have To Do Is..."

Re:Simple methodology

[Zero__Kelvin]

"One would hope that a good manager would have enough practical and direct experience in writing software to at least come up with a half-decent estimate, no?"

No. That is the whole point, which you have seemed to miss. I'm the software engineer and even I can't come up with a reasonable estimate; why the hell would some manager several layers of indirection distant from the design be better at it?

Re:Simple methodology

[DarkOx]

That should not happen as often as it does though. Part of being a "professional" where it comes to software architecture is anticipating reasonable future needs and planning for them.

If a one-line spec change blows the estimates out of the water many times that probably indicates major rework had to happen. It should not be that way most of the time. If it is the development team did a poor job of planing a head, likely, not always if someone changes "suitable for car wash automation" to "suitable for nuclear reactor automation" fine, you can toss the old estimate out the window entirely and none of the fault is your own.

Re:Simple methodology

[AqD]

A well defined project can be estimated.

Well-defined by design not by functionality or goal. It's pointless when you can't be sure whether your design is seriously flawed.

Which happens to us all the time because we do new types of projects every 2 years or so. My current project built on World Wind Java ended up having the core parts of WWJ re-analyzed, abandoned/replaced and partially rewritten due to very terrible rendering performance on modern hardware, which cannot be seen in prototype with small amounts of objects, plus lots of time tracking and fixing bugs inside Eclipse RCP and JavaFX.

Can it be avoided? yes, by studying the source code of all libraries used in every new projects. That would probably cost us more than the entire project itself.

Re:Simple methodology

[Anonymous Coward]

Exactly - that is why there was been a growing recognition that Agile isn't getting the job done. At my last company a bunch of Agile zealots came stampeding my way trying to say that we had to change all our development processes to be "agile" because, well, just because. So I went through a simple requirements exercise with them. The first and foremost requirement is that when sales/marketing sales that they want 100 features in the product and there is no way that we can doo all 100 features, we have to be able to prioritize them. One of the primary factors in prioritization is looking at the effort/reward ratio. Something that is little effort but has a big payoff is high priority. Something that is big effort with little payoff is low priority. Now, please tell me how Agile is going to tell me how much effort is going to be required to complete a particular project when the whole Agile approach is to start coding and figure it out as we go?

silence

OK, here is another requirement. We need to put together timelines because Engineering doesn't operate in a vacuum. We are part of a company with many parts all of which have to be coordinated. Sales needs to be able to set expectations with customers. Marketing needs to make plans for when and how to introduce features to the market place and make forward looking statements on upcoming features. Support needs to plan out how it is going to support upcoming releases, which requires logistics in terms of when to apply which resources to which tasks. So how does Engineering give people these timelines if we don't take the time up front to plan out what we are doing prior to banging on our keyboards?

silence

The big problem with Agile is that it was created by Engineers in a vacuum who failed to recognize their responsibilities to the rest of the Company. Couple that with an increasing number of studies that are showing that software quality is lower now and that it seems that Agile is part of the reason (basically, the idea that somehow the coordination of effort required to make a large project come together would magically happen "organically" is being increasingly debunked) and, well, clearly it is time to re-evaluate how Engineering needs to operate within a Company.

Re:Simple methodology

[blue9steel]

Of course they also don't usually decide that they'd rather have a skyscraper instead of a bridge when you're mid-project.

Re:Simple methodology

[BronsCon]

Depending on how far along in the project you are, it's reasonable to expect changing a single word in the spec to have a major impact on the project. Consider:

Paint my car black.

So you do all the prep work, car's primed with a dark primer, black paint is mixed and ready to go, then this change request comes in:

Paint my car white.

Well, now you've wasted the paint you just tinted black, and you can't paint white on top of dark primer (well, you can, but you need many more coats), so you've got to redo the prep work. That means waiting while the primer fully cures, so you can sand it off properly; otherwise, it'll gum your sandpaper. then re-prime, then you can paint. Assuming you don't see

Paint my car forest green.

in the meantime.

That was one word. Yes, one line matters.

Re:Simple methodology

[Digicrat]

And then there's Scotty's Miracle-Worker philosophy:

Kirk: How much refit time before we can take her out again?
Scotty: Eight weeks, sir. But ye don't have eight weeks, so I'll do it for ye in two.
Kirk: Mr. Scott. Have you always multiplied your repair estimates by a factor of four?
Scotty: Certainly, sir. How else can I keep my reputation as a miracle worker?
Kirk: [over the intercom] Your reputation is secure, Scotty.

Lt. Commander Geordi La Forge: Look, Mr. Scott, I'd love to explain everything to you, but the Captain wants this spectrographic analysis done by 1300 hours.
[La Forge goes back to work; Scotty follows slowly]
Scotty: Do you mind a little advice? Starfleet captains are like children. They want everything right now and they want it their way. But the secret is to give them only what they need, not what they want.
Lt. Commander Geordi La Forge: Yeah, well, I told the Captain I'd have this analysis done in an hour.
Scotty: How long will it really take?
Lt. Commander Geordi La Forge: An hour!
Scotty: Oh, you didn't tell him how long it would *really* take, did ya?
Lt. Commander Geordi La Forge: Well, of course I did.
Scotty: Oh, laddie. You've got a lot to learn if you want people to think of you as a miracle worker.

Re:Simple methodology

[Maxo-Texas]

I had no particular problem with estimates.

At a minimum, you could break out easy "construction/recognized pattern" work from risky new stuff.

As far as managing programmers... it was humorous.

Few liked giving estimates. So they would say it couldn't be estimated.

So I would ask, will this project take 2 years... and they would say, "oh no- of course not" and after a bit, we'd get down to 6 months or 6 weeks or 6 hours...

So then we'd time box it to what could be done in a month and move any risky items up to the front so we could establish if a new technology wasn't going to work before we put a lot of work into the project.

Then, I recorded over/under for every project and found (over about 24 programmer data set) that programmers consistently overshot or undershot their estimates. So after a few projects, I had a pretty good idea of their deliverables.

Finally status reports and status meetings with function points and overall percentage delivered kept things on schedule or let us know well ahead of time there was a problem with the estimate/schedule.

Programmers were not my problem- executives were.

They...
a) pushed us to violate standards.
b) ordered overtime without ordering it. As in assign 80 hours work and then insist it be completed when everyone knew it couldn't be completed. Made worse by the fact the indian contractors said "I'll do my best" for "no- you are batshit crazy" and then things fell apart when the indians were unable to deliver. Of course, the indians were very good at delivering to the (crazy/incomplete) specifications on time. At least enough to be testable. I'm not sure if it is because they were contractor types or that they were indian- perhaps a bit of both. I learned in a contracting shop, you always say you can meet the estimate (to get assigned the work) instead of giving a realistic estimate. Then renegotiated it later when it wasn't going to make the schedule. If you didn't, then the three other people bidding on the work would get the work. Executives seemed to have zero memory for the fact that you delivered on time on estimate while the other people were usually late.
c) made everything priority 1a. they had no ability to prioritize as far as I could see. Which really just pushed prioritization down to me or the programmers.
d) cancelled projects without warning.

Re:Simple methodology

[sjames]

First, keep in mind, to the MBA, a line in the summary plus 20 supporting pages is "a line", at least when it benefits him to call it "one line".

Next, that one line can be HUGE sometimes. It can require whole new additional sub-susyems. Must interface with existing PCs -%gt; Must interface with existing PCs including the Univac in the basement. (and by interface with, they also mean translate between).

And, as BronsCon points out, sometimes one WORD is a problem depending on when it changes.

Dropping or adding the word 'not' can also make life interesting.

Re:Simple methodology

[ATMAvatar]

The compile step is building the bridge. The coding step is drawing the blueprints, which *does* involve iteration.

Re:Simple methodology

[Jane Q. Public]

Then something changes and blows the estimates out of the water. But the MBA's think since only one line in the spec changed, the schedule should stay the same.

And that's when you re-negotiate, or quit.

From my experience, it's easy to make bad estimates because bad estimates are easy to make. If it's a big project, take your worst possible guess, and multiply by 1.5.

One of the biggest reasons bad estimates are so easy to make, is that the stakeholders "forget" to include too many of the details. And even, sometimes, some of the big essential features. It's debatable how many of these things are really "forgotten", versus not known or just intentionally left out to lower the estimate.

Therefore, I go about it this way:

I write out clearly what the specs are. Often this is just a repeat of the stakeholder's own specs. I estimate each major "feature" in the specs.

My standard contract states that this is only an estimate, based on the written scope of work just described. If at any point it looks like the work will take more than 10% over the estimate, it is time to examine why and the terms must be re-negotiated.

If any significant changes or additions are made to the scope of work, the terms must be re-negotiated.

These stipulations make sure everybody is talking, and if anything goes over estimate, everybody knows why. It also makes sure that if they add more work, I get more money.

I have a third stipulation I put in every contract: single point of contact. The stakeholder(s) must appoint one person, and one person only, who is to give me instructions and discuss the specifications. Nobody else, even the CEO, is allowed to do that.

I insist on these. So far, I have not had anybody turn me down because they thought the terms were unreasonable. That last clause keeps me from getting caught between conflicting instructions from different people. Other than that one, these are SOP for many engineering contracts. I just borrowed from those.

Re:Simple methodology

[mrprogrammerman]

I think you can usually take your best guess and multiply 2.

Re:Simple methodology

[fyngyrz]

Because "It's Simple! All You Have To Do Is..."

Not just managers. Even working for yourself won't save you. Customers as well.

Re:Simple methodology

[Anonymous Coward]

Now, please tell me how Agile is going to tell me how much effort is going to be required to complete a particular project when the whole Agile approach is to start coding and figure it out as we go?

Sorry bro, but if you got silence in response to that question, those weren't "Agile zealots." They weren't even "Agile newbies." They were ignorant cunts.

Agile does NOT say "start coding and figure it out as we go." Agile says "Take some small subset of the functionality you've declared "the most important," estimate that, and deliver that."

Somebody has a list of 100 features? Great, estimate business value and level of effort - use story points, t-shirt sizes, etc. That alone gets you a pretty good way to prioritize your list - things that are "small-to-medium LOE, high value" go to the top of the list. Then you estimate those specific features, and start working on them - to start delivering business value as quickly as you can. Then as you go down your prioritized list, you continue refining and estimating the other features and pull them into your work queue as they reach the top of the "Effort x Most Valuable" ranking.

So how does Engineering give people these timelines if we don't take the time up front to plan out what we are doing prior to banging on our keyboards?

If you're a true Agile team, you're cross-functional, which means that proper representation from ALL of the organizations needed to deliver the product is present, embedded in your team. While devs are banging out designs and code, Testers are working on test plans and test automation given the specs that the devs produce, and Support people are building operations documentation and support matrices etc. - all in parallel, all coordinating with one another as the process goes. A team with proper representation from all functions SHOULD be able to provide necessary dates and timelines, because there is no "Dev team" just throwing dates over the wall to Support and Manufacturing.

Agile DOES answer these questions - if you're able to sit there and pretend that you've never heard the answers, then you're incredibly ignorant, or incredibly invested in somehow feeling like you've "discredited" Agile by sneering at it.

Re:Simple methodology

[unimacs]

Apparently the zealots didn't understand agile very well or they would have been able to answer your questions.

Agile doesn't mean there is no plan. Agile just acknowledges that the larger the set of requirements and the longer the timeline, the less likely it is that those requirements are going to be accurate (or won't change) and that the timeline will be met.

So rather than trying to establish all the requirements up front and then delivering when all those requirements have been completed, the idea is to focus on a minimal set of requirements in each cycle or iteration. Cycles are short, - a month at most. Prioritization is key.

Re:Simple methodology

[blackicye]

From my experience, it's easy to make bad estimates because bad estimates are easy to make. If it's a big project, take your worst possible guess, and multiply by 1.5.

I've found that this rule generally holds true for most projects which have complexity or labor involved.

In the case of construction and renovation, it doesn't even need to be that big of a job to exceed the worst case guesstimate multiplied by 1.5

Re:Simple methodology

[presidenteloco]

My methodology:

If someone gives me their estimate for a software project or task, I double it and add 30.

If someone asks me for an estimate for a software project or task, I rough it out, then double it and add 30.

It's really amazing how much stress that avoids (oh, and it also does a passable job of converting Celsius to Fahrenheit.)

Re:Simple methodology

[Registered Coward v2]

A well defined project can be estimated. Change Orders estimating needs to be done properly and BILLED - the cost of re-analyzing is a real cost and the business needs to see it. Once you get that across the number of change requests decreases dramatically.

Software engineering is Engineering... some of the costs are inverted, but otherwise it's the same project management as other engineering projects.

Correct. Our rule was "Give away the initial project because we'll retire on the change orders..."

Re:Simple methodology

[Kielistic]

Next, that one line can be HUGE sometimes

This system will be multilingual.

This system will properly respect all time zones.

Two very simple sentences that a lot of people think can be tacked on very easily but take a lot of work. Especially, as you said, if you are swapping a "not" out.

Re:Simple methodology

[Bengie]

In my limited experience, what people claim to be "agile" is "I don't want to have to design or plan". Agile is a method for programming, and has little to do with design.

Re:Simple methodology

[Outtascope]

Agile doesn't mean there is no plan.

To further that thought, agile isn't about not planning, it's about not being stuck with a crappy monolithic plan for an entire project. A plan that isn't crappy because of incompetence or negligence, but because the initial plan can't possibly account for the realities that every project encounters as it progresses (missed or changing requirements, technical failings of infrastructure that could not be predicted, change of business needs, etc).

Agile is intended to get you to stop trying to jam a square peg in a round hole. The alternative is to pound on that bitch 'till it's round. Which one is likely to result in a better engineered end product?

Re:Simple methodology

[Darinbob]

I've been programming for over thirty years, and I can't do estimates. At all.

Even when I think something is easy, it turns out to be hard if there's some snag I didn't foresee. If I think it's hard then I may find out it's not so bad after all because there are already tools to help put. Something that seems intellectually easy may take a long time because it involves a lot of tedious programming or detailed testing. Writing documentation takes me forever, I can not do that quickly at all unless someone wants a plain text file (which no one wants anymore). One big problem here is that I almost *never* program something that's been done a hundred times before, it's always something new, something that requires learning a new system, new hardware, something requiring reading through a lot of documents first, something with a huge amount of unknown variables, and so on. I also want to deliver something that works with minimal to no bug fixes required later, as opposed to shipping on time followed by a long sequence of updates.

Now certainly for some things I can say that I'll be done that night, because it's something really easy, like reverting a change, fixing up an obvious bug, and so on. But even then I occasionally hit a snag that's unpredicted.

The big problem is that the people who want the estimates don't understand this. And they come in all types. There's the person who thinks that a little bit of pressure makes people perform better (ya, you gave us someting to do in 3 months that needs 9 months). There's the person who thinks the estimates are accurate and golden and then sells the product before any work has begun. There's the person who thinks that your estimate means you will be working 100% of the time only on that single project and nothing else. There's the person who has a preconceived idea of how long it will take and considers that any longer estimate is "sandbagging". There are the people who will put me on two of the companies most important projects of all time, at the same time.

At one place, I gave an estimate of a simple task to be "two weeks after the main project finishes I will have the integration finished". So the sales rep wrote down a specific date based on that. The main project was late, but when that date arrived the sales guy comes up and asks me for my piece, which I had not started yet and could not start yet, then he panics because he's promised it to the customer (before even the product I am integrating with even exists). Ultimately I ended up with only two days to work on it.

I seriously wish that people would switch to a model where they announce a product after it is finished, and schedules are not based upon when the next conference or trade show is.

Re:Simple methodology

[Areyoukiddingme]

Then you measure real progress against that first-take estimate. Usually by about 6 weeks in on a team-sized project, you'll have the real multiplier.

Only if you're willing to accept sort-of-ok software. The 80/20 rule has not been repealed. That first 6 weeks is the easy stuff. The REALLY easy stuff. The last 20 weeks is the hard stuff, if somebody cares about polish, fit, and finish. These days a lot of people are skimping on the polish, because it really does chew up an inordinate amount of time.

Re:Simple methodology

[Darinbob]

And the specs and plans are all irrelevant when there are external forces conspiring against them. Oops, the parts we need aren't available yet or have problems... Or for something actualy happening now, the partner is late and the product manager is sweating bullets and asking us if we can still meet the deadline in two weeks despite not having done any integration yet. I was on one project that was about 6 months late because it took 6 months to get legal to approve an NDA agreement with a chip maker, which wrecked my 6 month goals/bonus.

Re:Simple methodology

[Darinbob]

Sometimes there are plain bad companies. Not just a manager, but a large hierarchy of managers going up to the top. One company acquired by another was basically treated like crap the whole time. At one point the bosses cancelled one project completely because it missed the deadline, no extension allowed. Soon after a very similar product was proposed and so the designers started pulling out the documentation from the previous project. No, the bosses said they can't do that, the project had been cancelled and that this was a new project so therefore the design must start from scratch.

(no names here, but it starts with S and ends in iemens)

Re:Simple methodology

[BronsCon]

I have a project that was due last week that I haven't started on yet. I got the deposit check the day I handed over the SoW, but didn't get the signed SoW back until last week, 6 weeks later. As per the terms of the SoW, I'll reschedule it when I find time.

Most of the delays I encounter are caused by someone else; either the need to refactor someone else's shit code (that I wasn't allowed to review before providing an estimate, of course), delayed approval for the work, delayed access to resources, any number of external forces. Very rarely do I exceed my estimated *hours* for a project unless changes are requested (but then I'm not exceeding the estimate, either, since I make the client either agree to a new estimate or accept a refund of any portion of their deposit not already applied to the hours I've worked), but all too often I find myself completing projects well past their due date because some resource wasn't made available to me until after that date had lapsed.

Fortunately, I foresaw that unforeseen things would happen when drafting the boilerplate language of my SoW and covered all of those cases. I go over the entire SoW with my clients before starting a project and make sure they know what the triggers are for a re-quote, what will cause the project to be delayed, and under what circumstances they're entitled to a refund of any deposit they pay (e.g. if they back out of the project once I've started work, I'm deducting my hours from that). As a result of that, and perhaps a bit of luck, I haven't had any disputes over project scope, budget, or timeline, and the one client I did have back out of a project simply said they no longer had the budget (they were being sued) and told me to hold on to the remainder of their deposit as they'd be back to finish the project after they lawsuit ended, hinting that, even if that didn't happen, the small sum would make no difference going forward (of course, I'm sitting on that money for now, and if they fully back out of the project I'll insist that they either accept the refund or sign a document releasing the funds to me).

That was one thing that really pissed me off when I was working for someone else; I had no control over external interruptions or delays and it was usually the person interrupting and delaying me who was holding me accountable for all of them. I'm not out to scam anyone, but I always felt like I was when dealing with my previous employer.

Re:Simple methodology

[lgw]

The first and foremost requirement is that when sales/marketing sales that they want 100 features in the product and there is no way that we can doo all 100 features, we have to be able to prioritize them

Yes, that's exactly how all Agile development works. Did that not work out for you?

Now, please tell me how Agile is going to tell me how much effort is going to be required to complete a particular project when the whole Agile approach is to start coding and figure it out as we go?

That's not quite Agile - you need enough of a plan and a design to know WTF you're doing before you start, you just don't need an in-depth design of stuff you won't even touch for 6 months. But you know what the components are, and about how big each one will be, at least relative to each other, and you know where the dependencies are. So how do you know ho long it will take? You measure you're progress! When you know you're 20% done, and it took a month, then you know it's a 5-month project. I've been within 5% on estimates for large projects, even as they changed significantly over the course of the project we could be that accurate on updated estimates.

OK, here is another requirement. We need to put together timelines because Engineering doesn't operate in a vacuum. We are part of a company with many parts all of which have to be coordinated

Schedule, scope, resources, quality. One of these four must be give when estimates are wrong or changes creep in. You have to pick one, or it will be quality be default. Agile is just saying "OK, so scope is what gives".

I've been doing agile-style development since before that word was coined, and I've never failed to deliver the must-haves on time for a project. Really. Honestly goes a long way.

The big problem with Agile is that it was created by Engineers in a vacuum who failed to recognize their responsibilities to the rest of the Company

So, where'd you get your MBA from?

Re:Simple methodology

[lgw]

That first 6 weeks is the easy stuff. The REALLY easy stuff. The last 20 weeks is the hard stuff, if somebody cares about polish, fit, and finish.

Not if you're good at your job. It's only if you suck at estimation that you have "the first 90%, then the second 90%". If you know from experience how software projects go, that's what all your estimates are based on. Only a junior engineer will think "this will take me 3 weeks to code, so I'll tell my boss 4 weeks just to be safe"! Coding is ~1/3rd of a typical software project (and less for environments with inflexible auditing or external testing requirements).

Of course, that doesn't stop managers from saying "ship it" the first time they see anything work. Heck, some will say "ship it" when the see the mock-up. But you have to manage those guys, or switch jobs.

Re:Simple methodology

[lgw]

Manager: "Get on with it! Work whatever hours you have to, we're late now!" ... instead of:

That actually happened to me once! It was a Thursday. By Monday I had a written offer for my next job and gave notice. That's the one nice thing about Silly Valley - bosses are as interchangeable as developers, so you're free to find one who's not a moron.

Of course, if you're stuck, you just triple (minimum) all your estimates, Scotty style.

Re:Simple methodology

[duarte_vasco]

Why estimate that way at all? If you have been working on a project for 6 weeks you already know the "real" capacity (throughput) of the project. Just take the velocity from the last 6 weeks (how many requirements/user stories) did you deliver to a production-like environment, and divide the remaining stories / requirements by that number. This will give you the number of weeks (take the average velocity/Week) that the project requires to be completed (if no requirements/stories are added ;) Me and many others have been working this way for many years (I started in 2005) and with great success...

Re:Simple methodology

[bsolar]

Even with bridges you might have to show a preliminary to-scale model which gets discussed and iterated upon. The difference is that with bridges there is a fundamental difference and a huge delta in costs between building a to-scale model and building the real thing. So you don't build a bridge that way because you don't have the same capabilities as with software.

Re:Simple methodology

[Vorga]

We've battled hard with internal managers and the customer to get some sanity in estimation. It's taken 3 years. The chrun and banter for estimation all comes back to mitigation to the vendor and the customer, for both delivery risk and financial risk. We start with a ballpark guess which we're happy to be wrong, but it gives a rough order of magnitude. If the ballpark is too high the work dies there. Then we do T&M consulting to construct a solution design, which also comes with an estimate (+/- 30%). During the T&M we happily accept variations to scope, as this is why it is T&M. After the Est is accepted we do a tech design, and then provide the quote as Fixed price. I'd love a fully T&M engagement, but many customers consider full T&M too open to abuse by dodgy vendors. Such is life in software development. At any point before accepting the quote the customer can pause and only pay the T&M for the design phase or ballpark phase to date. They only pay for the time we've spent, which limits their exposure and our risk. Customer likes it because they know in advance how big work will be, and we like getting paid for work and having expectations managed. If they accept the fix price quote, we work, and handle scope changes as changes; and we have a design spec and a tech spec to handle what is in or out of scope. There are very few arguments about scope these days. Our contingency in the estimate and quote is a very small percentage of the overall Dev and Test effort, but it almost always covers the little things we might have missed, which means no stress when we need a little extra funding to cover the gap. A key part for estimation for us is a work breakdown to at least x days or x weeks for each sub-part. If you don't know that then you're really not ready to construct an estimate and you need more thrashing out of the scope of work. Do the MBA types like it? I'm one, so well, yes. Devs seem to like it too.

Re:Simple methodology

[duarte_vasco]

Making the cost of changes visible (or painful) is one way to make decision-makers in the future consider the cost. However it still does not answer the question "when will we be done". I've been using (rolling wave) forecasting to make that clear. For example: it may be expensive to re-analyze one portion of the project requirements, but what will be the impact on the release date? You know, by Xmas we must be out or we will lose one-third of the yearly sales... Forecasting continuously, and removing/adding/changing scope in the project allow you to manage your release date on your own terms. Without spending time "guessing" how long something will take, and constantly adjusting the project content to be able to meet your business goals.

Re:Simple methodology

[duarte_vasco]

I can relate to these (anti)patterns of behavior. However, executives make those decisions because they have no way to evaluate (objectively) the impact of their decisions on the project. The software engineering community has been trying to solve this by adding more (and more, and more) complex estimation processes to software engineering. Only to discover that estimation fails very consistenly (I tackle that here: https://vimeo.com/111100275 / video from Oredev/2014). Because estimation is such a bad method to know the impact of decisions, the behaviors you describe have horrible consequences (I mean, how hard can it be, we are just asking you to change a field in a form....). We need a much better alternative to make project-schedule relevant decisions. I propose rolling wave forecasting, which builds on continuous and incremental releases that the Agile community has been advocating since the early 2000's. It is possible to make consequences visible, if we are just ready to accept that estimates are not the right approach to answering project critical questions.

Re:Simple methodology

[dotancohen]

In regards to client handling, you sound like you are at the point where I want to be! Would you mind sharing your SoW with us, or with just me at least? My Gmail username is the same as my /. username. I really appreciate it, and you could possible save me quite a headache as I'm still too young (37) to have made enough mistakes with clients and continually shifting requirements!

Thanks!

Re: Simple methodology

[jhoger]

car.color = config.carColor; :-)

Re:Simple methodology

[BronsCon]

You've got a few years on me, actually! I've just spent a little over 4 years working for a guy who's made some real boner mistakes and got to learn form him. :)

I'll drop you an email in the AM, I really should be asleep right now.

Re: Simple methodology

[BronsCon]

Well, yes, if a car was a data construct. Way to miss the point.

Or were you going for funny? in that case, yeah I'll grant you that. ;)

Re:Simple methodology

[hattig]

Simple guide for managers:

Double developer estimates then add some. Indeed, don't use days as a timeunit, just vague (fibonacci) numbers.
If task includes the words "timezones", "character sets" or "dependency on another team" then triple ... no quadruple the estimate.
And then consider unit tests, component tests and regression tests.

Oddly enough some things that a manager might look at and think "oh, that'll take loads of time" due to their lack of experience (for example, service should return JSON instead of XML) actually can turn out to be quite quick (as simple as configuring a different content negotiator / resolver on your project). OTOH the developer will probably have to redo all the component tests as well.

Re:Simple methodology

[TapeCutter]

Talk to them in MBA speak, call it "scope creep", they may not know what it means but they've been trained to avoid it.

Re:Simple methodology

[StormReaver]

One would hope that a good manager would have enough practical and direct experience in writing software to at least come up with a half-decent estimate, no?

No.

I've been writing software for 30 years, and I have never calculated an estimate that was even remotely accurate. Every project is unique, and must be treated as if it has never been done before (which it hasn't). Time estimates are snake oil sales.

[unrelated rant]

I hate, Hate, HATE, HATE!!!!! Slashdot's forum rules. Who has the time to wait a billion years between postings?! Who's the moron who comes up with this shit?! I registered at soylentnews.org a few days ago, and this stupid Slashdot notice:

Slow Down Cowboy!

Slashdot requires you to wait between each successful posting of a comment to allow everyone a fair chance at posting a comment.

It's been 3 minutes since you last successfully posted a comment

is the last straw. I've been on Slashdot for about 18 years now, I think, and I've had it with all the stupid site decisions of the last couple years. I'm done with Slashdot. This is my last posting to Slashdot. Hello, Soylent News!

[/unrelated rant]

Re:Simple methodology

[Anonymous Coward]

I've been programming for over thirty years, and I can't do estimates. At all.

I've been programming for over thirty years, and I'm pretty good at estimates.

It comes from a practice that I learned at Microsoft. Before Microsoft, I did the usual "Oh, about two weeks" estimate for everything that I was shown.

Design your feature or product. Then figure out how you're going to build it. Then break down that build into tasks that are no less than half a day and no longer than three days. Add it all together and divide by the fraction of a week where you're actually going to be productive. At Microsoft, I was using 0.8 because I'd inevitably lose a day a week to 'stuff'.

Caveats? Sure. Unfamiliarity with the problem domain. Unfamiliarity with the development tools. Lack of control over one's own development time. Lack of access to development resources. And many more. If I'm working in an environment where I cannot use the above technique to get a good estimate of how long it will take to build the feature or product I'm responsible for, then I'm either the wrong engineer or I'm working for the wrong people.

Note that when you do the above, you will come up with a shockingly-long period of time. But when you present that number to your boss, you will have the data to show that are a result of a detailed understanding of your tasks. You'll have your ducks in a row. Your boss will love you because the two of you can talk intelligently about how the schedule can be adjusted by altering the tasks in it.

If your manager wants you to alter the time it takes to complete tasks, you're back to "working for the wrong people".

Re:Simple methodology

[Gr8Apes]

Agile does NOT say "start coding and figure it out as we go." Agile says "Take some small subset of the functionality you've declared "the most important," estimate that, and deliver that."

So your version of Agile is create mini projects? Because no where does Agile say that mini-projects are part of the process. (One of the many failings of Agile is that no where does it really say anything meaningful) What you're describing is project planning, not Agile.

If you're a true Agile team, you're cross-functional, which means that proper representation from ALL of the organizations needed to deliver the product is present, embedded in your team.

Um, no, this only happens in organizations that are small enough that everyone's involved in 1 project. In the real world, I may have 100+ people working on a large project. There's no way everyone involved with all aspects are part of all teams. All they'd do all day is meetings, which, btw, is another one of the major failings of Agile, it leads to meeting paralysis.

While devs are banging out designs and code, Testers are working on test plans and test automation given the specs that the devs produce, and Support people are building operations documentation and support matrices etc. - all in parallel, all coordinating with one another as the process goes.

That's a nice fairy tale.

Re:Simple methodology

[Gr8Apes]

Agile is intended to get you to stop trying to jam a square peg in a round hole. The alternative is to pound on that bitch 'till it's round. Which one is likely to result in a better engineered end product?

If I need a round peg....

Re:Simple methodology

[Gr8Apes]

That's not quite Agile - you need enough of a plan and a design to know WTF you're doing before you start, you just don't need an in-depth design of stuff you won't even touch for 6 months. But you know what the components are, and about how big each one will be, at least relative to each other, and you know where the dependencies are.

What you're describing is not listed in the Agile manifesto, and certainly not in the first few years of Agile crap. This is progressive project planning, and it existed decades prior to the Agile plague.

So how do you know ho long it will take? You measure you're progress! When you know you're 20% done, and it took a month, then you know it's a 5-month project.

That's utter crap, and you know it if you're any good at all. You have no way to measure it that accurately unless you're doing cookie-cutter code where the next 10% of features are just as hard as the previous 10%.

I've been within 5% on estimates for large projects, even as they changed significantly over the course of the project we could be that accurate on updated estimates.

Why weren't you 100% on target? After all, you updated estimates as you went. The better question is how close were you to the original estimate with allowances for the major changes along the way? I've been within 10% on a number of projects I was in control of. With a specific Agile project being run by a PM, a BA, and 2 stake holders we were about 500% off. Notice a problem there? Not a single technical person. Because Agile told them you didn't need a technical person running the project, and that costs would be contained, because, well, they were using Agile. They even had their Agile certificates hanging on the wall.

Re:Simple methodology

[Gr8Apes]

Considering I've seen US Fed gov software development contracts pre-Agile and still think Agile is crap, I guess that proves you wrong. Granted, waterfall isn't great either, but therein lies another fallacy, for Agile people, it's waterfall or Agile, for everyone else, there common sense project planning which is in between those two.

Re:Simple methodology

[datavirtue]

In other news....my 6 year old daughter wants to do away with brussell sprouts.

Re:Simple methodology

[bluefoxlucid]

A good project manager uses historical data and the expert judgment of the project team to estimate the complexity of the project and determine the time required to perform the work. This is done in many ways: projects are compared to similar projects and scaled to size for a ball-park figure; projects are broken down into work packages, themselves broken down into activities and tasks, all of which the project team estimates the complexity and time required for, accounting for how long it took them to do similar things; time variance from historical data and current contexts are accounted for, providing a low, most-likely, and high mark (2 weeks in the best case scenario; most likely, based on prior work and known information, 3.5 weeks; some disaster scenarios bring in a 9 week estimate--that kind of lag HAS happened).

The more work is done, the more accurate estimates for budget and time become. Agile projects deliver in phases, iterations, and increments, and so can estimate work later in the project based on risk events earlier in the project--opportunities that cut time and can be exploited to cut more time, threats that cost time and may further cost additional time. In this way, "Historical Information" even includes work performance information for the parts of the project already completed.

It's all probabilities.

Re:Simple methodology

[Gr8Apes]

I design and build a data store, complete with an API in between. (the Foundation) It was designed with certain requirements in mind, because performance is a key metric I'm going to be measured on when it's done. Some Agile moron comes along 6 months later, and decides that he needs to filter with permissions in a new way that the DB was not designed for which works with today's traffic. Ideally this change requires a new foundation, because now, instead of an arch bridge, we're building a suspension bridge. The foundation requirements changed, and if we were doing it right, we'd tear everything down, but in the agile world we just slap some extra code (mortar) on the foundation, and voila, it runs/stands. Then traffic doubles, and the bridge fails.

You will pay the piper, sooner or later. Agile advocates later, which is why it's so popular with business types. That's fine if you never plan on being anywhere longer than a year or two, and don't really care what happens after that.

My last 2 sentences are how I run projects. It's not Agile in any way, although Agile did claim some of those features as its own, eventually. Do recall Agile, when it started, promoted XP, essentially code for how to take 2 developers and quarter their output. It's gone through several redefinitions since then, and just because it touts some ideas I've been following for years prior to Agile's conception doesn't mean those ideas are "Agile".

Re:Simple methodology

[Gr8Apes]

The blueprints is the requirements and planning stage, with API definitions. This does involve iterations. Coding is the building process, where checks (testing) validate the components before the next layer is added.

Re:Simple methodology

[Gr8Apes]

And that attitude describes exactly why Agile projects fail and have massive cost overruns and are late, because, surprisingly, there are more similarities between building bridges and building software than most suspect. Change costs money and time, and a change in a fundamental core component, can have outsized effects on everything else, including instability and poor performance, which can be overlooked during the Agile development process. I personally witnessed such a project where a component was designed and built, passed all its tests, but once it was to a point it could be connected to the other portions of the system, it failed miserably and the next 12 months was spent "fixing" it, which really never got to the point it needed to because every fix led to the discovery of a new problem.

Re:Simple methodology

[bluefoxlucid]

Then the people you are working with are stupid.

Get Rita Mulcahy's PM Crash Course (the green book) or RMC Project's CAPM Exam Prep, and the PMBOK 5th edition. Give a read through. Then you'll understand.

Re:Simple methodology

[bluefoxlucid]

Then, I recorded over/under for every project and found (over about 24 programmer data set) that programmers consistently overshot or undershot their estimates. So after a few projects, I had a pretty good idea of their deliverables.

Doing it right.

Made worse by the fact the indian contractors said "I'll do my best" for "no- you are batshit crazy" and then things fell apart when the indians were unable to deliver.

This is a cultural thing. Asian cultures are strongly hierarchical: you always agree with the guy above you. Never argue. You need to either read the cues or break them of that.

Re:Simple methodology

[west]

Next, that one line can be HUGE sometimes

This system will be multilingual.

This system will properly respect all time zones.

Two very simple sentences that a lot of people think can be tacked on very easily but take a lot of work. Especially, as you said, if you are swapping a "not" out.

This system will execute on Z/OS and iOS.

Re:Simple methodology

[Maxo-Texas]

In my experience, unless you really really fight it, all programming collapses to waterfall.

In my opinion, it is MUCH better to deliver a testable object every time-boxed period. Any changes (even just move one field) require a realistic re-estimate which must be signed off. But the actual creation of a testable object with feature sets that the testing department can test and confirm is key.

First- testing big O time is exponential. So solving smaller sets of bugs takes much less time.
Second- foundational bugs are found much earlier and can be fixed easier- or called out as significant problems.
Third- you get an actual percentage of completion this way. We've delivered 27% of the promised features and we are 30% along our schedule. We are a little behind schedule so let's work a little overtime next period and get caught up.

Never ever delay a build to get a feature in. Well, maybe a few hours- but not by a day.

Re:Simple methodology

[BronsCon]

And yet, you still get know-nothings who insist that you're wrong and want to push forward anyway. Thankfully, for the ones I've encountered, money is not an object and they're more than happy to pay for the work to be redone. I had one client try and argue the point and I told them very plainly that their options were to continue the project as originally detailed, take the incomplete work, as-is, or pay for the changes they were requesting. Which option they took is irrelevant, but I'll say the project got done and I got paid.

That doesn't negate the fact that people will still insist on changes, no matter how careful you are in communicating issues you find in their requirements and getting clarification before starting work. You're absolutely right, though; many people don't know how to handle it when it happens.

Re:Simple methodology

[BronsCon]

And my comment was about clients making changes to the spec. It doesn't matter how good your estimate is or how perfect your project plan is if the client decides to change the project after work has begun. You can refuse the changes, but they can also refuse to pay. You can give up the job, but hey, you're still not getting paid if you do that. The best hope you have is for the client to understand that their change, no matter how trivial it may seem, means more work for you and, therefore, more cost for them.

Or, to put it another way, go ahead and estimate the cost of that paint job. Then build a project plan for it that holds up in the situation I described. I'll be pricing you against a few local shops who are known for quality work; you high has to be competitive or I'll go elsewhere.

This isn't as much of an issue when your client is internal to the company, but that was clearly not the case in my analogy.

Re:Simple methodology

[KingMotley]

You realize you just said exactly what gr8apes said, right?

Most teams following Scrum ... keep the top (most valuable) items in their backlog up to date, clear, well-estimated, and ready for inclusion in a sprint. Work that's further out than 2-4 sprints ... can't accurately predict the state of things anyway, so there's very little point or value in trying.

So, you have no plan other than for a few weeks, and you are just sitting down and banging code out since "there's very little point or value in trying"...

Nowhere in any agile framework will you find something that says "lack of a plan is just fine - bang out that code and ignore everything else."

Yet, that is what you JUST said.

Re:Simple methodology

[lgw]

What does "running the project" mean here?

Agile projects need a Product Owner (and that's usually where projects diverge the most from the ideal - I've never met a PM who actually attended scrums), to stack rank work and answer team questions about requirements. But he doesn't "run the project". It's often handy to have a formal Scrum Master, if you're doing scrum, but he's doesn't "run the project".

Everywhere I've been, management "ran the project", just as with waterfall - they agreed on the delivery date and the staffing. One point of Agile is that the scope falls off, rather than the date slipping, but that's not all that different from tradition.

Re:Simple methodology

[eric_harris_76]

We've had the unpleasant experience of having an estimate being turned into a commitment. The company learned better.

Mostly. Our team recently made our own commitments from our own estimates. We probably should have put some serious fudge factor in our estimates, due to unknown impact from additional non-optional activities, before making those commitments.

On to cheerier things.

A couple of interesting articles on the topic.

http://www.mountaingoatsoftware.com/blog/separate-estimating-from-committing (especially relevant)

http://www.svpg.com/managing-commitments-in-an-agile-team/ (coming at it from a different direction)

Re:Simple methodology

[Maxo-Texas]

I looked at rolling wave and it looks good as long as the programmers are not allowed to do the easy predictable stuff first. That path leads to huge spending on failed projects.

My experience was that unless a legal or customer deadline wasn't behind it, executives preferred predictable delivery over efficient delivery.

I think another thing behind the executives odd behavior was they had a priority to get a "big win" fairly early in their time on the job. And they were often calculating in the fact they would be gone in 3 years or less to some other position.

So different priorities than a long term programmer who might want to tune and refactor the code to a polished easily maintainable gem.

Re:Simple methodology

[lgw]

Isn't that what I just said, except without the Agile jargon?

Re:Simple methodology

[Gr8Apes]

No, in fact, that's not what he said - he said Agile says "you don't need a plan."

I did not, in fact, state that at all. What I said is "One of the many failings of Agile is that no where does it really say anything meaningful" and "Because no where does Agile say that mini-projects are part of the process"

I said "Agile says 'plan in small increments, and revisit your plan to refactor and rework it constantly.'" What part of "keep the top priority things up to date, clear, and well-estimated," says "don't plan anything!" to you? Because in my world, estimating, clarifying, and 'scheduling things for inclusion in upcoming sprints' is EXACTLY what constitutes "making a plan."

Perhaps because I deal with projects that cover 100s of developers with budgets in the millions? And a business doesn't want to know in 8 or 16 weeks that their budget is going to be run over by 200%, or their timeline is going to be double? Ie, before committing to rewriting, lets say, an airline reservation or large shipping logistics system as examples, they'd like some realistic idea of the cost and time involved. Agile as the "manifesto" and in practice is incredibly inept at doing what I consider real work. It may work fine for small projects and teams, but that doesn't legitimize it any more than a working car made of paper mache is ready for 150mph on the autobahn.

So, you have no plan other than for a few weeks, and you are just sitting down and banging code out since "there's very little point or value in trying"...

No, you unmitigated buffoon,

That's pretty much exactly what you said, and now having been called out on it, you're resorting to name-calling. About as well thought out as the manifesto....

you have a rolling window of "several weeks to months of planned work," for the lifecycle of your project. You engage in planning THROUGHOUT the project life cycle, you don't just say "We'll plan the first 2 weeks, and then wing it from there."

And yet that is wholly the same thing as winging it for the scope of what I'm discussing. 2 weeks, 2 months, whatever your window is, is irrelevant. I don't need to know 12 months in that the costs are going to double, or the time line is going to be missed by even 1 month, or that I'm going to have to cut half my features, which basically means I miss delivery. This is the real world, and people don't like driving cars with 2 wheels and no seats, or flying in a plane with no windows. If you still don't get it, you're a... well, you ask it yourself, so, wear the badge with pride.

Anybody who is still able to pretend that this is what Agile teaches in 2015 is a fucking moron. Are you a fucking moron? You might just be.

Re:Simple methodology

[Gr8Apes]

What does "running the project" mean here?

The person who "owns" the project. Generally they run it, as it's their ass in the fire.

Agile projects need a Product Owner (and that's usually where projects diverge the most from the ideal - I've never met a PM who actually attended scrums), to stack rank work and answer team questions about requirements.

I don't believe I've been on an agile project where PMs did not run the scrums. Most because they thought they should, and 1 that actually ran the project like I would have. That project succeeded by the measures stated when I joined, but failed by the original measures. I'm not sure I'd call that project Agile nor SCRUM, but more like a development sweatshop, unless that's what successful Agile devolves to. All the other PM run projects failed, due to missed deadlines, missing functionality, and/or cost overruns. I have missed on occasion in projects where I was responsible, as has everyone, but never by more than 20% on time and I've always delivered everything in the original requirements. (see below)

...One point of Agile is that the scope falls off, rather than the date slipping, but that's not all that different from tradition.

I always agree to a core scope that must be met, and then the nice to haves that are negotiable. This approach leads me to a much better success rate, happier clients, and successful projects. The agile approach has left disaster and/or disappointment everywhere I've seen it. Because the world is promised, and only some is delivered, on "successful" projects. Success does not mean delivering everything on time and within budget. You at an optimistic best get 2 of those 3 with Agile.

Re:Simple methodology

[Gr8Apes]

You are clueless. There are cases where the original requirements missed a key bit of functionality that wasn't realized until a later requirement change caused a core change in the DB structure. In fact, I just finished altering a DB structure for exactly that reason. It took 4 days, but then, this current work is designed by a team that approaches software like me, so what's a relatively large change in structure and functionality was accomplished with relatively little effort and almost 0 affect on upper layers. We have well-defined application boundaries and integration API layers which by their nature are extremely loosely coupled. I have seen similar changes on other projects take months and teams of people to accomplish.

Re:Simple methodology

[lgw]

The person who "owns" the project. Generally they run it, as it's their ass in the fire.

Ah, well, the ideal anyway for Agile is that's the team.

don't believe I've been on an agile project where PMs did not run the scrums.

Wait, what? OK, by "PM" do you mean Product Manager (guy who's constantly visiting customers, or at least on the phone with them, often has an MBA), or a Project Manager (useless wanker). I've never seen Agile done well at companies that still employed the latter (but then I've never done consulting).

I always agree to a core scope that must be met, and then the nice to haves that are negotiable. This approach leads me to a much better success rate, happier clients, and successful projects

I agree that's the secret.

The agile approach has left disaster and/or disappointment everywhere I've seen it. Because the world is promised, and only some is delivered, on "successful" projects

Sounds like "Scrum consultants" selling snake oil, then moving on to the next victims ahead of the angry mob. Agile achieves 3 things if done right: much less throw-away work, early integration for fewer last-minute surprises, and a dev team who's emotionally committed to the dates, rather than hating management for the dates and wanting to hurt them in return. Those can make a pretty significant difference, but if you have intelligent, non-dickish management to begin with then only the first really changes (and if management is bad enough, nothing gets better).

Re:Simple methodology

[lgw]

How could it have "passed all its tests" if it wasn't connected to the rest of the system? It's hard to do agile without continuous integration; doesn't surprise me it was a mess. But integration blowups are the norm in my experience on waterfall projects - they're the main thing that leads to "the first 90% of the project, then the second 90% of the project".

But the primary win from agile is in avoiding throw-away work. You always work next on what's the most likely to survive unchanged, you only do the design work you need to write the code that you're going to work on (which often includes the entire high-level architecture for the first line of code, but still), you only document what you've actually done, and so on. Bridge specifications are unlikely to change after the project was funded. I've done sever 18-month waterfall software projects, and never seen one where more than half of what we thought the project was at the beginning was what we delivered at the end. Make it cheap and easy to change the requirements, because the requirement are going to change, and there's no holding back the tide.

Re:Simple methodology

[Gr8Apes]

Ah, well, the ideal anyway for Agile is that's the team.

The team's always in the fire, but generally the "big" guy is the one that takes responsibility for driving the project. You can't have a 100+ person "team" owning a project. Someone needs to be the grown up.

Wait, what? OK, by "PM" do you mean Product Manager (guy who's constantly visiting customers, or at least on the phone with them, often has an MBA), or a Project Manager (useless wanker).

the latter, and generally aptly described. Although on larger projects, they can be useful as long as they stay in their tracking role.

I've never seen Agile done well at companies that still employed the latter (but then I've never done consulting).

I've done some consulting over the years, mostly to rescue failing projects. But I've spent time in enough companies as an established employee to know the consulting experiences were by no means unique.

Sounds like "Scrum consultants" selling snake oil, then moving on to the next victims ahead of the angry mob.

As you can tell from my posts, I pretty much consider Agile itself snake oil.

Agile achieves 3 things if done right: much less throw-away work, early integration for fewer last-minute surprises, and a dev team who's emotionally committed to the dates, rather than hating management for the dates and wanting to hurt them in return. Those can make a pretty significant difference, but if you have intelligent, non-dickish management to begin with then only the first really changes (and if management is bad enough, nothing gets better).

I guess my point is that Agile just generally doesn't work the way it's pitched for projects that I work on, and is highly unpleasant and/or inefficient when people attempt to use some or all of it. To me, the only person that really benefits from the whole thing is the PM (either one actually) because they can see the lack of progress that results, all the while selling it up the chain as an accomplishment. I've seen several directors on up fired over the years for doing precisely that, only to be replaced by another Agile proponent that immediately starts having hour long scrums, because, obviously, people aren't getting enough work done. This is fine when I can stay on the outside, but these things are usually black holes that take in everyone near them. Then it's time to bail.

Re:Simple methodology

[Gr8Apes]

How could it have "passed all its tests" if it wasn't connected to the rest of the system?

It's a component. It's self-contained, like, say, a database or search appliance. You're not telling me they have to be attached to the rest of the system to be tested, do you? However, what happens under realistic loads with the oddities of connectivity sometimes fall outside even the most comprehensive testing, especially when distributed concurrency rears its pretty head.

It's hard to do agile without continuous integration;

I don't recall if they actually had their checkbox marked off, but we certainly did, with several hundred tests, all written to the spec. Funny thing, there were 3 specs, with multiple versions that had occurred over time. Somehow, the "official" spec in my hands wasn't the complete spec being developed by the other team, they'd decided in a scrum that changing a couple of details would make it easier for themselves. They wound up undoing those changes, and rewriting half their component as a result, extending the timeline by 300%.

But integration blowups are the norm in my experience ... - they're the main thing that leads to "the first 90% of the project, then the second 90% of the project".

Integration is about 90% of what I do. It doesn't matter what project style is used, if the integration isn't properly spec'd and planned, you will be doing a second 90% after the first, and possibly a third and more. (NOTE: I removed waterfall from your quote, because it's relevant to all. Integration can be a real problem.)

Make it cheap and easy to change the requirements, because the requirement are going to change, and there's no holding back the tide.

It's never going to be cheap and easy in general to change requirements. That's as true of building bridges as it is for building software. There are changes that are easy - make the railing painted black instead of pink, for instance, vs expensive - gold railings vs wood. But changing the base structure, going from arch to suspension to truss in any order will never be cheap. (Think going from relational to Mongo to Hadoop) The purpose is the same, get a car from one side to the other, the car manages it the same way - driving on a roadway, but everything else about it is different. For data stores, the data is stored and comes back out, but the aspects of how its done and the implications for the application server are rather different, depending upon how you set them up. One or more may not even meet the base requirements used as a starting point for designing the application.

ignorant hypocrites

[Anonymous Coward]

so, estimating time is a waste of time, but complaining about estimating time is not?

GET BACK TO WORK, MONKEY.

Re:ignorant hypocrites

[gnupun]

so, estimating time is a waste of time, but complaining about estimating time is not?

No, you complain only once, whereas software estimation has to be done over and over for every task... for many years. Software coding/design is similar to solving a maze -- you just can't give an accurate estimate how long it will take to solve the maze. Estimates are good for repetitive or non-creative tasks that have been done for many years and you know exactly how long some task takes.

Estimation in the software world is a scam perpetuated by managers and management to get developers to work extra hard.

Re:ignorant hypocrites

[unrtst]

Holy crap! Are people actually buying into this BS?

Software coding/design is similar to solving a maze -- you just can't give an accurate estimate how long it will take to solve the maze.

WTF? Yes, you can give an ESTIMATE on how long it'll take to solve a maze. Go get a book of mazes; Do 10 of them; Time yourself for each one; You now have min, max, and average times for a maze of that size/complexity. Next time someone shows you a maze, you can make an educated guess about how large/complex it is, then give your best case, worst case, and normal/average estimates. Do the same for programming.

Estimates are good for repetitive or non-creative tasks...

They're also good for creative tasks. I went to college with a major in fine art and worked for years as a monument engraver (etching portraits/landscapes/etc on tombstones). If someone asked about how long it'd take to etc a 5x7 portrait, I'd have a VERY accurate estimate for them. Is that not creative enough for you? How about every single project for every single class for every year of art school? ALL of those had timelines, and every student became quite good at estimating how long each project would take so they could get them all done on time. And you could ask most of them (at least those with higher than a B average) how long it would take them to do a certain thing (ex. sculpt a bust in clay from a live model) and, if they were familiar with that medium, then they could give you a very good estimate.

Estimation in the software world is a scam perpetuated by managers and management to get developers to work extra hard.

If management is asking the devs for their estimate, then how in the hell is it management fault for any of those timelines? Let me put this another way... devs, BE SMARTER. You've been asked for estimates on more than one occasion. Most that are new to development will low ball (stating a best case estimate). If/When you do that, you're just setting yourself up to fail. The best outcome will be that you are on time, and every other outcome sucks for all involved.

If you're not good at estimating, just try, come up with your figure,then double or triple it. If you always come in under the estimate, then you manager may start adjusting your estimates... who cares? Let them. It's their fault if you don't meet their manipulated figure. However, if you're not giving a large enough estimate in order to get your work done, then it is YOUR FAULT if you fail to hit your own estimate!

All that said, there are cases where providing an estimate is unreasonable. On one end of the scale, if a manager is asking for very accurate down-to-the-hour estimates on vague but somewhat small tasks, then it's asking for too much (almost more work estimating doing the damn thing). On the other end of the scale, if it's some grand idea for a giant project with no plan yet, you'll be pulling figures from your ass just as much as he's pulling the project plan out his ass... but it is what it is. Just go big.

A lot of it is just about setting expectations. If you give them high estimates, then you're more likely to meet or exceed their expectations. They may dislike the figure at first, but when you come in under budget they will be very happy and forget all about how high the estimate started out. A high estimate is not "padding", it's setting expectations that you can meet (see the maze estimate above... use "max" and you'll be able to meet or beat your estimate every time, which is what all involved really want out of your estimate).

Re:ignorant hypocrites

[unrtst]

Hate replying to myself, but I hadn't RTFA before posting. I'm not entirely against what they're saying/proposing. It's a different way to do things, rather than just some folks whining and complaining and avoiding estimates for stupid reasons.

Re:ignorant hypocrites

[jtwiegand]

Complaining is only considered valid work in meetings.

Re:ignorant hypocrites

[RabidReindeer]

The difference between software and mazes is that with mazes you are doing the same task every time.

I know, for example, that I can take cold iron and have it running most of what makes it a useful production server machine in about 4 hours. I'll sign my name in blood on it. Because I've done that same task over and over.

On the other hand, software is a creative work and creativity implies that you are NOT doing the same task every time and that at best you are guessing.

The problem is, everyone else is second-guessing, and they're guessing wrong. And they often will not accept a realistic estimate. They'll push for a "realistic" estimate, then blame you when you cannot meet it.

Actually developers guess wrong too - they usually think it's going to take half the time and work it really does (based on stats I've seen). But developers could simply allow for that by doubling what they think.

Except that in many cases, as I said, they cannot resist the pressure from outside.

Re:ignorant hypocrites

[naasking]

You now have min, max, and average times for a maze of that size/complexity

Estimating the size/complexity of a software project is exactly where all the error lies. The original quote was correct that it's like solving a maze, you just assumed they were talking about mazes on paper that you could estimate size and complexity at a glance. No, this is a full sized maze that you walk through and have no idea how deep and complex it is.

Re: ignorant hypocrites

[bulled]

Oh, you want an argument, this is abuse.

Re: ignorant hypocrites

[stoborrobots]

writing the article is a DIRECT response to being asked to provide estimates for ACTUAL WORK ... writing the article was most certainly done in lieu of that actual work...

.. assuming, of course, that the article wasn't written after the "actual work" was completed...

Re:ignorant hypocrites

[Oligonicella]

No, this is a full sized maze that you walk through and have no idea how deep and complex it is.

I did estimates my entire career and I believe only once did I walk into such a situation. My response was, of course, "Don't know. Not enough information."

In all other instances, the client provided much more than just the doorway to the maze, so making estimates backed up with rationale wasn't all that difficult. Also, during the initial discussions, I made sure to inform them what kind of information I needed to *make* said estimates. Don't ever recall being called unreasonable.

Re:ignorant hypocrites

[Kielistic]

I've experienced another:

m: estimate!
d: one week.
m: okay. (2 days later) we're starting to panic so we've decided to dump some other people into the project that don't know anything about it or the technology involved. This means it should be done by tomorrow.

Re:ignorant hypocrites

[dcollins]

...if they were familiar with that medium, then they could give you a very good estimate.

That's a keenly important conditional. My partner is a fine artist in fabric and mixed materials. She commonly has to spend weeks experimenting with new joint compounds, procedures, etc. (which can take days for one to dry to see if it works, etc.) For her next project she wants tapestry-sized plastic weaving to be glued stiff so it can be hung in space without a curtain rod. How long will it take to determine the right process? Is it even feasible? We don't know yet.

Arguably software development is more like that; you're always writing new material procedures on most new projects.

If management is asking the devs for their estimate, then how in the hell is it management fault for any of those timelines?

The last time I worked software, management took all my estimates and arbitrarily cut them in half, saying, "We're smarter than most other companies, so we can do it in half the time." Used that to close the contract with the outside client, etc.

Re: ignorant hypocrites

[Darinbob]

This isn't abuse, this is new hire training. You want the sweat lodge next door.

Re:ignorant hypocrites

[Darinbob]

I've never written the same program twice. I can't give an estimate on the next program because I can't guess at what new and unique hurdles will appear. I might be pulled off different proejct before it even finishes (that's not at all a rare occurence), or the parts just don't work (not rare), or I'm being asked to be an expert on something I'm completely unfamiliar with, or a thousand other things that I can't predict. If it's a sprint and I'm supposed to do A, B, and C, I often end up doing E instead which stands for the Emergency task that popped up unexpectedly.

And you can;t' double or triple your estimate because you'll be called out for it. The estimate is only there to confirm the deadline that was created by management before they even asked for your estimate. We're given three months for a project that I think will take a year, most others acknowledge it will be at least 6 months, but management kept saying "it's a challenge but I think we can rise to it." (these are people who think giving 110% is to be taken literally)

I am not new at this at all, I'm double the age of some of the people I work with, but I am still bad at estimates. Not just with software, but I can't even estimate distances or heights or weights. I'm bad at juggling too, I can't speak Spanish, my piano skills are very rusty. Not everyone has skill $X; so you're good at estimating projects, other people are bad at it, so stop expecting people to be good at everything.

Re:ignorant hypocrites

[Darinbob]

I am absolutely an expert. Sometimes a lot of of my time is wasted by people needing me to help them out, delaying me from getting my own work done. And I can't estimate. Sometimes because I'm not good at telling people to go away and leave me alone, but usually because unexpected things happen, or I'm being shuffled around between two many projects and can't multitask well, or I'm doing something new I've never done before (but that's ok, people know I'm smart and assume I'll get it all figured out).

And yet, someone will hand me a coredump then ask me how long it will be until I know now to fix the bug is. That's ridiculous, they may as well hand me a sealed envelope and ask me how long to solve the math problem that's inside it.

Even though you're Anonymous Coward, I suspect you're really a pointy haired boss sneaking into a nerd site to spy on us.

Re:ignorant hypocrites

[Darinbob]

Most managers are completely awful at management. Of course, most programmers are pretty mediocre at programming too. But bad management can hurt a lot more because there are fewer of them and they wield much more power. Sometimes these bad managers used to be the mediocre programmers who were promoted to where someone thought they'd do less damage.

But sometimes managers are good. The difference is whether the employees are expected to be helping the manager look good or if the manager is trying to make the employees look good instead.

And most of my job is actually jumping into shark infested waters carrying a book entitled "shark training made easy".

Re:ignorant hypocrites

[zwei2stein]

"Creativity" of software development is way overstated. It is not creative work in most cases.

Making yet another CRUD GUI is not exactly creative. Neither is any kind of usuall application you would be creating in business setting. Those mazes come presolved and you just trace line on paper.

In other cases, if you do not learn about problem being solved enough to be able to give good estimate - why are you professional? You should have business design document and technical design document and almost all gotchas covered already. What should remain is implementation itself and that is not much of creative work either - even if you have to wrestle with programing language to implement some concepts.

Re:ignorant hypocrites

[dwpro]

While much of your comment is correct, I don't think you can correlate the complexity of sculpting or painting to using technology (at least, complicated, integrated technology).  I can't foresee a sculpting project where you find that your scalpel is incompatible with the version of clay, so you're going to have to go dig up clay on the back yard or use a butterknife instead.

Re:ignorant hypocrites

[swb]

I think you've hit the nail on the head, and it's true for more than just writing software.

IT projects usually have two basic parts to them:

1) The part we know how to do and have done before. Usually it revolves around a set of tasks we've done before and that have a consistent outcome and usually exist sort of in a vacuum -- install an operating system, configure a switch, install an application.

2) The part we know how to do, but doesn't always have a consistent outcome because the specific thing hasn't been done before even though similar things have been done before. Similar tasks have been done before, maybe even identical tasks, but they haven't been done in this specific environment. Often involves integration with other systems or involves a lot of dependecies that are too hard/complex/time consuming to have a complete a priori understanding or produces an outcome which has a predicted but not completely predictable outcome.

Problems in (1) are usually easy to overcome because the solutions tend to be more easily known. Problems resulting from part (2) tend to have open-ended timelines because you often don't know what the problems are and have to create solutions.

I think people who haven't done IT work long enough or only get close enough to "supervise" it confuse the two, and assume that because even though (2) works often enough it seems to be reliably predictable.

I started borrowing Donald Rumsfeld's Iraq war quote about "known unknowns and unknown unknowns" when dealing with customers and managers about phase 2 type tasks. He may have been widely criticized for making excuses about the war, but I think there's some insight to the quote because it seems to somehow capture something about the unpredictability of outcomes with depdendencies.

Re: ignorant hypocrites

[MSJos]

This.

One manager I know is Miss 80%. She used to be more hands-on, and mostly everything I've seen her build was half-baked. And since she didn't understand the 80/20 rule,  she didn't understand that cleaning up her mess would take a lot more than her initial effort. To her, everything was simple, because she'd really only done 20% of the work - the easy part.

Re:ignorant hypocrites

[bluefoxlucid]

A core dump is two projects: find out what's wrong, fix it. You cannot estimate how long it will take to fix a problem until you know what the problem is.

How are you an expert when you consider a core dump to be a reasonable place someone would try to estimate bugfix time from? You can't estimate until you can plan the work, until you can draw a work breakdown structure and show what must be done. Even projects are chartered with a big budget and time estimate based on "this is 3x bigger than something else, so it takes 3x longer and is 3x as expensive", and then broken down into work that all comes together and says "okay, it's only going to be 2.1x as expensive and take 2.3x as long". That initial budget estimate? It comes from a dozen or five dozen or hundreds of prior projects, all with varying times, so you can say, "Stuff of this size and complexity has a low-water mark of like 5, a high-water mark of like 11, and tends to take more like 7.2" and decide how important the project is and thus if you want to budget for more like 5 or more like 11--and the same goes for the broken down work.

You can't even estimate what a bugfix is from a core dump. Someone brings you a core dump and says, "I need a bug fixed." They may as well bring you a blueprint and say, "I need a house built." Until you open the damn blue print up and see if you have a 1200sqft row home or a 4500 sqft Victorian, you have no fucking clue what you're doing, and can't tell them how long it's going to take. Once you unroll the damn thing, you can give them a ballpark estimate by glancing at the paper once; take a few hours to study the blueprints and work out what work actually needs to be done, and you can give them a better estimate.

either Fred Brooks

[turkeydance]

or Scott Adams quote goes here.

Re:either Fred Brooks

[Darinbob]

A soft voice says "plugh".

Well someone has to do it

[Sowelu]

Business can't plan or talk to customers or have any strategy whatsoever without at least some estimate...that's just the real world. If devs don't give estimates, managers have to make estimates. If managers don't make estimates, business makes estimates. You want devs to do the estimating.

Re:Well someone has to do it

[TheCastro1689]

I agree and disagree, the problem is managers don't understand the programming. If a manger knew that you were going to build off a similar product with low overhead and low time he could make that call. But a lot of it has more to do with how much are the devs budgeted.

Re:Well someone has to do it

[omnichad]

WHY CAN'T WE HAVE BLANK CHECKS

I thought this was funny, but it probably isn't - especially now that I have typed in all lowercase words to get pass the yell filter.

Re:Well someone has to do it

[BradMajors]

If a manager can't bring in a project in on time and on budget he is useless.

Re:Well someone has to do it

[BradMajors]

The only way to provide a reliable estimate is based upon past experience. A reliable estimate would hence be the same time it took for a similar prior project to be completed.

No understanding of programming is required.

Re:Well someone has to do it

[BradMajors]

Yes. The manager's estimate would be preferable.

Re:Well someone has to do it

[pjt33]

Without an understanding of programming, you can't reliably tell how similar a prior project is.

Re:Well someone has to do it

[Opportunist]

In this case, please inform me what the fuck that useless manager is here for? If I have to do his job, he's essentially a waste of precious oxygen.

Re:Well someone has to do it

[Marc_Hawke]

An understanding of 'similar' is required though...and without knowledge of the programming there's no way you'll know that. When the programming is all 'magic,' everything is similar.

Re:Well someone has to do it

[penguinoid]

Business can't plan or talk to customers or have any strategy whatsoever without at least some estimate...that's just the real world. If devs don't give estimates, managers have to make estimates. If managers don't make estimates, business makes estimates. You want devs to do the estimating.

I suspect the problem is not so much the estimates, it's that a manager demands an estimate from their underlings, tells them they don't like that estimate and give them a quicker estimate, promises their customers certain completion by the quicker estimate, and then complains when the programmers can't finish on time or have buggy code. That's without the obligatory change in the project requirements halfway through. At some places the estimate is nothing more that bending over for the manager.

Obligatory Dilbert: http://dilbert.com/strip/2015-02-22

Re:Well someone has to do it

[boristdog]

IF managers listened to the developers.

My last major project:
Manager: What resources do you need for this project (Replacing a huge, entirely file-based factory product manufacturing system)
ME: At minimum, I need two more people, plus someone to take over most of my smaller duties. And a couple new DB servers.
Manager: How long will this take?
ME: Two years minimum, since we are replacing an entrenched 15 year old system.

Manager's report to director: He can do it by himself in 6 months.

3 months later:
Manager: Why isn't this finished? You started over a year ago!
ME: I started three months ago, you gave me nothing I asked for and you've given me several other projects since then that you said were higher priority.
Manager: Well stop wasting time and get to work! I told the director it would be done by now!
ME: You gave me nothing. At least let me have a CS temp.
Manager: There's a woman in finance who has a sister that might want the job. She knows Excel really well, she can probably learn databases and programming.

Aaaand I think we've all been there.

Fortunately the sister didn't want the job and I got a great college grad as a temp. I would not have finished the project (in two years) without his help, we hired him after a year, too.

Re:Well someone has to do it

[chipschap]

I was a technically literate manager, having done lots and lots of programming myself. My job was simple: run interference with the client so that my team stayed funded. The team was very happy to let me do that job, which required a lot of travel and unpleasant politics.

In turn I trusted the team. I asked them for realistic estimates to give the client. If the team thought they weren't going to make it on time, I asked them for a heads-up as far ahead as possible, and I would take the news/new estimate for the client. I did not criticize the team, either to them directly or to the client.

They knew they were asked to do their best but software being what it is, they were not held to preliminary estimates. (The only issue was with downright incompetence or negligence, which was very rare.)

It's interesting that I was considered a good manager by the staff and the client, but not by my own management, who said I wasn't hard enough on the staff. Well, sorry, I got results and I kept us funded.

So, was I a "useless manager"? I hope not. I didn't produce anything tangible, and that bothered me, but I hope I played a useful role.

Re:Well someone has to do it

[BarbaraHudson]

In this case, please inform me what the fuck that useless manager is here for? If I have to do his job, he's essentially a waste of precious oxygen.

And this is a surprise?

Re:Well someone has to do it

[Zak3056]

I would not have finished the project (in two years) without his help, we hired him after a year, too.

First thought: your manager was a tool, and generally a waste of space--actually he wasn't even THAT useful, since he actively made things worse overall.

That said... the above quote is a bit damning. You claimed you needed two additional people, an empty task list, and two years. You did the job in two years, with other tasks encroaching on your time, and with a single new grad that you only had for 21 of those months. Either your project was a death march (not ruling that out, mind you), or your estimate was woefully off--maybe to the point that the dipshit manager, if you two had a history, simply didn't trust your ability to give him a good answer and modified it per past performance.

I'm sure there were many more factors in play than you mentioned above, which probably invalidate what I'm saying, but it might be worth taking a step back here and asking yourself if you made any mistakes you could learn from (other than working for Mr Clueless, of course).

Re:Well someone has to do it

[careysb]

Maybe you are a fine example of a manager. I've been in a couple of positions where I was asked for estimates and the managers thought they were too high and threw a dart at the calendar. Turned out my estimates were correct. Time to leave.

Re:Well someone has to do it

[BronsCon]

He may have been able to write more maintainable, more stable, better tested, and altogether cleaner code, given what he asked. Possibly in half of his worst-case estimate of 2 years. In the long run, the crap that needed to be slapped together to get the project "done", in 4x the time he was allotted, will end up costing the company more than giving him what he asked for in order to do it right. You must be an MBA.

Re:Well someone has to do it

[Oligonicella]

IF managers listened to the developers.

True. To bifurcate: IF your devs listen to the manager. I've watched dev members sit around investing inordinate amounts of time on the more trivial parts of projects because they found them personally fascinating. Watched those same types provide sky high estimates specifically to have fuck off time.

We've all been there too.

It's only one sided in any given situation but overall, I'd have to say I've seen more screw off software people than managers.

Re:Well someone has to do it

[jdavidb]

Business can't plan or talk to customers or have any strategy whatsoever without at least some estimate...that's just the real world. If devs don't give estimates, managers have to make estimates. If managers don't make estimates, business makes estimates. You want devs to do the estimating.

I just don't want the boss to be disrespectful to me when the estimate is not accurate. Get us some estimation training or something. Don't give me a lecture. I'm too old for lectures.

Re:Well someone has to do it

[Ryn]

If you want robots that waste not second...good luck. Engineering is a creative process.

Re:Well someone has to do it

[Vorga]

I'm a manager. I stay in meeting so that you don't have to. I build relationships that can withstand political friction. I cover our screw-ups, into nice language. I convert tech bullshit, into manager bullshit, then into customer bullshit. I convert scope creep into money. I get the contracts signed so we get paid, so the bank gets their mortgage payments. Useless? No. We are all replaceable though.

Re:Well someone has to do it

[Red Herring]

No, "computer science" (in the real sense) is a creative process.

Engineering is applying what you know to solve a specific problem to get a product out the door on a known schedule, and a known cost, and known quality. If you want to be "creative", perhaps engineering is not for you. There is a real, important difference.

Structural engineers know this.
Chemical engineers know this.
Process engineers know this.
Mechanical engineers know this.
Electrical engineers know this.
Traffic engineers know this.
And yes, computer engineers know this as well.

If you don't know this...

Re:Well someone has to do it

[gnupun]

In this case, please inform me what the fuck that useless manager is here for? If I have to do his job, he's essentially a waste of precious oxygen.

I think the manager is essential mainly to make sure programmers are not wasting too much time on games, slashdot, other social media, chit chat, etc. He also monitors the programmers and the situation around the project to provide feedback/status to upper management so they can decide whether to continue the project, change it or outright cancel it. He can be useful to programmers in connecting them with other developers or customers, or providing hardware, software or other relevant expensive stuff.

I figure a good rule is 2x what I think it'll take

[Anonymous Coward]

Because you have things that don't work as you expect, you have feedback to change the program, you have writing documentation and test suites, and your first gut instinct guess will expect very little of this, if you've been programming a long time. I've never overestimated with that rule, but I've only ever been really lucky to be within spitting distance of the time I thought it would take (just a little longer than I thought).

It really is a very inaccurate science, however, and management really don't understand it very well, in the main.

Without estimates you can't budget...

[SirGeek]

If you can't, based on the available facts (the design, requirements, etc.) , determine how long it will take to implement a project then you need to find someone who can. And you have to understand that there ARE external factors you can't control and if they change, the estimates change.

And management needs to know that your estimates are based on specific things (the requirements not changing). Any changes to requirements, etc. will impact schedule

If your company puts the budgets into stone, that's a different issue.

Re:Without estimates you can't budget...

[JWSmythe]

Lets see... What would they say? This is the one-sided conversation, since it doesn't matter what you say anyways.

"Ok, we can accept that estimate."

"Ya, ya, ya, whatever."

"We'll have that information to you by the start of the project."

"The information isn't ready yet, we'll have that by the time you need it."

"I thought we had that to you already. We'll have to check with the information source."

"The PMs have some changes."

"Here's the information, but there are some small changes."

"No, those are small changes, they won't impact the timeline."

"No, you can't have more time, we already made commitments."

"The PMs have some changes."

"What do you mean you won't have it in on schedule? You agreed with the initial estimate."

"You're going to stay here until it's done, I don't care how long it takes."

"I don't care that you've been in the office 30 hours straight, this is your fault."

"We're hiring an off-shore company to help you with the project. Get them up to speed."

"The PMs have some changes."

"Since we have the off-shore team, we need to cut your department back."

"I read an article saying Java is the future. Redo it in Java."

"What do you mean we're waiting on the off-shore company?"

"We fired the off-shore company. You're good, you can get it done in time."

"Ok, hire more people into your department, but we're only offering half the salary, and no more bodies."

"Why is this project so far behind? Don't you know what you're doing?"

"The PMs have these changes."

"Why aren't you done? We're weeks from the deadline!"

"You didn't meet the deadline. Don't you know deadlines are firm. We have commitments."

"I don't want excuses, I want results."

"You and your idiot team are fired. Get out of my building."

[2 months later]

"We need you to come back and finish the project. We need it by next Monday, that should be plenty of time."

"Here's all the new specs. They should be easy to do."

"What do you mean total rewrite, it's only a few chances. You are an idiot. Get out."

[1 month later]

"We need you to come back and finish the project. We need it by" {click}

"We need you to come back and finish the project. We need it by" {click}

"We need you to come back and finish the project. We need it by" {click}

"Why do you keep hanging up on me?" {click}

Good method for improving

[phantomfive]

Estimating is a crucial skill for developers. If you don't have it, you'll make bad decisions. For example, answer the question, "should I use framework A, or should I write some code myself?" If you can't estimate how long it will take to use the framework and compare it to how long it will take to write the code yourself, then it is impossible to make a realistic decision. Similarly, "should we write the code using method A, or write the code using method B?" If you don't know how long they take then you can't make good decisions.

This is a trick I learned from agile (although I'm sure it's been around longer than that). Each time you make an estimate, pay attention to how accurate it was, then next time adjust and make a better estimate based on that. Estimating shouldn't take much of your time.....if it does, then you're doing it wrong.

These guys seem to be complaining that their managers aren't doing anything with the estimates. Which probably means they don't understand what the managers do (but it could also mean their managers are bad).

Re:Good method for improving

[phantomfive]

Good managers are almost as rare as honest politicians.

Then teach your manager to be better. Reject the corporate structure (it may shift many times while you work at a company, even though not much else changes. That is an indication it's not particularly real), accept that improvement can come from even the lowliest programmer, and then help your manager do better.

If you think that your manager is completely bad in every way, you're not going to be able to help him. You need to be honest, recognize his strengths, and help him overcome his weaknesses.

You'll probably also need to overcome some of your own weaknesses, unless you are perfect. I can tell you one way to not accomplish anything: start a twitter campaign telling the world how bad your boss is. That will not make your system better.

Re:Good method for improving

[Great Big Bird]

This is a very insightful post —the original poster was asinine to even suggest that we need to get rid of them. For "software engineering" to grow up, we need more accurate means of pricing projects — otherwise you have no business.

Re:Good method for improving

[phantomfive]

This is a very insightful post

Thanks!!

Re:Good method for improving

[Just Some Guy]

If you don't have it, you'll make bad decisions. For example, answer the question, "should I use framework A, or should I write some code myself?" If you can't estimate how long it will take to use the framework and compare it to how long it will take to write the code yourself, then it is impossible to make a realistic decision.

That's a bad example because that's almost never my criteria. I could write my own framework almost as quickly as I could suss out the quirks of someone else's, and that's usually a teensy part of the overall project lifetime anyway. Instead, I judge on things like "do I want to spend the rest of my time here maintaining this thing?" and "who's going to own security updates?" and "will it be easier to hire people with experience on this one or on the one I haven't written yet?". Sometimes there's no good framework A to use, or maybe framework A exists and is popular but is unfit for this specific purpose, so we write something in-house. Either way, notice that "time to get started" is a trivial or nonexistent part of the equation.

Re:Good method for improving

[phantomfive]

That's a bad example because that's almost never my criteria

Yeah, but you're just some guy.

Re:Good method for improving

[phantomfive]

I see this thinking a lot, but I honestly don't understand how you can meaningfully compare something you've done with something you haven't.

Practice doing it, and soon you'll be able to do it accurately yourself. Think of a typical website.....the content is different, but the structure is just a bunch of pages, storing stuff in a database, maybe doing some authentication. A difficult project would be inventing a new algorithm. That's rare, and working on projects that are completely unrelated to anything you've done before is equally rare.

If you're working on a two-week sprint, by practicing and noticing how accurate your estimates were, then within two months you'll find that 80% of your estimates are accurate.

Caveat emptor: if you have a manager constantly pushing to make your estimates shorter, then learning to make accurate estimates is very hard.

Parkinson's Law

[jrq]

Estimates are necessary, so that you can determine project cost and (sometimes) duration. However, most projects succumb to my amended Parkinson's Law which states "work contracts or expands so as to fill the time available for its completion" .

Predicting the future is hard

[Krishnoid]

Joel Spolsky has a take on this problem, called Evidence-Based Scheduling, which tracks past estimates against their deliveries, and uses that to improve future estimates.

Re:Predicting the future is hard

[Marginal Coward]

I've seen that sort of thing used fairly effectively. If you keep track of how long past projects took, it's fairly easy to estimate a new project based on an estimated ratio of how complex the new project is compared to the most similar past project(s). This takes practice and a bit of a knack to do well, but it doesn't take much time, and it's certainly better than nothing.

What doesn't seem to work well, in my experience, is breaking down a project into microscopic detail and individually estimating each detail, e.g, the "Microsoft Project" approach. I can understand the appeal of that sort of thing, but it's almost impossible to use a data-driven approach at the microscopic detail unless you use some sort of "Personal Software Process" tracking tool - which very few people want to actually do because it feels like you're instrumenting yourself as a lab rat in someone's twisted experiment.

In any event, having a plan and a schedule, though inevitably imperfect ones, helps motivate everybody and helps keep things on track. The more enlightened managers of the world will allow these things to be revised along the way as needed, as their imperfections get revealed.

Re:Predicting the future is hard

[Sklivvz]

I work for Joel Spolsky and we have a single estimate: 6-8 weeks.

Estimation is basically useless because it has the unspoken assumption of "perfect design", which is an invalid one: software development is not a repeated exercise, it does not have consistency.

Having a single estimate does this to your projects: much smaller things, just do them; much bigger things, they are too big so don't do them, things vaguely in that scale, do them iteratively but define what you are trying to achieve in writing so you know when to stop.

No project managers, product managers work at a way higher scale (decide which are the projects worth working on).

Re:Predicting the future is hard

[Kjella]

Unfortunately most of the projects I've been on are of the "We want to replace old system A and make a new system B that also has features X, Y, Z" variety. What they want from the new system is usually okay, it's somewhat documented already, you've identified some stakeholders, you can show them prototypes, you can ask for clarifications and their testing validates the feature. Developing new code for genuinely new features is actually quite easy and fun.

The old system on the other hand is more like software archaeology, nobody really seems to have the specs - or maybe a spec 1.0 from 10 years ago that's got nothing to do with reality - and if you're replacing it it's probably because it's crap, uncommented code in an arcane language with poor frameworks and third party components. So you dig and keep digging and try to implement something similar without knowing what's a feature and what's a bug, who'll come yelling if you break something or features nobody told you about and you weren't aware anyone was using disappear.

I had a bit of an epiphany today at work when i finally found out how structure a major piece of the redo I'm working on and I've so far spent ~2 months digging through that code. From a similar job I was doing in another area I thought maybe 2-3 months total, now I'm guessing it'll be 6-12 because of all the rework I have to make and every apparently simple thing has exceptions and special cases. It's wasn't my bad estimation, it's that the conditions are entirely different. Like comparing travel speed for a walk in the park to chopping your way through a dense jungle.

Re:Predicting the future is hard

[RabidReindeer]

Very often the original system was hacked out in a couple of days/nights by one or 2 heavily-medicated (caffeine, alcohol, whatever) people.

It more or less did the job even though it has one or 2 really awful (but infrequent) bugs and is difficult to maintain or expand.

Then one day someone decides it's time for the Second System (see Fred Brooks).

They put together a team and a schedule and - like as not - spend a lot of money on fashionable faddish tools and consultants and a lot of time coming up with bells and whistles (a/k/a the Second System Effect - see above).

65 days into the 90-day schedule, they realize that they don't have any actual working code, just an immense collection of stick-figure UML diagrams, panic and put all the programmers to work coding on 100-hour weeks.

Guess what the end result is? Hint: read the papers. Somewhere between 66 and 75 percent of all major software projects fail.

Re:Predicting the future is hard

[Registered Coward v2]

Joel Spolsky has a take on this problem, called Evidence-Based Scheduling, which tracks past estimates against their deliveries, and uses that to improve future estimates.

That is a good idea, but you have to be careful how you track work. We used to do estimates based on hours worked and thus come up with a price. Since the work was fixed price the actual hours were irrelevant to the customer's cost; but very relevant to our team. Going over was bad; and going under yielded us no benefits other than having to explain low utilization; even if we had huge profit margins. As a result, our total hours billed to a project always exactly matched the estimate. One word of caution, never let the sales person estimate the cost or duration; they are rewarded on sales and so have an incentive to underbid to get work and let the team try to deliver.

Re:Predicting the future is hard

[pherthyl]

>> What doesn't seem to work well, in my experience, is breaking down a project into microscopic detail and individually estimating each detail

I find this very useful.. Not microscopic detail but detail enough that it forces you to really think about the project. I find without this developers tend to leave out tasks entirely.

Re:Predicting the future is hard

[dcollins]

So is the system in the linked 2007 article no longer used, never was used...?

Re:Predicting the future is hard

[Darinbob]

This only works though if a new task A can be compared to older task B. That so rarely happens to me. It might be common in some other areas though, as in "add a new web page with a different customers name up top". But when last month I worked on improving a crypto engine and this month I'm fixing the network MAC layer, and next month I'll do board bring up, then the evidence just doesn't accrue very fast. Then there's a problem that the amount of outside interference is not a constant, some months I can get lots done and in other months I barely have time for my real tasks.

Re:Predicting the future is hard

[Marginal Coward]

I guess everybody has their own approach to this, but in my case, I end up with a plan containing a couple of dozens of items, grouped into a few major phases. The whole plan will fit nicely on a single page of a spreadsheet. That said, I typically work on small projects, and larger projects which use the sort of approach I do might require several spreadsheet pages.

I've done some plans using project-planning software in the past, but they seem to encourage way more detail than is necessary. It's easy to get bogged down in all that stuff and spend a great deal of time on it. Later, as you actually try to follow details such as "begin task X on Wednesday, the 23rd at 10 AM", you realize that it isn't really practical to do so faithfully.

Re:Predicting the future is hard

[Sklivvz]

It's part of fogbugz. We use trello.

Depends on what "delivered" means

[mbeckman]

I've worked on a lot of software projects that delivered the original specified product on time. Sometimes the target changes, and the stakeholders need to be willing to give developers the extra time they request to meet the new objectives. Too often I hear, usually from upper managers, "We are still shipping on schedule. Tell the developers to work harder." Of course, that's not realistic, and the result is a predictable "failure to deliver." Alas, the developers get blamed, when it's really management's fault.

On the flip side we have so-called "agile" development teams who simply define a deliverable as whatever they've completed on delivery day. These developers rarely tell management until the 11th hour that what they're going to deliver is below spec. Agile development has its strengths, but this aspect is a giant weakness. The solution is not to eliminate schedules. It's to adapt them to changing conditions and be proactive about slippages.

Re:Depends on what "delivered" means

[AnarchyLime]

Not sure what agile teams you've worked with, but my agile teams have a delivery window predicted by the current backlog of work. If possible, don't define hard dates, define a delivery window based on confidence.

Re:Depends on what "delivered" means

[Atrox666]

I always get my coder to give me an estimate and I keep track of both the agreed upon and expected delivery based off history.
I always negotiate any changes I make to anything with the coder IN WRITING as stated in the contract.
If I'm changing the spec the coder gets to change his estimate of both time and price.
As a result I make damn sure my spec almost never changes.
It incentivizes me to know what the hell I want and not turn into one of the shitheads I've done projects for.
My coders work with a contractual gun to their head in the form of penalties for failure to deliver. I'm not a complete ass about it but when they fail to deliver they lose a lot of power in the relationship.
I hold myself to the same standard and if I go changing the spec it's lots of extra time and money for the coder.

If you can't estimate a job then I don't want you anywhere near my business. I don't care if your code brings tears to QA's eye.

My rule of software estimates

[sandytaru]

The amount of time it will take to complete a project is inversely proportional to the perceived difficulty of the project. This also applies to tasks and deliverables within the project. The project that you think will be easy and fast will be neither. The project that you think is going to be a tangled nightmare will turn out to have some surprising shortcuts and simplifications that make it not so bad.

Part of this is the stupid human trick factor - if we think something is going to be difficult, we approach it differently and with more caution. As a result we're more able to identify things to make the project go smoother. Conversely, if we think something is going to be easy, it's because we've only determined one path to approaching it, and if that path turns out to be non-viable, we have an immediate delay as we scramble to find other solutions that will work.

There's also the psychological factor at play for the customer - if we say something is going to be ghastly and difficult and will take us a long time ( because we think it will) but then turn around and deliver it ahead of schedule and under budget, we look good and feel good. This does not work if you say something will be ghastly and difficult but secretly think it will be easy and fast, however.

Re:I figure a good rule is 2x what I think it'll t

[Anonymous Coward]

i take half of what i think it will take, then up the units.... 4 hours = 2 days.... 4 days = 2 weeks. 4 months = 2 quarters.

Re:Estimates are for planning - not penalizing

[phantomfive]

When estimates are accurate, projects get done on time. Then business trusts programmers, money is made, and life becomes better.

People do not predict well and it is not smart to use it against them.

You can improve. Most programming we do is boring, and similar to something we've done before. For things like that, you should be able to make your estimates reasonably accurate.

Sometimes projects have large unknowns, and it's impossible to know how long it will take. In that case you need to propagate that information up to the people who need it, either managers or salespeople or whatever.

Re:I figure a good rule is 2x what I think it'll t

[poity]

https://www.youtube.com/watch?...

You didn't tell them how long it would REALLY take did you?

First, get your story straight...

[QuietLagoon]

...Software project estimates are too often wrong, and the more time we throw at making them, the more we steal from the real work of building software...

...developers' back-of-the-envelope estimates...

From the above two quotes, it looks as if the author does not even know if too much or too little time is spent on estimates.

.
No wonder his managers are frustrating with the time estimates provided. The guy has not a clue.

It's research

[plopez]

Programming is basically research and development You are building something which has never been built before. As such how do you estimate something if you have nothing to compare it to?

Actually though I have a limited amount of experience using Function Point Analysis. Some of the aspects of it are old, it was created before web apps became pervasive, but that doesn't matter. The principles are the same and there are dimensionless constants you can use to tune the FPA model to your organization. Eventually you end up with an empirical model of your programming team, development environment, tool suite, management efficiency, complexity of the problem domain, people quitting or dying, etc. You need to avoid over fitting however and recalibrate from time-to-time.

Agile keeps story points and velocity which serve the same purpose as the dimensionless constants of FPA. Unfortunately managers try to translate them to mythical man hours. But if you keep velocity or function points, whether you are using Agile or not, you should be able to get some idea. You have to be a bit scientific about it.

That and 'T-Shirt Sizing' is about as good as it gets.

Re:It's research

[Red Herring]

CSB: In the [very large] company I work for, there are two very different groups of programmers. There are the programmers that are EEs, working on products that happen to contain software components, and there are CS programmers. The thought process difference between them is, in general, quite stark.

The EE-background SW folks look at the SW task, spend (an often unreasonable amount of) upfront time to study the project, map it out, and then provide a schedule that pretty much matches what the reality ends up being, and get the product (of which SW is a component) out the door on time. And the code they write is crappy, unmaintainable, crap that does exactly what it's supposed to, no more, no less, and they are famous for rejecting any RCR that will break schedule. Engineering management loves them.

The CS-background folks (who cluster in a different group) will provide a poorly though out schedule quickly, and then immediately start missing deadlines. The code they write will be exquisite, maintainable, and it will have every feature and new hotness that anyone comes up with. Sales and marketing loves them, because they can sell whatever they want and promise anything to the customer, but it often misses the market window.

Moral: "Engineering SW" and "CS" mindsets are, in my company, quite different. Just because a CS-type tells you they can't provide a schedule, or the schedule is bunk, doesn't mean it's not possible to provide a schedule for a SW project. You're probably just asking the wrong kind of person. That may be OK, or not, depending on what you're building and what your project is.

Go Agile.

[Anonymous Coward]

Almost all project that I've done in Agile are on budget, on target.

The trick ? Prioritize the most value added feature and do so until the end of the backlog. When the time's up arrives, you got the most valuable feature on the product. Even the bugs caught while developing are prioritized.

The end product will be near of what the customer was expecting. As a bonus, the modification asked by the customer in the middle of the dev is also prioritized by the value added to the product.

I wish I had training

[kwikrick]

I have similar feelings about estimates in my work: It's never accurate, so why bother. Of course, I understand we need estimates.
I wish I had some training in how to make good estimates.
Recentely, I've been giving my managers a range: between 1 and 3 weeks for this feature. Half a day up to a day for this bug fix. Of course, manamament just takes the average, add ups the numbers and doesn't tell the client about the probability distribution.

Not just software

[CastrTroy]

I always here that software projects are often late and over budget, but I don't think it's worse than any other industry. I've seen countless examples of construction projects that ran over budget and took longer than expected. Often the reasons for this are the same. Either the requirements changed halfway through, or the project was made more complicated than it needed to be to accomplish the task. There's a few bridges in my area that have been huge boondoggles in the past decade, and they all try to look impressive, where a much more conservative design would have be easier, cheaper, and faster to build, and still would have solved the transportation problem. But everybody wants a bridge that looks pretty.

Projects that deal with a small workload and don't have changing requirements are much more likely to stay on budget and on time. This is how things should be broken up. Build small pieces and deliver the pieces as they become complete. Don't set out to build an entire 5 year task as a single project.

Re:Not just software

[majid_aldo]

disagree. the individual activities of construction are repeatable and well-known. this is in contrast to software.

Hmmm ....

[gstoddart]

On the one hand, it's pretty much impossible to do any project planning with no estimates.

On the other hand, some things are impossible to estimate until you do them.

Years ago I worked with a manager who kept repeating that bad estimates were a project risk and we should give good estimates. We kept telling him that an estimate is, by definition, based on incomplete knowledge and before you have done the work and that if he had a time machine we could give him better estimates.

If I knew exactly how long it would take, and what unforseen things I'd be running into ... it wouldn't be a frickin' estimate, now would it?

People treat estimates like you're expected to have perfect knowledge of the future, and then build their world around those estimates.

I have seen a tremendous amount of bullshit and stupidity from people who do not understand what an estimate is, and how it's done.

I don't think you can get rid of estimates entirely ... but management needs to stop being so stupid about how they interpret them.

If we could tell you for a fact exactly how long it would take, it wouldn't be a fucking estimate.

I rank how people do estimates right up there with how some PMs want you to track your time ... once had a PM say he wanted me to account for my time in 5 minute increments. And I told him in no uncertain terms that would mean 2 out of every 5 minutes would be spent documenting what I'd done the last two minutes, and there would be an additional 1 minute of lost time in each 5 minute increment doing to context switch back to what I was working, and that effectively 60% of my time would be wasted on his stupidity.

And then I told him to piss off.

Re:Hmmm ....

[srichard25]

I agree. Estimates aren't a problem for teams that understand what "estimate" means. It isn't exact. It can't be exact. As long as you respect it for what it is, then it is a powerful planning tool.

The next time a business person gripes about estimates not being accurate enough, ask him/her to estimate (to the minute) how long it will them to drive home during rush hour traffic. When they start complaining about how an unexpected accident would cause the estimate to be very inaccurate, then a light bulb will go off.

Re:Hmmm ....

[david_bonn]

A couple of observations.

There is a hellacious difference between an estimate and a deadline. A lot of the problem is abusing estimates and quietly or unconsciously turning an estimate into a deadline. The process for producing the two is totally different, or at least should be.

One very good manager I had would never directly ask me for an estimate. He would ask me how long it would take to figure out how long it would take to code x. If you can make a reasonable estimate of the size and complexity of the problem you are trying to solve you can begin to make a reasonable estimate of how long it will take. Without that information you are pulling numbers out of your ass.

One other good manager I had emphasized to me that estimates were, well, estimates. It was as big a problem if I always overestimated as it would be if I always underestimates.

Re:Hmmm ....

[CBravo]

At our place, being PM does not mean being my boss. As PM you get to write specs and acceptance criteria. My boss is someone else. The process is managed by someone else again, the scrum master. It works nice.

Selling ourselves down the river

[Anonymous Coward]

Research into procrastination has noted that people have much less concern about their future selves than their present selves and are willing to sell their future selves down the river for the sake of present ease. But when the present marches into the future, and we are confronted with the work that our past selves refused to do, we pay the price in unmet deadlines, all-nighters and general torment.

http://www.nytimes.com/2015/01...

The result becomes one of people providing estimates that allow them to screw off in the present and postpone the agony of late nights, weekends and looming deadlines for taking it easy today. This seems to always happen in our organization. A current project has had the hardware in place and ready for the teams to load their software on and configure the system for 6 months, but they waited until 3 weeks before the project is due to start and now they are changing the project design to make it easier on themselves in the present as well. Estimates = wasted time. Just do it.

Correction

[Anonymous Coward]

[Bad] programmers argue that estimates are wrong too often and a waste of time.

FTFY

"It's hard, so we won't do it"

[mveloso]

Part of being a software professional is understanding how long it takes for you to do things

If you don't know how, go read a book. It's not hard. The money that's used to pay you depends on your estimate being somewhat correct.

Imagine how upset you would be if you asked for your paycheck, and payroll said "we're not sure when we're going to pay you, but it should be sometime soon."

Re:"It's hard, so we won't do it"

[Gliscameria]

Thanks, I was beginning to think that software developers were completely dissociated from the rest of the world. Writing the software isn't the only part of the business. Marketing and sales are going to have to know about when things are done so they know when to have everything ready, and that's not even getting into the financial and executive parts of things. Not to mention you are basically saying that you have no idea how much the project is going to cost. Really, if you can't at least provide an estimate then it's either not your job to do so or you need to find a new job.

Re:"It's hard, so we won't do it"

[Anonymous Coward]

I don't know about you but I'm not making cookies. Sounds like you're not doing any design either...or implementation in the real world when bugs appear, refactoring is required, etc.

Re:"It's hard, so we won't do it"

[gstoddart]

and that's not even getting into the financial and executive parts of things

Yeah, and tell us, what is the track record of the financial and executive teams ability to prognosticate?

Yes, you need to be able to estimate to run the business.

But let's not pretend the average CEO, sales wanker, or marketing idiot has ANY better track record at making guesses about the future. In fact, in my experience, they're overly optimistic, not founded in anything real, and mostly pulled out of their ass of based on what Gartner tells them.

We can give you an estimate, but people have to understand that an estimate inherently carries uncertainty, and that they're equally inept over the long run of estimating the parts they're responsible for.

I've lost track of the times I've rolled by eyes when a CEO tells us what six months down the road will be ... and they have the ability waste far more money on fools errands and bad predictions.

We're not dissociated from reality ... we're the ones trying to explain reality to people who live in fantasy land.

But don't act for a minute like our estimates carry any more risk than those bullshit sales figures the idiots at the top are making.

Re:"It's hard, so we won't do it"

[Gliscameria]

We're not talking about accuracy or track records, we're talking about providing an estimate. These organizations estimate and project constantly, and they are wrong a lot, but it works overall. It gives some structure and a goal for the project. Sales projections are clearly worse, pure voodoo, but we still do them, because they work somehow. At the very least, these estimates come from someone charged with knowing a lot about the subject, so it's a good idea to clue everyone else in on what you think the timeline could be.

Re:"It's hard, so we won't do it"

[mveloso]

Actually, I've done all that. It's really not that hard to estimate, unless you don't pay attention to how long it takes to do stuff. It's not like you're trying to figure out how to fuse atoms, and have to build the manhattan project from scratch.

Writing something new is more difficult, but when you break it down into chunks it becomes easier. You do have to have a good understanding of how you work.

Just like anything, it takes practice. If you have enough people estimating you can bayes the answers, sort of.

Re:"It's hard, so we won't do it"

[Ash-Fox]

Managers who use schedules to push workers harder are shitty managers and will likely end up with a bad product. That is the sort of problem that fixes itself.

From personal experience, you can have a shitty manager that is overworking your entire team, but the team still makes a quality product and that manager looks good despite being shitty.

Tilting at Windmills

[Cytotoxic]

I used to tilt at this very windmill myself. It took me a long time to realize that people really, really need to hear an answer to "how long?". When we were in startup mode a long project was a matter of a few weeks. Everything had to go into production immediately. So we got used to banging stuff out in small chunks and doing it as quickly as possible. Entire project timelines were completed in less time than it takes to draft a proper requirements document.

But as we grew in size, the new people were not happy with our development team. Even though they would ask for a new report at 10am and have it in production before they returned from lunch, they still felt like we were not responsive. It was one of those reports that began to help me understand the psychology.

There was a problem with a very complex Crystal Reports document one morning. The Director of the department called to let us know about it. I told him we were already working on it and would have it fixed as soon as possible. "When will it be fixed?" he asked. Well, I had no clue. We had only learned about the problem 10 minutes before and hadn't figured out what was broken. So I explained this to him and told him that we had this as our top priority and it would be fixed as quickly as possible. I certainly thought that should make him happy. Well, it didn't.

He was rather pissed that I wouldn't give him a timeline. The day before we had made some changes for him that took about 2 hours, but he was upset that we didn't let him know how long it would take. Being an engineering type, when I hear "how long will this take?", I hear a request for a certain degree of precision. The problem with short projects like this is that by the time you have enough information to give an honest estimate, you are pretty much done. Maybe you have 15 minutes or an hour to go, but nothing worth reporting to anyone. Well, after explaining my position a few times and just making him more angry I finally gave up and just gave him a made-up deadline of Thursday afternoon. He was perfectly happy. I had just spent 15 minutes trying to explain to him that we were working feverishly and would be done as soon as we could (which would likely be a couple of hours, but who knows) and he hated that answer. "We'll have it for you in 3 days!" made him very happy. Even though his production was impeded by the lack of this particular report. From a human psychology standpoint he would rather know that it will be done in 3 days, barring delays, than not know when it will be done and have it in two hours. I personally think that is a dumb way of doing things, but I am the outlier, not the director.

After that learning experience we began implementing a more comprehensive SDLC process and providing timelines for projects. Everyone was much happier with the development team after this. Even though their projects went into production much more slowly. They loved the perceived control that having a timeline gave them. We developers know that these things are basically fictional documents - just educated guesses really - but it provides real customer satisfaction, so we keep with it. In fact, we kept evolving this idea into more and more involvement from the business unit as we moved into Agile and SCRUM methodologies.

I would say that unless you are working in an organization of less than 25 people, providing timelines is an absolute requirement from a purely human standpoint. This comes from hard experience - even though I think that everything about a timeline is probably B.S. and all of the effort that goes into preparing one would have been better spent solving problems and building something useful. In the real world you can't ignore the psychological needs of the group.

Re:Tilting at Windmills

[Zorpheus]

When you give an estimate you give your boss something to plan with. He can get things prepared and plan what is done after you finish. It allows to use the time well, and it assures that it will be fixed, but is not done yet.
I think it does not matter so much that you take a bit longer that way. He can make plans for the rest of the company with that estimate, so it allows everyone else to make better use of their time, and it brings things into order.
At least that is what I think.

Re:Tilting at Windmills

[Ryn]

It allows your boss to have a scapegoat. YOU are giving him the estimate, he's not having to come up with one. "They told me X, it wasn't ready in X, we made plans, I even added 10% padding and they still did not deliver".

Re:Tilting at Windmills

[Tom]

From a human psychology standpoint he would rather know that it will be done in 3 days, barring delays, than not know when it will be done and have it in two hours. I personally think that is a dumb way of doing things, but I am the outlier, not the director.

The psychological issue is that you don't know, but you have a hunch, you have some insight. You know it's probably going to be a few hours.

But for non-techies, all this stuff is a total blackbox. When you say "I don't know" they panic, because for them that means anything from a day to a month or maybe infinity. Uncertainty is a horrible psychological state and people try to avoid it. It's an instinct. When you don't know if that shadow is a monkey or a lion, it's better to panic just in case.

By saying "three days", you give him certainty. Now he knows the shadow isn't a lion.

Re:Tilting at Windmills

[goose-incarnated]

TBH I'd be rather pissed at someone who could not give me a ballpark or upper range.

If I know I can write that report later today, then I can let our supplier know that they'll get the report tomorrow morning. If I know that there is no way it is going to happen in the next week then I can inform the other people down the chain and they can plan accordingly.

Put yourself in their position - you're waiting for a lift from a friend but he hasn't shown up. When you phone him all he can tell you is that he will be late. How late? Will you have enough time to grab lunch before he arrives? Can you take a nap? A vacation? But yet, he still refuses to tell you whether taking a piss or taking a vacation is more appropriate.

I hate to say it, but it is your "engineering" (which really isn't) PoV that is the illogical and irrational one. The logical and rational answer is that people can be scheduling other things to fit into the time available, and they know this. You basically asked someone to wait indefinitely and then get all surprised when they get annoyed - of course they are going to get annoyed. You were behaving in an irrational manner!

My estimates are all made with disclaimers - "I can give you X, Y and Z within $TIME_PERIOD, subject to A, B and C being true throughout $TIME_PERIOD". And since I've never ever ever had A, B and C remain true through any development project my estimates have not yet been tested for accuracy. What does happen is a revised estimate when A, B and C change, and then due to having more information I have a more accurate revised estimate.

Your way is not logical, nor is it at all rational, nor is it considered engineering in any circumstances. It is not a psychological need that your estimate gets used for, it's an actual physical and concrete need. Your need for self-worth is what maintains your self-delusion that those people don't really need what they are asking for. Truly, we live in an age of narcissm.

Re:Tilting at Windmills

[Cytotoxic]

Wow, project much?

You managed to cram reading comprehension fail, reiterating my main point while telling me I don't know what I'm talking about and preening all into one post. Great job!

As to the substance of why estimates are a good thing, we are in perfect alignment. But how do you provide an estimate for the issues I was talking about? Let's say the one of the financial reports on Hyperion is not working right. Upon immediate inspection your guy doesn't see what the problem is. How do you provide an estimate? This is something you intend to have fixed in the next hour or two. You have your best people on it.

But it could be a problem with the report. Or a bug in the report server. Or a problem with the SQL server. Or a problem with the optimizations on the view it is calling. Or maybe it is a problem with the account mapping application - or maybe someone in accounting changed the map and didn't realize what the consequences would be elsewhere. At this moment in time you literally have no idea at all what the timeline is. It could be an hour. You hope it is an hour. But it could just as easily be a bug in Hyperion that will require a fix from the vendor. That would take weeks. Now, in an hour or two you'll have enough information to provide a decent answer. You'll know if the problem is bigger than you can handle with your staff. But they want an answer right now. This is where the real clash comes in .

They are demanding an answer for the reasons you have outlined. But you can't give one. Not the honest answer anyway. Because the honest answer at that moment is "I don't know", and nobody wants to hear that. The solution to this problem for us was to let them know we were looking into it and we would set a time to give them an estimate in a couple of hours. Now, in this situation the problem will be solved before the estimate is due most of the time, which just feels weird. But it addresses all of the issues that people have with the process. By saying less than we know, we communicate better. Saying "I'll look into it and get back with you by 3 o'clock" works perfectly. But coming from our culture of "everything ASAP" this felt wrong. My real intention was to have the whole thing done by 11am. But that wasn't a number you could promise to anyone, because you have no clue if you are dealing with horses or zebras. Once we figured out that the new people would rather wait a little longer and hear a firm answer, everything was fine. But in our startup years that would never have worked.

Re:Tilting at Windmills

[toddestan]

That's pretty much it. He knows his boss is the same way, and doesn't want to give some wishy-washy we don't know but we're working on it type of answer when his boss asks how long it's going to take. By giving him an answer, even an answer he may even know you made up on the spot, he's got something that he can tell his boss, and when that estimate turns out wrong he's got someone else to blame.

Miracle Worker

[burning_plastic]

Just make sure you multiply your estimates by a factor of four...

Re:Miracle Worker

[Gliscameria]

Our rule for new projects was x2 for a best case scenario and then raised an order of magnitude(hour-day-week-month-year). For example - If everything goes well this job should take 1 day of continuous work - so we estimate in our hellhole of a reality it will be done in 2 weeks. It was surprisingly accurate.

Typical Estimating

[danbert8]

If you need an estimate real bad, you'll get a real bad estimate.

The real problem with "time writing" is that people end up lying on the reports to meet goals or metrics and then the estimates created from that data are useless. Managers want both though. They want accurate estimates AND time tracking that meets productivity goals. Sadly their productivity goals would require robots to meet...

There's One More Pitfall

[31415926535897]

Let's say you've figured out pretty well how long something will take and you give your realistic estimate. The game from the managers then becomes, "That's too long, you need to give a more realistic deadline."

This pressure on the other side is often why developers set deadlines that are too short and then miss them. The penalties for that are less of a problem if you come off as sandbagging (even when you aren't). Managers who have no clue what the complexity of the problems trying to be solved not trusting developers are the real problem.

Re:Is this really a problem unique to devs??

[duck_rifted]

If non-trivial problems could be solved within a guaranteed time frame, then you'd have already have a doorstep delivery date for your starship. Good management can make sure that core features are delivered on a schedule and help ensure that the schedule is not actually impossible to meet. If the schedule is not lenient enough, then the best management in the world can not guarantee delivery while also providing a guarantee for stability, security, and performance. Were this a simple matter, then the service of software development would not be worth so much.

You have three choices. You can give me enough time to finish the project correctly and receive an awesome result, but you will pay for that extra development time. You can give me too little time to deliver a quality result, but you will pay me for patches. You can give me too little development time and opt out of paying for updates, but you will pay for it when there are problems. Choose well.

There's a very old, very over-used saying about technology that people must be forgetting. It comes with three possible qualities: good, fast, and cheap. You may choose two.

Estimates are like contracts

[omnichad]

They have to cover every detail and be iron clad. But by the time you've come up with all of these specifications, most of the work is already done. So how do you recover the cost of an estimate if the other party says no?

I came from the web development world as a one-man department of a larger computer store. Quoting projects was an absolute nightmare and the specifications always changed even when they somehow manage to fit the definition of the words in our estimate.

Estimates are not the problem... it just sucks

[martiniturbide]

Is this real or maybe my english reading skills are not good enough to understand it.

Incorrect estimates are the problem. When non-experience people estimate, or when management force to under-estimate that is the real issue.

Estimates are necessary in software development, but of course that "estimating" sucks just like "deadlines", "taxes", "death", "breaking up with your girlfriend in person", "no wifi", "cancer", "divorce" and many other things.

I have a solution, why don't you go to your boss and say "Estimating sucks, and I would not do that because I don't want to compromise any time. So you can "Estimate" my monthly salary according to whatever deadline we will never accomplish".... that sounds fair.

Re:Is this really a problem unique to devs??

[bughunter]

No, it's a very common problem in engineering in general, and not unique to software. But the reaction "let's eliminate estimates" appears to be.

As an engineering manager, I learned the hard way many times how estimates turn into deadlines. Your estimate is reported to the manager's manager and so on up the line, and someone uses it in planning their shit.

Your estimate, in which you did not build any schedule margin, then becomes an item in the critical path of someone else's plan, someone who didn't build in any margin either, or —worse— who was pressured to make a completely fictional "plan" which is really just a backwards-calculated paper justification to "prove" that a job could be completed in an impossibly short period of time by assuming nine women can make a baby in one month and things like shipping, reproduction, and quality assurance take place in zero time. This "plan" makes upper management happy. Temporarily.

You, leader of a small team that is working merrily away, accomplishing real work and solving the occasional unexpected problem (OEM pinouts were wrong, widget zeta delayed in shipping, amplifier stage behaving like oscillator, etc.), are asked for a status update. Because of your unexpected problems, your estimated completion date is now two weeks later than your previous estimate.

Now the middle manager, who knew he wasn't going to meet the "plan" he was forced to develop, now has someone to place the blame on. He knows he's going to be in the path of a metric fuckton of shit, but he's placed himself uphill of you.

It's clear even in TFS that the real problem isn't estimation, it's poor program management, lack of requirements management, and often also marketing-driven decision-making.

In other words, the same old shit.

Re:Estimates are for planning - not penalizing

[Opportunist]

If you penalize me for going over time and budget, my next estimate is 30 eons and 200 trillion USD.

Watch me come in well inside the budget.

Estimates are easy.

[Ryanrule]

Take your best guess. Multiply by 3. Done.

Get rid of time estimates?

[Yunzil]

Peter Molyneux is that you?

Every project will be behind schedule

[Opportunist]

By definition. When you look at our current corporate culture, you know it has to be. For a simple reason: Companies bidding for jobs. And more often than not, the cheapest offer gets the deal.

Who is the cheapest? Usually the one that cut the most corners and underestimated his cost (i.e. time) to deliver the most.

Re:I figure a good rule is 2x what I think it'll t

[JWSmythe]

He was a wise man...

The corrections and collaborations are the problem

[Karmashock]

Often the estimate includes an estimate of how many times the client will change something or whatever. And really THAT has to be made a part of the estimation system.

Estimate a time if they basically don't say anything more and provide required information promptly.

Then say ANY change to that what so ever is going to change the estimate in UNPREDICTABLE ways because you don't know what they're going to do.

Here someone is going to say that this is well understood and that developers deal with this all the time. I'm not saying anything else. I am saying that you can't estimate projection completion times with that variable because it is totally unpredictable.

Wow!

[endus]

Engineers think project managers and deadlines are a waste of time and a pain in the ass, while project managers think they are essential. Now that's what I call news! Whodathunkit!

This is business. Management wants to quantify everything to manage resources, manage spend, control cost, maximize profit, etc. It makes perfect sense at the same time that it doesn't really jive with how engineering works a lot of time. One thing for developers to keep in mind, though, is that *doing* something is never as important as *telling people* about how you did it. Metrics mean way more to the people who sign your paycheck than the code you write does and you should design your metrics accordingly.

The other component is PMs themselves. How many really good PMs have you worked with in your life? Grand total of 1 for me. Most PMs are people who don't really understand technology and have created a whole system of super-important metadata to "add value" to the process. When it's done properly a PM can help a lot, but mostly its just blustering and wasting everyone's time. These people want to protect their jobs, and their jobs are defined by timelines and metrics.

Re:Wow!

[dbIII]

Engineers think project managers and deadlines are a waste of time and a pain in the ass

It's because project managers used to be engineers instead of children with degrees in toga parties.

looking back

[Anonymous Coward]

I tell my clients that every project is a voyage into the unknown. The alternative is off-the-shelf software. We wouldn't be doing a project if we could just buy a solution.

I have found that experience is the best foundation for correct estimates. After a few dozen projects, it gets a lot easier. I guess that's not much comfort to people who are just getting started.

When I was new to the development life style, I found that the best result came from doing the best job of scoping that I could, estimate that, then double it and add 20%. I know this sounds like a joke, but it's not.

My Dilbert Moment

[Anonymous Coward]

I estimated 40 hours once for a complex vision system project. After completion manufacturing went from a 5-10% SHIPPING defect rate on a display to no defects. I completed it in about 55 hours. Our customer was very happy. Our accountants were not. I was called in and repeated asked why it took so long.. and I had to answer to an accountant. I had to repeat over and over again that "it took 55 hours to complete". There is no other answer. This was also in the early 90's when vision systems were in its infancy. Management could have probably examined my work and filed a few patents on it. Instead I was forced to waste my time with an accountant.

To be honest I probably programmed less than 40 hours on the project but I was also responsible for procuring the camera equipment and getting the mechanical details correct and I thought I should charge it to the project - there was no way to break parts of the project out - and I am not great at mechanical details so I am not surprised that I did not do that instantaneously like management would have liked.

How does somebody doing complex technical work answer to some pen pushing dimwit that is probably spending its time reading Facebook and exposing the company to fishing viruses???

I've lived Dilbert and it still makes me puke when I think about it all!!

Deadlines and Milestones instead of Estimates

[dave562]

I have found that the best middle ground is to work with the developers and project team to set deadlines and project milestones. While down to the tenth of an hour estimates are not necessary, there have to be goals to hit.

The best managers are going to let the developers provide their own estimates, and then calibrate the timelines accordingly. Some people are good at estimating time. Others are horrible at it. The project manager needs to know their team well enough to account for those factors.

The of thumb that I have always worked with is double the estimated time. Under promise and over deliver. This does lead to some grumbling up front, "It is going to take HOW long?!" But after successfully delivering ahead of time, enough times in a row, people come around.

The biggest challenge is keeping people honest. Some people have a hard time admitting that they are not going to make a deadline. It is important to give those people room to fail, so long as they are responsible about it. "This deadline has some flexibility, as long as you give me 48 hours notice that you are going to miss it. Don't come into my office the day I am expecting a deliverable and tell me that you need another week."

The other side of that is having to be a good manager, and push back on the business team to give the developers room to work. "We told you we would deliver it by X and we are still on track to deliver it by X. STFU you about your cranky client whose expectations you cannot manage despite us being explicitly clear with you about what our timelines are. And no, we are not going to add that extra feature that you promised them but failed to include in the scope."

In related news

[quantaman]

The city announced work on a new interchange involving the major arterial road running through the city, significant delays are expected while construction is underway.

When asked for a timeline on when the construction would be completed the lead engineer answered "Who knows? We generally underestimate these things by months or years so I might as well not bother."

Work is expected to commence sometime after they finish their current set of maintenance roadwork.

Good night, this was your 11 o'clock news at 11:23 because we needed a little more time to finish writing our stories.

Use it as a weapon

[trout007]

Use your estimate precision as a bargaining chip for things your need. Typically what impacts my estimate is requirement uncertainty. So give an estimate with an error +200/-10% error. If we could just finalize these particular requirements my estimate would improve. This way a manager will focus on getting the requirements fixed.

I was once given a "project" and I asked for the requirements document. They said there are no requirements so I happily said great I'm done designing!

Re:Estimates are for planning - not penalizing

[phantomfive]

You must be working in a shop that does the same thing over and over again, year after year.

No. I don't think many people work in that kind of shop.
No excuses. Don't use that as an excuse for bad estimates, it's something you can improve.

Estimates

[jgotts]

I've been programming for about 30 years and it took me at least 20 years to learn how to make good estimates. If you can't estimate how long something is going to take, then you probably haven't been doing it long enough. I don't care what your field is, or even if it's engineering or not.

Programming is just like any engineering endeavor. Be a professional and admit that not every job is a weekend hack. If you work somewhere where you have to lie about work taking less time than it takes, then quit that job and go work somewhere else.

One difference between a so-so programmer and a great programmer is someone who finishes his or her work when he or she says he or she will. There are other differences, such as communication skills or lack thereof, but having people be able to trust you is pretty key.

Another way of looking at is this. Programming cannot be rushed. That doesn't mean tell people it will be done when it's done! What it does mean is learn to estimate the worst case scenario. You will not be punished for writing a great piece of software and finishing it a few weeks early. You will be punished for saying it will be finished by a date assuming that nothing will go wrong at all and that you are some kind of programming genius, better than every other programmer out there.

Consider this example. I think I might be able to finish something in a week. The estimate I give is two weeks. Another programmer typically says a job of roughly the same magnitude will be done "in a few days." I have given myself plenty of time to do the job right, and extra time to fix bugs and possibly deliver the project early. The other programmer screwed up and it does take him a week. After a week, his work has tons of bugs. It goes out to production and every one of those bugs is embarassing and takes man days to resolve, rather than a few man hours. I've seen this scenario happen many times. Programmers fall into this trap of their own making. Maybe it has something to do with machismo. Stop pretending to be more manly or smarter than you are and be a professional.

Wasn't this the main point of "Agile"?

[hey!]

Find a compromise between predicting too much of the future and just managing a project by the seat of your pants; get into a rhythm where you check how good your estimations and learn to get better at them.

Of course you can't develop every project this way; I've used Agile and it's worked for me. I've used waterfall and it's worked for me too. You have to try to be sensible; you can't completely wall of other people's need to know when you'll accomplish certain things, nor can you build a solid plan based on pure speculation. You have to have an intelligent responsible way of dealing with future uncertainty, a plan to cut it down to size.

I've even had the good fortune at one point of winning a $750,000 grant to build a system for which no firm requirements had been established. It was kind of an uphill-flowing waterfall: we knew how long it would take us and how much it would cost but we had no firm idea of what we were supposed to build. If that sounds like a recipe for disaster, it was; but my team was *successful* and built a product which was still be used and supported over a decade after the grant finished.

What's missing from many programming estimates is honesty. It's a matter of ethics; you can't take people's money and say maybe someday you'll deliver something useful to them. People don't have unlimited time and money to accomplish all the things that need to be done in the world. It's an honor being entrusted with people's aspirations, and a serious responsibility. It's hard, even nerve-wracking, but you've got to care enough about the impact of your planning on other people to make the effort to do the very best job you can.

And what I've found is that if you do make the effort you can do a surprisingly good job of estimating a project if it's in an area and with technologies you're reasonably familiar with. If you look closely your specific predictions will often be way off, but if you care enough to be brutally honest the pleasant surprises tend to balance out the unpleasant ones.

T&M or Scoped Bid; There is no third option

[Anonymous Coward]

I run into this all the time.

The company I work for makes lump-sum-bids for design, construction, programming, and commissioning projects.
We define supplier scope, purchaser scope, explicit exceptions, cost, and schedule.
And we deliver it. The Clients are happy and we make a profit.

We also do T&M work: We offer excellent folks who will do--within the bounds of ethics and competence--whatever the Client asks.
And we deliver them. The Clients are happy and we make a profit.

Mixing the two is not a good idea. If you need to know the cost and schedule but haven't nailed down scope, (or can't take the time to do so) something is wrong. Either you want something for nothing or you're looking for a fall guy for the project you've already driven into chaos.
In the rare instances where I've tried to "help out a guy estimate something", it's come back to me as "We Agreed..."

There's no profit in that for me; and it pisses off the Client when they discover no, you can't have a pony.
There's just no reason to do it.

Re:Estimates are for planning - not penalizing

[Bender0x7D1]

If you penalize me for going over time and budget, my next estimate is 30 eons and 200 trillion USD. Watch me come in well inside the budget.

So we aren't talking a government project?

Re:Is this really a problem unique to devs??

[BoberFett]

I'm out of mod points, so I'll just say: Good post.

Need 2nd order estimations

[swan5566]

Give your estimate, but also give the variance on that estimate. If it's low, they will ask why. This will lead into the technical barriers and uncertainty discussions, which is really what the business types and PMs need to do and don't do enough of.

Estimate is the wrong word. Use "budget".

[strayant]

This is a matter of communication. These numbers should serve less as a prediction and more as an allotment. Instead of "what is the estimate for this task?" the question "what is the budget for this task?" is far more accurate and manages everyone properly.

Re:Estimates are for planning - not penalizing

[phantomfive]

Agreed. The problem is, they don't listen.

You are not responsible to meet any estimates you don't make. If your manager makes your estimate, let him meet it.

Update the estimate, honestly.

[kogut]

Any initial estimate will go to hell quickly. The important thing is good communication between managers and developers to mitigate the effects of slip and to continually update the estimate.

If the developers know by the 2nd week that they're already behind because difficult things were glossed over initially, then they need to feel comfortable telling the managers rather then pretending that they'll catch back up. And the managers need to understand this type of communication is beneficial and not just get angry about schedule slip, disincentivizing further communication.

Correspondingly a good manager will aggressively probe intermediate milestones.

Manager: "Feature X should be be done now. "

Developer: "Yes."

Manager, "Show me."

Developer: "OK! But we first need to update the front end to reflect the new data model."

Manager: "So Feature X is not done."

Developer, "Well, no."

The real problem

[sjames]

You can't have an estimate without knowing exactly what you will be doing. Finding out exactly what you are doing is also known as the design phase. It is the highest paying part of the project and can easily be 50 to 90% of the whole project. But they want the time estimate up front...

Once given, they will want to stick to the estimate as if it was carved on stone tablets by Moses.

If the project involves replacing of interfacing with a legacy system, or for that matter gathering requirements from people you don't know yet even estimating the design phase timeline can be a problem.

If it requires anything that can be described as experimental, you need to perform the experiments before you can commit to a timeline. The estimated time to do the experimentation will be even further out of the question if there is any possibility that one result could lead to an additional experiment.

Reality of Scotty's Merikles....

[TiggertheMad]

Kirk: How much refit time before we can take her out again? Scotty: Eight weeks, sir. But ye don't have eight weeks, so I'll do it for ye in two. Basically, we will be violating every safety standard in star-fleet, and there is a 20% everything will blow up completely leaving us stranded in deep space w/o power.

After which managers toss the "bad" estimates

[msobkow]

My experience has been that management comes to the developers for estimates. They provide those estimates to the end users. The end users bitch, whine, and complain that they need it to be done in half that time.

Management then comes back to the dev team and tells them they've agreed to get the project done in half the time that was estimated.

Then both management and the user community bitch when their "estimates/targets" aren't met, and who is blamed for the issue?

The developers.

The developers always are to blame for computer problems, never the bad specs, the conflicting specs, the unknown variables, the use of "new technology" that some vendor flim-flammed onto the department/team, or anything or anyone else.

Screw 'em. Now that I'm retired, I'll never have to give anything more than the most vague ballpark estimate of how long it will take me to do something ever again. Instead, on my pet project, I just bullet point some of the things I intend to work on next -- and even that is subject to change. The lack of stress and the freedom to live my life according to my own whims and needs has proven an invaluable source of improvement in my "quality of life."

What a shame I've never encountered a job that would let you do that.

Re:Is this really a problem unique to devs??

[Anonymous Coward]

If you know that not providing a margin will eventually lead to a manager making a decision that everything is your fault and you choose to not provide a margin anyway, then the situation really is your fault. You have the power to stop it from happening but choose to walk in front of that bus anyway only to blame the bus driver for it. Be a professional, use your excellent brain to fix problems not just in your discipline but your workplace also. You will prosper if you do. If you don't, well you already know how it's gonna go.

"Humans don’t want accuracy;they want reassu

[mdgreene]

Humans don’t want accuracy; they want reassurance. The Nobel laureate and retired Stanford University economist Kenneth Arrow did a tour of duty as a weather forecaster for the U.S. Air Force during World War II. Ordered to evaluate mathematical models for predicting the weather one month ahead, he found that they were worthless. Informed of that, his superiors sent back another order: “The Commanding General is well aware that the forecasts are no good. However, he needs them for planning purposes.” Source: JASON ZWEIG, WSJ http://www.wsj.com/articles/le...

I disagree

[dbIII]

I disagree - no budget is going to help with time dependant issues or bottlenecks that cannot be "bought off". The most famous and trivial example is bringing a child to term - not happening any faster with ten mothers. Seasonal conditions are another. Production capacity of suppliers is another. Availability of staff is another. I'd say it's only really a major factor when comparing with projects with inadequate budgets.
Run three shifts and some time problems go away but not all of them.

They are doing it wrong then

[aepervius]

"On the flip side we have so-called "agile" development teams who simply define a deliverable as whatever they've completed on delivery day. These developers rarely tell management until the 11th hour that what they're going to deliver is below spec. Agile development has its strengths, but this aspect is a giant weakness. The solution is not to eliminate schedules. It's to adapt them to changing conditions and be proactive about slippages."

The PO is supposed to be aware of this at least on the sprint level, and if necessary communicate with the client. Furthermore one of the tenet is that the team is responsible,m and responsibility is to warn that the spring goal may not be reached and draw consequence on it for next sprint goal (make them less big etc...). Which also means since you are delivering on the atomar level of a sprint, that it is known sprint after sprint how the evolution of the project is by the client. Now if you are speaking of telling on the 11th hour on a SPRINT level, well Yupi-doo. Compared to most project which tell goal will not be met at the 11th hour before delivery of the FULL project ?

Re:Is this really a problem unique to devs??

[kaiser423]

Great post. This then doubles with EVMS -- the new government way of creating estimates and schedules. You have to hack out the software development so fine that at the proposal or initial program startup phase, you better know every class and architectural detail, because you're not allowed to add anything without an expensive replan, but you're also not allowed to have non-specific tasks, or tasks that take longer than 44 days -- but that's the baseline task, so typically the software portion of it can't be longer than ~10-20 days. I do pretty good with getting the schedules in, but that's because I'm a PM that writes lots of code too, and does EE/RF work, so I can hack it in. But in a lot of other schedules from other people, it's really just GIGO -- garbage in, garbage out.

Re:Estimates are for planning - not penalizing

[Darinbob]

In the real world though, there's a whole lot of penalizing that gets done. Thus a whole lot of stress.

It's not actually all that hard!

[Tony Isaac]

It just takes a little training and practice. The trick is to divide a project into pieces (bullet points are sufficient) where each item takes two days or less to do. If a line item is estimated to take longer than two days, it needs to be broken down some more. At that level of granularity, it's possible for most programmers to make a reasonable guess as to how long that item will take.

Put another way, it's more important to count the line items, than to count the hours. The trick is to get the line items the right size.

Steve McConnell's book does a good job explaining how this works.

Re:Is this really a problem unique to devs??

[Darinbob]

But there are so many managers up the chain that take all these estimates as the truth. Before the project starts, before any data sheets have even been read, we're asked to come up with all tasks we think will be needed, then give an estimate for each task. This all gets put on a chart and managers stare at it. Then you'll be asked what percentage of time you can work on each one, and you'll say 50% maybe because they get really mad at you if you are honest and say 10%. Then the chart gets printed and put up on some walls.

The problem then is that the chart is not updated. The developers are too busy to figure out some goofy planning software so that they can budge their estimate and the managers don't always do it. Along the way half those tasks you thought were needed either vanish or are replaced by something else and some new ones are added (because now you have some datasheets and requirements, you thought of something you forget, etc). We open and close tickets that aren't on any project plan. The planning and actual work just don't coincide, they're done by different groups of people with completely different sets of motiviations.

I have been at one place though that managed to estimate things well and had some good planning. I honestly don't know how they did it, except that they moved slooowly. They weren't developing a brand new product from scratch (like I am now) but just adding incremental features, lots of time spent planning before starting by people who are engineers and not managers, a set of bug-fix releases were pre-planned, and there was a multi month back end for QA. They spent more time on documentation than some entire projects I've been on. And it was waterfall. But that was only for two years of my career.

Unknowns make estimates problematic

[egarland]

There are definitely times when estimates are appropriate in programming, but the more unknowns there are inside a project, the more you need to nail those unknowns down before estimating is worthwhile. Sometimes, prototyping should be done without estimates.

easy

[Tom]

But it's so easy to make a good estimate, takes less than 10 seconds:

Take your instinctive estimate.
Double it.
Increase units by one (if you think "hours", make it days. If you think "weeks" make it months, etc.)

So if you think it'll take 2-3 days, tell your manager it'll be ready in 4-6 weeks. Don't forget that in management school, they teach these fuckers to under-promise and over-deliver. He understands.

Wrong approach. Estimates aren't a problem.

[Qbertino]

Getting rid of estimates is the wrong approach - of course. What companies need is a clear product, pipeline and system strategy. Have that, and estimates are trivial.

If I know I have a set of servers and they all run system X and toolkit Y and management process Z and I'm hired to handle XYZ and we/I have optimised my skills and toolset for said chain because we've chose it as our strategy, I can give estimates that are precise to a margin of half an hour per week of project time on the drop of a hat.

However, give me deciders and/or co-workers who don't have a clue and make decisions way out of their league that I have to follow on, shitty testing and deployment, 10$/hour students making core system decisions, local servers I have to salvate from a junkpile and decisions on what system we're going to use in the next project based on what software the custmers drinking buddy had heard of and was rambling about after 7 beers half a year ago on their last pub tour and my estimates will be just as shitty as your decisions.

Leave me out of the loop until the very last moment and then barge into the door with some system I've only looked into for 15 minutes in the last 10 years and my estimates will be based on how true the vendors describe their product in the flyer. And we all know how true that is.

Estimates are fine if....

[mark-t]

... the functional requirements are clear, and do not change during the development lifecycle.

Except what usually happens is that a feature change will come along *AFTER* the design phase has already started, or else the requirements weren't unambiguous enough in the first place. Oh... and in the real world, at least in my experience, a developer isn't often in the position of being able to say "we can't do that", unless the developer is also very amenable to finding a different employer in the extremely near future.

This usually puts the programmer in the position of having to be a sort of prognosticator, and anticipating what the most likely types of design changes are going to be while doing development, and designing software that will be robust enough to accommodate such changes with only a modest increase in time spent, without losing any work on design that has already been completed. Sometimes, particularly with an experienced programmer, or one who really knows the people who are likely to make design change requests, and so may be able to predict what they are liable to really want beyond what was stated in the functional requirements documentation, this kind of forecasting is within the grasp of a human being to accomplish But it's never easy.

version control

[bingoUV]

Start using version control, and at least undoing becomes trivial.

Re:version control

[BronsCon]

You still end up rewriting code. Version control is great for many things, including rolling back to a previous point in time, but please tell me how to undo something I did last week without rolling back everything I've done since. I'd actually really like an answer to that, with regards to Git, if you've got one.

Re:version control

[bingoUV]

1. Using exquisite use of branching in git, it might be possible.

2. In your example, there was nothing "done since" the wrong paint so it is irrelevant.

Re:version control

[BronsCon]

My example was meant to highlight the fact that a single word change in the spec can mean having to redo work. No matter how easy it is to *undo* the work (in the case of programming, it's often much easier than in my analogy, but this is not always the case), the fact remains that the work must still be redone. Just because my analogy did not include having done work that needs to be kept, subsequent to having done work that needs to be redone, doesn't make my honest curiosity irrelevant; and your response seems to indicate that, save for having planned for that very occurance, it is indeed not possible (and even with planning, only maybe, if you're willing to create a new branch for nearly every commit... and you're lucky).

Honestly, I was hoping you'd tell me it was easy. This is one of those instances where I'd jump for joy while screaming at the top of my lungs "I WAS WRONG! I WAS WRONG! THANK YOU, LORD, I WAS WRONG!" Not that I have a use for it at the moment, but it's come up in the past and I'm sure it'll come up again in the future.

Re: version control

[BronsCon]

And if I don't want to use a GUI? Seems your steps should still be applicable to the CLI. That being said, you're rewriting history when doing that, and that is considered a huge no-no when using Git, especially when you have a team of developers. You'll have a local branch without the commit you just skipped over, but everyone else will still have that commit in their history, meaning they'll all still have the "bad" code. It's probably fine if you're the sole developer, but it's generally not advised to assume that will always be the case. Methinks I'll not follow your advise on this.

Oh, fuck me, this old story again?

[fair_n_hite_451]

This post gets repeated about once a year here.

News Flash Developer types -- you are never going to see this in your lifetime. Estimates are a fact of life. Deal with it.

If you are in an organization that has a problem with turning WAG estimates into hard deadlines and then bitching when the inevitable unfolding of the project changes the situation is a large way, then protect yourself by learning to speak to the project managers in their own language.

Define some rough "degrees of certainty" around your estimating, and guidlines to the degree of accuracy you feel.

Class "0" (when the project is in the approval stages) +200% / -50%
Class "1" (when the requirements have been documented) +100% / -25%
Class "2" (when the work packages have been built out) +50 / -10%

... or whatever your particular organization and SDLC model supports. And then, when asked to provide an estimate, communicate clearly as to which class of estimate you are providing. This isn't rocket science.

Re:Oh, fuck me, this old story again?

[gnupun]

Class "2" (when the work packages have been built out) +50 / -10%

I hope you realize the task at hand has not been done before in the same way as Scotty repairing the warp drive or someone clearing dry leaves from around their trees. This software task has not been done before. So how exactly do you calculate X in your formula:
X +50 / -10%
??

Dozens of posts in this slashdot thread about "It's easy. Just multiply your estimate X, by 2, or 3 or 4." Fine, but how exactly do you come up with X other than pulling it out of your ass?

Re:Oh, fuck me, this old story again?

[kmoser]

Dozens of posts in this slashdot thread about "It's easy. Just multiply your estimate X, by 2, or 3 or 4." Fine, but how exactly do you come up with X other than pulling it out of your ass?

Experience, my friend, experience. Unfortunately most programmers rack up lots of coding time but they never seem to get much better at estimating how long something will take. My educated guess is that programmers tend to be overly optimistic about their abilities and too eager to please. Believe it or not, my estimates (and I've made hundreds, if not thousands, over the years) tend to be quite accurate. But I realize I'm in the minority.

Re:Is this really a problem unique to devs??

[bughunter]

Oh, I agree, I could have prevented a metric fuckton of shit landing in my lap. I know that now. That's just how i learned it the hard way.

Now, any estimate I give includes plenty of margin. Like the top post says, poor managers get worst-case estimates, plus a healthy margin for the inevitable negotiation that will take place.

The same applies for cost estimates. I learned the hard way the first time I was asked to present an estimated cost to complete forced by an unexpected 16-week delay in critical long lead part from an overseas supplier. I made a diligent effort to present an accurate ETC to the customer. No margin, no padding, just my honest, well-documented estimate of the cost to complete the project.

I was expecting to be dealing with the engineers and project managers I'd been working with all along, who were competent technically and I got along with well. But instead, the customer (major aerospace prime contractor) sent in their best hard-ass negotiator who was an MBA with no understanding of the technical side.

Mr. Hard Ass refused to accept that I wasn't bullshitting him. And the engineers I got along with so well didn't do a thing to back me up. They just sat there looking uncomfortable. After two days of going over the schedule and estimate line by line, and me refusing to cut anything other than the slightest costs, Mr. Hard Ass went over my head to the CEO, who agreed to a 25% percent reduction in the estimate across the board. He just ate the cost.

I got dressed down hard for not padding my numbers, but he was decent enough to understand that the ultimate blame lied with the suppliers who waited until 4 weeks before their agreed delivery date to notify us they'd be 16 weeks late. And it was a lesson I will never forget.

Asian cultures are strongly hierarchical.

[Futurepower(R)]

"This is a cultural thing. Asian cultures are strongly hierarchical: you always agree with the guy above you. Never argue."

That's an important insight. I often talk about that with friends.

Summary: Poor management, dishonesty

[Futurepower(R)]

Interesting.

You said, "... poor program management, lack of requirements management, and often also marketing-driven decision-making."

Overall, that is poor management of technical projects. The biggest single problem? Dishonesty, on several levels.

The first step in improving management is to get everyone to understand that there is poor management.

Re:Simple methodology - Sharing your SOW

[kdiffily]

Any chance you are willing to share your SOW template?

Re:Simple methodology - Sharing your SOW

[BronsCon]

I may do so at some point in the future, since there seems to be some interest in it. I would want to run anything like that by an attorney first, since I'm fairly certain it does constitute legal advice.