How Fast is Your Turnaround Time?

petrus.burdigala writes "I work for a mid-sized commercial software company (~20 Mloc) and we are frequently challenged by our supervisors to get fixes around the clock. Overall, we manage to get a 'bullet-proof' patch in about 4-5 weeks (from coding->QA->Build/Packaging->shipment), which I consider not so bad. But the other day, we got an urgent request from our support team to come up with a decent fix in 48 hours. I think they're a tiny bit unrealistic. So I wanted to get feedback from my peers: are we doing that bad? It takes months for other software vendors to issue zero-day exploit fixes, are our customers being unreasonable?"

Comments

Small Startup Experience...

[Black-Man]

You have to serve the client who is paying the bills - and we had a very vocal one (Nik*). We had a running joke about the release d'jour. But it wasn't a joke. We literally would push a new build to them every day which contained minor bug fixes. It was maddening! But no one had the balls to stand up to the 800lb gorilla, so the madness continued. As a side-note, they were acting as a beta tester and anyone in the software business knows what that can mean.

Re:Small Startup Experience...

[PitaBred]

Consider that week or so of waiting as a chance to learn a bit about business politics, your company's political hierarchy and start getting things changed. If they don't change, leave. You're the only reason they're staying afloat, if everything technical eventually falls back to you.

how long is a piece of string

[LiquidCoooled]

It depends upon the nature of the problem and the competency of the developers.
If you know enough of the code tree you can tell when first reproducing and examining the failure whether it is a one off mistake or a larger procedural fault.

Single instance stupid errors (doh! moments) can be rectified and put through testing fairly quickly, however if your initial examination uncovered a larger problem then obviously the process will take longer (if at all - consider workarounds).

If the original dev/test team has been replaced over time this becomes a more difficult issue and every bug must go through complete verification simply because the extent or ramifications of the code modification will not be known.

In some instances we have had fixes out of the door the same day an issue was noticed, in others months go by before a final fix is put in place.

Lack of information

[Wellington Grey]

But the other day, we got an urgent request from our support team to come up with a decent fix in 48 hours. I think they're a tiny bit unrealistic.

Well, we really can't answer that question with knowing how big the problem is. If it's an embarrassing typo on a dialog box, then 48 hours is reasonable. If it's a windows vista security patch, then 48 days would be unrealistic.

-Grey

Can't compare

[Sloppy]

It depends on what you're maintaining and how complicated it is. I've gotten fixes out in 2 or 3 minutes. That doesn't mean I'm fast and you're slow, though. "How fast is your turnaround?" is like "how long does it take to write a computer program?" It's hopelessly vague.

Re:Can't compare

[Shagg]

Exactly... "How long is a piece of string"?

In addition to all of the other comments about the scope of the problem, number of resources, etc, you also have to take into consideration what you're changing. Obviously there are huge differences between patching the avionics system on an airplane vs a banner ad on a website. I've given estimates anywhere from hours to months before. There's no such thing as "X is the right amount of time for a patch" without a lot more details.

One thing you can always do is try and work with the customer to make them aware of the issues. You can tell them that it's possible to get a patch out to them faster, but you will be skipping a lot of the QA in order to do so (depending on what flexibility you have with the standard company process). The risk of it failing would be theirs. If they're OK with that, then you might be able to expedite things. It all depends on what you're patching and how important it is to them.

Parent is right.

[iknownuttin]

It may just be me but I think that's why they are called "customers"

Sometimes, customers are unreasonable and if they are, they should be treated with respect and the problem explained to them. Yes, they may be incredulous, but if you hold your ground (if they're being unreasonable), treat them with respect, they will come around.

The fact that the parent was moderated down just shows me that the arrogance, contempt, and stupidity in corporate America is alive and well - especially in IT.

Re:Parent is right.

[einhverfr]

For security issues we usually have a 48-hr to 1-month turn around time. I think a turnaround time for urgent issues of more than a month is excessive. Minor issues get fixed based on a number of factors and the turnaround time varies from 24 hours to 3 months.

In case you are wondering why the floor on the security issue fixes is a little longer, we usually put the initial problem and the fix through extra review so we ensure we truly understand the problem and that the fix solves all likely related issues. Then we have to decide whether to do a patch or a maintenance release. This process adds additional time.

Having said this, it is impossible to know from the description whether the customer is being reasonable or not. I have had issues where I had to come up with a fix within 24 hours and other cases where the demand was unreasonable. Hope this helps.

1) What is the business impact of the bug?
2) What is the data integrity impact of the bug?

Re:Parent is right.

[pla]

Sometimes, customers are unreasonable

"Sometimes"? Heh... Good one.


and if they are,

"if"? Man, where do you come up with this stuff?


they should be treated with respect

Ahahahaahahhaaaa... Heh...


and the problem explained to them.

HAHAH[choke]
[gasp]
[snort]

Ahem... Please, stop, I can't take anymore.



The fact that the parent was moderated down just shows me that the arrogance, contempt, and stupidity in corporate America is alive and well - especially in IT.

Some people deserve contempt and our scorn.

They act as though we can save the world before dinner when they want something, and call us miserable worthless slacker bastards the next. They insist we fix their problem in 48 hours when they can't even describe the problem accurately enough to reproduce. They need us and beg us for help and resent every second of it. They treat us like disposeable/interchangeable cogs, then bemoan that we each have unique and difficult-to-replace skillsets.

You want to know why geeks look at most people with utter contempt? Because they spit on us first.

Re:Parent is right.

[pla]

You need to respect your customers, because if they went away, you'd be out of a job, it's that simple.

Technically true, but irrelevant. If cows went away, we couldn't have any more hamburgers. That doesn't mean we'd all starve to death, because we can eat other things. But you want know the funny part here?

We could do most of their jobs (perhaps with a bit of training). Not all of us, and not all of their jobs, but in general. They cannot, ever, learn our jobs. One of our surprisingly few actual skills, "problem domain reduction" (kudos to 19thNervousBreakdown for the term), most people simply can't learn, regardless of will or even intelligence. On the flip side of that, however, it means we can pretty much accomplish anything we try, from coding to plumbing to animal husbandry to stonemasonry.

Think, really think, about how many geeks you know who, during the tech crash half a decade ago, did just fine on a variety of completely unrelated-to-IT jobs. Personally, I did a stint in construction/carpentry, and produced some damned nice work (if I do say so myself) - with ZERO training beyond casual observation of standard proceedures. I don't say that to brag - Hell, I don't really consider it much to brag about - Just putting myself forward as an example. We can do their jobs. They can't do ours.


No, it's because you act like a self-important little shite who thinks they should be bowing on their knees and sucking your dick for every line of code you produced.

Not really, because I code for me. They just pay me for it. I'd do it in my spare time if I didn't do it as part of my employment. I stay up-to-date on the world of IT because I find it fascinating, not because someone pays me to freshen my skill-set or because the terms of my state-permission-to-practice requires some pathetically low number of hours of study per year.

Anyway, all of the above said, I do try my best to remain humble and polite to most people, geek or not - And for the most part, I succeed. I very much doubt most people who know me would call me a "self-important little shite". But still, the constant jabs come anyway - From "complimenting" me on my skills the same way you would compliment medusa on her hairstyle, to barely-tempered insults only blunted by the fact that we've usurped the language no differently than blacks using the word "nigga". "Dude, you're such a geek!" "Yeah, thanks". People look at us as freaks for what we can do, and you tell us to respect them?

Unrealistic

[gweihir]

With a little simplification, you have four parameters: Difficulty, quality, speed and available resources. Whenever you fix three, the fourth follows (with some unvertainity). It is well known, that there is a limit on how much you can improve the speed with more resources. So there is an upper limit on speed already. The second problem that difficulty is unknown when starting such a task. There is no fix for that.

So if these people fix speed and available resources, and difficulty is fixed by the task, quality is determined by these factors. Period. There is no arguing with hard, real limits. If they do also want to specify the result quality, then they have to leave speed open. Again, there is no way around that limitation. In fact they should be happy if the team manages the required quality at all in reasonable time. Not all teams do.

Maybe thisn will be an argumentation that is inderstandable for people with a business background. Engineers should already know this.

Software engineering is engineering. Engineering tasks in general have minimal time requirements. Look at structural engineering: Nobody would try to design and build a full-custom bridge in a week. Instead it takes up to a decade, depending on difficulty. And you can generally not speed things up by increasing the team size.

Difference between Patch and new version

[techpawn]

A patch (IMHO) is a bug fit to existing code. Given the resources we should be able to get a PATCH out in a week. However, if you need a new version of the software to address the issue. Then we're talking longer development/testing/QA times if which case 4-5 weeks would not be unreasonable. Bugs should be fixed as soon as they are spotted. If their is need for a whole rewrite then you may want to talk to your staff

It depends...

[gillbates]

48 hours is tad bit tight. However, I've turned things around in a similar amount of time.

But, the old adage is true: you get what you pay for:

• Granted, 48 hours is tight, but possible if you know the root cause, *and* the customer is willing to forego the usual QA process before delivery. It doesn't mean that you don't do QA, but rather, that you do it later and patch the patch, if necessary. In most corporations, this means that if the customer doesn't complain, QA doesn't get done for these "extra-special" releases.
• Four to five weeks for 20Mloc is probably about average. As a general rule, a good team will average about 1 week per department the fix has to go through: field team -> engineering (fix) -> department review -> QA review -> ship. However, in some organizations, particularly the smaller ones, defects can get turned around in 1 to 2 weeks, especially if the customer works directly with the engineer/developer, and the developer is authorized to make releases to the customer. Be aware that your customers may have dealt with such organizations in the past.
• 20Mloc is not really that big, provided that the project and code was well-organized from the beginning and the original developers are still on staff. But, if this is not the case, or you have a large developer base, where no one is actually an expert on the systems, or subsystems, you can add about a 50% overhead to your turnaround time. If the original developers are no longer there, add 100%.
• If you don't know the root cause of the problem, you can't promise anything, and you need to inform the customer of this - you simply can't make any guarantees because you don't know the scope of the problem. Once the root cause has been identified, a 48 hour deadline is still tight, but is long enough to allow the key developer to build and do some rudimentary testing of the fix. Should the customer choose to accept it at this point (and you *must* make the point that it is their choice), they must be willing to forego the normal QA process, and should sign a statement of understanding to that effect.

When faced with unreasonable deadlines in the past, I usually voice my opinion once, and just do the best I can. Your higher-ups are probably already quite stressed at this point, and adding stress to the situation doesn't do anything for your career or theirs. Rather, if you make the point that you're doing the impossible, you might just have a little bit more bargaining power when it comes time for raises.

But on the flip side of the coin, if management doesn't learn, and you find yourself constantly asked to do the impossible, you might want to consider employment elsewhere...

Re:Web based

[Gregb05]

*15 minutes.
It's bad enough that they directly state they're not really testing patches with a 15 minute turnaround, but the fact that they're making mistakes that can be fixed in 15 minutes speaks loudly as well.

That works both ways.

[khasim]

Maybe the customer is being unreasonable.

Maybe the developer is being unreasonable.

It isn't possible to determine which from either person's viewpoint. You will ALWAYS think that you're right and that the other person is unreasonable.

Which is why you need criteria for bug escalation. Generating an incorrect response on 1 type of transaction for 1 specific scenario that may pop up once a year is far less important than a bug that corrupts the entire database.

And if your product is considered "mission critical", I would expect a data corruption bug to be fixed within 24 hours. Even if it is nothing more than rolling back the recent patches and re-issuing the previous version.

Re:That works both ways.

[Anonymous Coward]

The battle between sales and support has raged for years. Forget sunni v. shite, red v. blue, Kobe v. Shaq. Sales v. support is the ultimate battle. Sales tells the customer what they want to hear. Support tells the customer the truth. I've worked both sides of the aisle. As a sales rep I try to be honest about what I'm selling. The software I sell does not support P2P. We don't support it. Need a server. I let the customer know. Some of the other reps kind of gloss over it really fast just to make the sale. Then the customer calls back to support upset. I'd rather avoid that by being honest up front.

Now in this case the customer may be being unreasonable with the 48 hrs demand. But it all depends on the issue. There have been times when my company has been able to get out a quick fix within 24 hrs and other times when it has taken 3 weeks. It all depends on what the issue is, and what the solution is. There has to be some kind of middle ground that could have been reached.

Re:That works both ways.

[AdamWeeden]

Which reminds me of the old business adage that far too many businesses and customers forget these days. "It is only a good deal when both sides think so."

Re:That works both ways.

[gstoddart]

Bullshit. A corrupted database of inventory of toilet paper can be far less dangerous than the one type of transaction for one specific scenario that may pop up once a year when that one transaction decides the rod position in a nuclear reactor.
In other words, you have to know the importance to the customer.

You know what? To the customer, they're all nuclear rods. They don't care about your problems, they care about theirs. And those problems are critical to them.

You wanna know how escalations work, find out how important the customer is to the company. If you can get someone to unreasonably escalate your call because you're a big contract, you can get a lot of attention. If you're a small customer, or, if the vendor has balls, you might have to wait. Because what you're asking for isn't that big of a deal to us.

In my experience, if you can convince a VP this is a show stopper, you can get a lot of screaming -- it's like being in the army that way. If you can't get a VP/VIP on board, you get to stand in line.

Your priority to us is in proportion to related revenues to you. :-P That's how corporations work.

Unfortunately, a fix can't always be done in the timeframe demanded. Sometimes, you have to push back. Sometimes, you don't get a choice. :-P

Cheers

Re:Web based

[AuMatar]

Even if the bug is obvious, it doesn't mean that your fix

1)Works
2)Works correctly for all corner cases
3)Does not have unintended side effects
4)Didn't accidently include some other changes you were working on before, which are not ready for production.

You still need to QA. Attitudes like yours are why the quality of software is so poor.

Re:four or five WEEKS?

[pclminion]

Exploits should be a high concern for any company

Which is exactly why exploit fixes must go through STANDARD QUALITY CONTROL. What the fuck good have you done if by fixing one exploit you introduce ten bugs and two new exploits? I don't care how urgently the customer needs it. I'm not going to give them something I haven't tested. That's insane. If they don't like it they can shop elsewhere.

It's about risk

[PIPBoy3000]

I work for a large healthcare organization and typically have very fast turn-around times (bugs often get squished within an hour). For clinical applications and other core applications, though, we're much more methodical and careful.

I often explain to the user that I can push changes out immediately, but it introduces certain risks. I then detail the risks they may face, and that if they say to go ahead anyway, at least they'll be aware of what might happen.

Re:SLASHDOT SUX0RZ

[renegadesx]

At the risk of getting modded "offtopic" I will say what everyone is thinking and take a hit for the team


IS THERE ANY WAY TO BAN THIS ASSHOLE!!!! (pardon the little pun I threw in)

Goatse was funny 10 years ago but its really stale.