Slashdot Mirror
How Fast is Your Turnaround Time?
petrus.burdigala writes "I work for a mid-sized commercial software company (~20 Mloc) and we are frequently challenged by our supervisors to get fixes around the clock. Overall, we manage to get a 'bullet-proof' patch in about 4-5 weeks (from coding->QA->Build/Packaging->shipment), which I consider not so bad. But the other day, we got an urgent request from our support team to come up with a decent fix in 48 hours. I think they're a tiny bit unrealistic. So I wanted to get feedback from my peers: are we doing that bad? It takes months for other software vendors to issue zero-day exploit fixes, are our customers being unreasonable?"
I work for a bank so we don't do box software, but our patches have to meet FTC standards and Federal bank standards.
It is uncommon, but not unheard of to have an 8 hour fix. In cases of customer data vulnerability, legislation has been made such that if we are aware of a problem, we have an automatic injunction against us continuing to do business unless the problem is resolved. So when we have a security flaw, our bank stops working untill it is fixed. So yeah 48 hours would have people fired for sure.
Compliance/security are the only two things that can spark a release with less than 72 hours notice though.
Yeah, your turn around time seems good and yes, the customer's request is beyond industry norm.
That might mean one of three things:
One: Customer is being foolishly optimistic.
Two: The entire industry is bad about turn around time, and can, if pushed improve it to 48 hours.
Three: Customer needs it really quick and is hoping to get it quicker by asking. They know 48 hours is well beyond the norm, but are hoping you can do it anyway, because the more time it is unpatched the more they are screwed. They know that if you don't ask, you can't get, so they are at least 'asking'.
Me, I think it is a combination of all three. Customer is being a bit optimistic, the industry is bad about turn around time, and also the customer knows it is a bit optimistic but is making the request anyway in hope you will provide amazingly good service.
excitingthingstodo.blogspot.com
At BSDi, the initial patch (which did have flaws, but it fixed the problem) for the f00f bug was same-day, I believe; might have been next-day, depending on where you're counting from. (Contrary to popular belief, this didn't violate any NDAs.) Now, that was an emergency patch -- it took a while to come up with a patch that fixed the bug without noticable ill side-effects.
We had a better patch later, but the initial emergency patch was VERY fast.
On the other hand, if the initial bug report is "Sometimes the program hangs, no, I don't know when. Maybe every week or two." -- well, that's gonna be hard. Exploits generally have the advantage that an exploit is by nature at least somewhat reproducible, and the hardest part is often getting a reproducer. I've had it take six hours to develop a usable reproducer, and three minutes to develop a patch.
Release time depends hugely on process and procedure. IMHO, an ideal procedure would have some kind of way to get a Temporary Patch out into the field ASAP when there's an exploit.
My blog: http://www.seebs.net/log/ --- My iPhone/iPad app: http://www.seebs.net/seebsfrac/
We generally get fixes for real bugs out within 24 hours, unless the problem is traceable to the OS, the only factor really out of our immediate control. Even then, we do a quick evaluation to see if we can replace the OS function. Over the years, we've replaced quite a few of them, but rarely within 24 hours.
But we know our code backwards and forwards; I wrote the majority of the current codebase myself, and I can generally get to within a few lines of the problem just by a bug's description... the rest is a matter of minutes and testing. This app is very large - comparable to Photoshop in terms of feature count - but it is also very stable after 15 years of whack-a-bug and a continuous drive to make the internal structure as orderly and regular as possible.
It is my observation that the more programmers you have involved, the slower your turnaround time (for everything from bugs to features) will be. Likewise the larger the entity, the slower it will generally move. Almost every layer of management and corporate compartmenting disease will contribute to slowing down the process.
For the apps that I use that I have had the experience of reporting bugs, it is my general experience that bugs often are never fixed at all. One browser, "Omniweb", truly my favorite in terms of features, has bugs that make it essentially unusable for me. Crashing, slowing, lockups and so on - really serious problems. I've reported them, they never were fixed, in fact the software was never updated. Eventually, I just went back to firefox. Then as Leopard came out, after years of doing nothing, they released a "Leopard version" in which, perhaps, I might find those bugfixes if I looked... but as I say, I have moved on and no longer have any enthusiasm for the product. Slow bug repair (or ignoring them) is synonymous with telling your customers you really don't care what kind of experience they have with your software.
Apple, with all their emphasis on customer experience, does this too. They've had bugs in hand for very long periods where they simply don't address them. If your bug isn't something they think will affect a lot of people, it isn't likely to be fixed. I've not yet purchased Leopard, preferring not to catch early-adopter syndrome bugs myself, but when I do, I would not be the least bit surprised to find you still can't refresh a remote share that's been changed by the remote OS; that the wifi differs hugely in compatibility between PPC and Intel hardware; that mail still hoses the sent mail box based on the return address; that shell fonts are poorly rendered; that shell ANSI compatibility is still broken; that the OS still provides locked-up beachballs at the most inconvenient moments; that the OS still puts the wrong things away on the HD when RAM gets tight, and consequently becomes massively unresponsive... Basically, Apple doesn't have good control of their OS, are unable to respond to bugs in a timely fashion, so much so that they triage out bugs based on report counts, and the common patter is that Apple provides a great customer experience. So while my own experience is that bug fixes are important and can be quick in turnaround, here's Apple showing us that you can make a complete thrash out of the entire bugfix issue and still come out smelling like roses. So is a few weeks too long? Probably not, if you have a good marketing department. :-)
I've fallen off your lawn, and I can't get up.
The problem is when the customer is being unreasonable, the "support" (or more likely sales) just agrees to everything they say and "sure, we'll fix it" because they don't a) know any better b) they wan't to sell, not take the conflict c) they're stupid and just passes this backward "fix this, NOW!"
Then you're going to have a bigger problem! It's the same thing in any kind of relationship, just bowing and scraping and always saying "it's my fault" is going to cause bigger problems in the future than just saying "nope, we're not gonna fix that. or "sure, well fix it, but not now, you'll get your patch when it's tested properly, in the meantime, do this instead"
The fact that the parent was moderated down just shows me that the arrogance, contempt, and stupidity in corporate America is alive and well - especially in IT. I wouldn't say its not contempt but you get what you pay for. I once worked for a small software company who would push out emergency patches (even if the issue was minor) within 24 hours if you had the $10,000 package including paid support. No questions asked. Heck, we'll fly one of our reps out there to install the patch himself.
If you had a single license and no paid support... Well... We might have a general update next month with a public patch. We might not. Have a nice day.
Of course when you sell software as a service then thats how it works.
As a side note, one customers feature request created a completely separate build just for that customer which was annoying to the programmers but since they paid good money for it, they got what they asked for. Although... I remember the programmers eventually including the features for everyone else as a optional package just to avoid that so in the end even the single client customers benefited.
"I am the king of the Romans, and am superior to rules of grammar!"
-Sigismund, Holy Roman Emperor (1368-1437)
The customer described a program they wanted (to run on an embedded system). I estimated 3-4 months. They asked for 30 days or less. I explained what they'd get if I banged it out that fast - something that would work most of the time and not lose too much data. They then explained that the program would save them over $1,000,000 a month. If it quit working, they quit saving money, but nothing else bad would happen.
So, I saluted and said I'd try really hard for 3 weeks for the first version, then about three months longer for a version that would work all the time. Which is what happened.
Do you know the impact on this customer of not having the fix that soon? Maybe it's worth it to them...