Slashdot Mirror
How Fast is Your Turnaround Time?
petrus.burdigala writes "I work for a mid-sized commercial software company (~20 Mloc) and we are frequently challenged by our supervisors to get fixes around the clock. Overall, we manage to get a 'bullet-proof' patch in about 4-5 weeks (from coding->QA->Build/Packaging->shipment), which I consider not so bad. But the other day, we got an urgent request from our support team to come up with a decent fix in 48 hours. I think they're a tiny bit unrealistic. So I wanted to get feedback from my peers: are we doing that bad? It takes months for other software vendors to issue zero-day exploit fixes, are our customers being unreasonable?"
It may just be me but I think that's why they are called "customers"
Excuse me while I gather the virgin sacrifice and assemble the pentagram required to solve your problem
How much of that 48 hour deadline did you waste reading /.
Get back to work!
You have to serve the client who is paying the bills - and we had a very vocal one (Nik*). We had a running joke about the release d'jour. But it wasn't a joke. We literally would push a new build to them every day which contained minor bug fixes. It was maddening! But no one had the balls to stand up to the 800lb gorilla, so the madness continued. As a side-note, they were acting as a beta tester and anyone in the software business knows what that can mean.
3 hours right here. Maybe you just need to rub your hands in some comcast coax for extra speed (k thx lame comcast commercial ftw!)
If an officer ever threatens to taze you, say you have a pacemaker.
What was that exploit again?
You can't talk about Wikipedia's flaws on Wikipedia
For high priority bug fixes, it usually takes 1 to 2 weeks to get a patch out once we determine that a patch is needed.
ÕÕ
It depends upon the nature of the problem and the competency of the developers.
If you know enough of the code tree you can tell when first reproducing and examining the failure whether it is a one off mistake or a larger procedural fault.
Single instance stupid errors (doh! moments) can be rectified and put through testing fairly quickly, however if your initial examination uncovered a larger problem then obviously the process will take longer (if at all - consider workarounds).
If the original dev/test team has been replaced over time this becomes a more difficult issue and every bug must go through complete verification simply because the extent or ramifications of the code modification will not be known.
In some instances we have had fixes out of the door the same day an issue was noticed, in others months go by before a final fix is put in place.
liqbase
I can understand a week, but honestly...if you're leaving your customers vulnerable for over a month, they might start looking elsewhere
Exploits should be a high concern for any company
of the issue. My team has had to get things out the door in 4-6 hours before. Just make sure you have people that have intricate knowledge of the app so that you can properly scope any changes, which allows you to perform good QA.
Regards,
Website Hosting
If a bug in our widely used web application is discovered at 12:00 I am often expected to put a fix on production by the time I leave at 6:00.
That said, I would hardly call our patches bullet-proof, though we usually manage to not screw anything up.
My company sells web based software (hosted). I have found, fixed and pushed out to 30 servers in as little as 15 minutes. I love our system.
I work for a bank so we don't do box software, but our patches have to meet FTC standards and Federal bank standards.
It is uncommon, but not unheard of to have an 8 hour fix. In cases of customer data vulnerability, legislation has been made such that if we are aware of a problem, we have an automatic injunction against us continuing to do business unless the problem is resolved. So when we have a security flaw, our bank stops working untill it is fixed. So yeah 48 hours would have people fired for sure.
Compliance/security are the only two things that can spark a release with less than 72 hours notice though.
But the other day, we got an urgent request from our support team to come up with a decent fix in 48 hours. I think they're a tiny bit unrealistic.
Well, we really can't answer that question with knowing how big the problem is. If it's an embarrassing typo on a dialog box, then 48 hours is reasonable. If it's a windows vista security patch, then 48 days would be unrealistic.
-Grey
Silver Clipboard: Time Management Tips
It depends on what you're maintaining and how complicated it is. I've gotten fixes out in 2 or 3 minutes. That doesn't mean I'm fast and you're slow, though. "How fast is your turnaround?" is like "how long does it take to write a computer program?" It's hopelessly vague.
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
The better engineered the project is, the faster development will be. The worse engineered and less centralized, the harder it will be. Also, the larger the size of the project, the more time it will take. Also the size of the company adds extra time as well; smaller shops can fudge on steps to get a fix out but larger shops have alot more customers to support and therefore can't fudge on QA.
This is my sig. There are many like it but this one is mine.
Yeah, your turn around time seems good and yes, the customer's request is beyond industry norm.
That might mean one of three things:
One: Customer is being foolishly optimistic.
Two: The entire industry is bad about turn around time, and can, if pushed improve it to 48 hours.
Three: Customer needs it really quick and is hoping to get it quicker by asking. They know 48 hours is well beyond the norm, but are hoping you can do it anyway, because the more time it is unpatched the more they are screwed. They know that if you don't ask, you can't get, so they are at least 'asking'.
Me, I think it is a combination of all three. Customer is being a bit optimistic, the industry is bad about turn around time, and also the customer knows it is a bit optimistic but is making the request anyway in hope you will provide amazingly good service.
excitingthingstodo.blogspot.com
Sometimes, customers are unreasonable and if they are, they should be treated with respect and the problem explained to them. Yes, they may be incredulous, but if you hold your ground (if they're being unreasonable), treat them with respect, they will come around.
The fact that the parent was moderated down just shows me that the arrogance, contempt, and stupidity in corporate America is alive and well - especially in IT.
I prefer Flambe as apposed flamebait.
With a little simplification, you have four parameters: Difficulty, quality, speed and available resources. Whenever you fix three, the fourth follows (with some unvertainity). It is well known, that there is a limit on how much you can improve the speed with more resources. So there is an upper limit on speed already. The second problem that difficulty is unknown when starting such a task. There is no fix for that.
So if these people fix speed and available resources, and difficulty is fixed by the task, quality is determined by these factors. Period. There is no arguing with hard, real limits. If they do also want to specify the result quality, then they have to leave speed open. Again, there is no way around that limitation. In fact they should be happy if the team manages the required quality at all in reasonable time. Not all teams do.
Maybe thisn will be an argumentation that is inderstandable for people with a business background. Engineers should already know this.
Software engineering is engineering. Engineering tasks in general have minimal time requirements. Look at structural engineering: Nobody would try to design and build a full-custom bridge in a week. Instead it takes up to a decade, depending on difficulty. And you can generally not speed things up by increasing the team size.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Overall, we manage to get a 'bullet-proof' patch in about 4-5 weeks (from coding->QA->Build/Packaging->shipment)
Not unreasonable, depending on the size of your release. (How many modules and how many LOC you're changing, the number of change requests or bug reports in the build).
But the other day, we got an urgent request from our support team to come up with a decent fix in 48 hours. I think they're a tiny bit unrealistic.
I think they're smoking crack.
So I wanted to get feedback from my peers: are we doing that bad?
With your regular release schedule, I don't think so.
are our customers being unreasonable?
Yes. That's what they do. If they want a crash development program to get this "patch" out the door that fast, they seriously risk software which does nothing but crash. Really, if they want it that bad, they run the risk of getting it that bad.
You have to ask yourself and your "support team" (sounds more like marketing to me): "Do we wish to ruin a perfectly good reputation for quality and reliability in one hurry-up bashfest followed by weeks of agonizing on-line debugging?" Really, advocate any kind of work-around and risk mitigation response before being pushed into an overly-hasty release that will linger on your reputation like a dead skunk.
Welcome to the Panopticon. Used to be a prison, now it's your home.
Seriously... but only once.
:P).
We were on-site at the customer, and we were involved in a shootout to see who would get a major contract.
We were "pre-demoing" to a bigwig, and it turned out we had an incorrect concept of one of the requirements. Hacked out a fix in 20 minutes.
Needless to say, the customer was impressed (as were my bosses
For production level code, though, I'd never do that, but it was a do-or-die sort of thing.
Oh, and we won the shootout (it wasn't even close, according to some of the guys who scored it) and the contract.
General Relativity: Space-time tells matter where to go; Matter tells space-time what shape to be.
A patch (IMHO) is a bug fit to existing code. Given the resources we should be able to get a PATCH out in a week. However, if you need a new version of the software to address the issue. Then we're talking longer development/testing/QA times if which case 4-5 weeks would not be unreasonable. Bugs should be fixed as soon as they are spotted. If their is need for a whole rewrite then you may want to talk to your staff
Ask not what you can do for your country. Ask what your country did to you
If your revision controls system is in a constant state of flux and you break daily builds more often then the US breaks treaties. Then yeah you can expect to have some problems getting a patch out to a customer.
On the other hand. Most simple bug I've dealt with take less then a day. More complicated ones will take 2. Anything taking more then 2 days is usually a new feature and not a bug.
I know I'm going to end up baiting some developers, but I work for a specialized ASP and see a ton of third party software from a perspective few get...
Normally, the smaller the company the more agile. No surprise. They also get patches out faster too. Also no surprise.
When we look at vendors of equal size, the ones who are really quick at sending out patches are in that situation because their software is more buggy, and they have a *lot* of practice. It never fails.
In response to your question, I would suggest that you should look more at the frequency of patches and less at the duration. Sure, it might not be as fast as your support group wants, but if you start reflexivly sending out patches every time someone yells, then your overall product will suffer since you can't possibly do the proper QA to ensure THAT patch you just whipped up doesn't break something else.
That brings me to the age old choice:
Pick 2 of the following:
Speed
Quality
Cost
how hot she is and how much I've had to drink.....
:P
Ok, who am I kidding, this is slashdot, it depends on how good the porn is
Monstar L
How much time do you spend on TPS reports?
The last time I did one I forgot the cover page and my 7 bosses all bugged me about it.
What was the issue?
Some things are trivial and fairly obvious mistakes, that are often easy to fix without breaking anything, in which case it's not too hard to get a patch out quickly. Format string problems for instance, shouldn't be too hard to correct.
Often a little extra validation can correct a problem too, just put in a check for a value being zero before doing a division etc. Things like this are also easy to test, and don't break anything else.
Longer turnarounds arise when the issue is more of a design flaw than a simple typo, or when your patch includes more than just a fix for the reported issue. I think security patches should only ever fix the issue at hand, with any feature updates available separately. I don't want the process of patching my system to introduce new features that could bring with them new bugs.
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
Ok, the name might suck, but the company I work at follows the Extreme Programming practice, a kind of agile programming. I have only worked there a few months, and had never herd of XP before, but am now converted. We work in pairs, which instantly adds a whole testing level. Deployments of code are done once every week, but sometimes more in an emergency. We write code test first, then run a build on our machines, then we upload it to a test environment where automatic tests are run. Finally on passing that, it moves to a stage environment where humans test the code, when they are happy a version number is noted, and that is uploaded to live. This means it can take a day for some code to be written, tested and deployed if required. It also means there is continual development, different departments can work on different versions, and then there is a weekly deployment of the latest stable code. It is a very interesting practice, and seems strange at first, but I would highly recommend it for certain types of companies. The company I work for took a few years to convert, and it was slow at first, but now it is an expert and even helps train other companies. It also builds its business upon being one of the quickest responders for code in our region.
In the case of a zero-day exploit, 48 hours is an entirely reasonable expectation. As far as I can tell, it is far from the norm, however. For less egregious flaws, feature upgrades, or normal bugs, a slower turnaround would be perfectly reasonable (and what I'd expect). But for a zero day exploit... Well, I firmly believe software companies need to be held more accountable for those by their customers. If that's what your customers are doing, then good for them.
At BSDi, the initial patch (which did have flaws, but it fixed the problem) for the f00f bug was same-day, I believe; might have been next-day, depending on where you're counting from. (Contrary to popular belief, this didn't violate any NDAs.) Now, that was an emergency patch -- it took a while to come up with a patch that fixed the bug without noticable ill side-effects.
We had a better patch later, but the initial emergency patch was VERY fast.
On the other hand, if the initial bug report is "Sometimes the program hangs, no, I don't know when. Maybe every week or two." -- well, that's gonna be hard. Exploits generally have the advantage that an exploit is by nature at least somewhat reproducible, and the hardest part is often getting a reproducer. I've had it take six hours to develop a usable reproducer, and three minutes to develop a patch.
Release time depends hugely on process and procedure. IMHO, an ideal procedure would have some kind of way to get a Temporary Patch out into the field ASAP when there's an exploit.
My blog: http://www.seebs.net/log/ --- My iPhone/iPad app: http://www.seebs.net/seebsfrac/
Just remember - the cuntstomer is always right, even when expecting the impossible.
D'oh, it would be helpful to know which kind of patch we are talking about. Is that security? If so, how critical? Is that a theoretical case, or a gaping hole that is a dead giveaway to any script kiddie? Is the problem just annoying or mission-critical? Is that going to be used in a network? As a server service? On which platform? If it is a new feature, does it require re-engineering of the code base, or is it a drop-in feature you can be over with in half an hour?
Most importantly, is your code base well-architectured, or is it a filthy can of worms you cannot possibly modify without breaking something?
In my (tiny-sized) company, if the boss or other users of our in-house software need new features or a bug fixed, I have the habit of listening first to requirements and estimating time-to-delivery later, and I have a hunch most developers do just that. Sometimes it's half an hour, sometimes it takes three months. Last estimate I gave was "anything between two weeks and six months", when the boss gave me some very vague description of the next package we are to develop.
Victims of 9/11: <3000. Traffic in the US: >30,000/y
48 hours is tad bit tight. However, I've turned things around in a similar amount of time.
But, the old adage is true: you get what you pay for:
When faced with unreasonable deadlines in the past, I usually voice my opinion once, and just do the best I can. Your higher-ups are probably already quite stressed at this point, and adding stress to the situation doesn't do anything for your career or theirs. Rather, if you make the point that you're doing the impossible, you might just have a little bit more bargaining power when it comes time for raises.
But on the flip side of the coin, if management doesn't learn, and you find yourself constantly asked to do the impossible, you might want to consider employment elsewhere...
The society for a thought-free internet welcomes you.
In general, the company I work for goes from end of development to GM in about a month, for boxed CD software that ain't bad. That assumes QA has been testing right through development. But it's going to vary widely based on the industry and size of the product.
I blogged about the opposite issue this morning.
http://www.rogue-development.com/blog/2007/11/i-love-my-job.html
Essentially it comes down to choosing a smart policy and sticking with it. If your company's policy is "Spend 2 weeks QA on every release." and you don't spend that two weeks, expect that release to suck.
My company can patch bugs within the same day the product's released. Never gone more than two days between a customer's bug report and a fix.
But we don't let bugs slip by. I know since I'm on the QA team. Never a bug. Never once... :P
"If I were to ask you a hypothetical question, what would you like it to be about?"
If you are talking about serious security vulnerabilities with-out a reasonable work around, then the customer has a reasonable expectation that you literally work around the clock to fix the issue.
If its a "normal" bug then I think several weeks or months is more the industry standard.
Think Deeply.
> are our customers being unreasonable
Not for security patches no. If the customer is being a pain, push it without QA and tell them they're welcome to take their chances outside the terms of any support contract. Then tell them how long your unit tests and such take to run and give them a realistic time for a supported patch.
Most PITA customers will opt to wait rather than risk downtime or data loss.
Let alone the importance of the problem.
That is why I am having trouble giving you a fixed number. I have been involved in fixes done in days and ones that take months. Context.
The real questions is
Does your management know how to prioritize tasks?
Then, do they know how who to put to to each task?
Finally, do they know how to accurately judge if the work is done correctly?
The problem I have always encountered is that a needed fix gets priority at first, then pet projects somehow get in there, and the deadline moves. Unless of course those affected have more pull than those assigning priority. I always found it amusing that our priorities are set by the people with most pet projects, wolves guarding the hen house.
* Winners compare their achievements to their goals, losers compare theirs to that of others.
You really have to supply some more detail to get any useful answer. And what is ~20 Mloc? About 20 million locations?
/.? Do you make internet-facing apps, and an urgent request means your customers just formed a spamming bot-net? Are in the medical/health care field, and an urgent request means folks might die?
What kind of software? What classifies an urgent request? Do you make games, and an urgent request means your bug just made front page
I think a better question is, how do you classify bugs? How do you make that trade-off between fixing a bug ASAP and taking the time to make sure the bug fix is done right?
Who is involved in the decision process? Is it just the technical & regulatory folks? Do you pull in business folks to help gage customer impact? Do you pull in sales and support to see if they can push a work around before the final fix is ready?
Those are all better questions than, "How fast do you do this task of unspecified scope."
I love the parent's subject-line analogy.
I'd add, it depends on product, the complexity of the codebase, the extensibility, modularity, readability, and extensibility of the codebase (eg, if it's highly modular it's easier to test a fix that's limited to the module/plugin)
I'd suggest that weeks sounds too long for an in the wild update without a security patch - or published workaround limiting your exposure. (eg, "use this method to restrict the IPs that can access it to trusted ones.") But that isn't me saying you're developing too slow, it's me saying that if it's going to take you that long you need to either find alternate solutions or create a architecture that allows you to quickly make access-limiting patches and layered security.
Actually, I'd make that more broad - if they want faster response to patches, what they need to do is to invest a lot on a highly modular, pluggable architecture so you can MAKE rapid changes. It's really a question of how much investment they want to make.
We routinely do same day fixes to certain kinds of things... but certainly the complex things take longer. And I think we're pretty unusual in that regard.
Looking for freelance Actionscript (Flash/Flex) or ColdFusion work and/or freelance developers. Email me, put Slashdot
You can have it fast.
You can have it cheap.
You can have it reliable.
If they want it in 48 hours, you should explain what that would cost.
If you want to streamline your process for patch deployment, I highly suggest you apply some six sigma or equivlant business process improvement strategy to it. Asking Slashdot is going to be counter productive. Your application is very different from anyone else's. Industry standard turn around time for software for a toaster is not going to be useful to you.
In other words, either you are working effeciently, or you aren't. It's that simple. If you aren't, fix the problems with your process. If you are, charge the customer what it would cost you to meet their needs.
If a customer has encountered a serious issue and is unable to operate, getting a hot fix out ASAP to them is sensible. Turn around time might be in the region of 48 hours. By hot fix I mean fixing the specific issue only compared to the state of the source when the last release was shipped - aka the maintenance branch for the last release.
In terms of period between each formal release; that will depend, but we try to have complete iterations (release or not) every three weeks. Ideally it should be two weeks, but we don't seem to do it on that time frame. Basically its one week of planning and design, two weeks of implementation.
Regardless, there is no 'correct' way - only the way that provides best value to the customer. If the customer can't use your software thats a *bad thing* - and you better fix it or they won't be your customer much longer.
I would suspect this largely depends on the kind of application you're developing.
Major problems in "cool blog software", for example, aren't really a huge issue. If it takes them a few weeks to be solved, then poor bloggers will be without some magic-pixie-dust feature for a bit.
In the telecomm world, though, customers expect a root cause for a "critical" defect in less than 24 hours (and there's a definition for critical, although I won't get into that here). My company actually writes that into our support contracts, so we need to root cause in 24 hours in a very real and legally binding way (although we do not contractually need to provide a fix in that time; sometimes we do, sometimes we just give a workaround).
Financial customers are even less forgiving.
A critical problem in, say, the fire control on an assault aircraft, is expected to never exist in the first place.
Our running joke used to be:
Marketing: We need it real bad!
Engineering: How bad do you need it?
Marketing: <puzzled look>
Engineering: Careful what you wish for... OK, Ops. Ship it!
I've had situations with customers who require a fix as soon as possible, because if the system is down they are losing money. When this situation occurs, we have two goals in mind:
(1) Get the customer up and running again as fast as possible. This is as often as not some sort of workaround that is not pretty, nor is it permanent, but it works. The workaround does get thorough testing (impossible within the time frame) but the customer is aware of this and willing to accept the risks.
(2) Get the customer a proper, version controlled, patch that they can install to fix the problem permanently. This can take weeks, most of that time being testing. If the customer is insistent we will ship them the proper patch before it is fully tested (again, making them aware of the risks) and continue testing so that we can send the customer some warm and fuzzy news later on (or, if we find a problem, another patch).
Life is like a web application. Sometime you need cookies just to get by.
Show stoppers get immediate attention; whatever it takes. People are losing money because they're DOWN. Fix it now.
Criticals get next attention when show stoppers are out. 48 hours, depending on interdependencies and QA needed to make it work; it's not part of an official stable code tree until later.
Minors are in the next stable branch release; every whatever you can handle.
Nigglies are changed when the stable branch releases.
Don't deviate from your policy, and make sure the sales people KNOW AND UNDERSTAND what this indicates and implies. No exceptions; see above.
---- Teach Peace. It's Cheaper Than War.
Depends what chair I'm sitting on, and whether the armrests hit the desk on the way round.
C-x C-s C-x k
I made them believe it was a hardware problem!
Engineering is the art of compromise.
It depends on the level of comfort you need to have with your release, and how quickly you can get there.
I also work at a mid-sized company in the enterprise software business. We do a lot of automated testing. For an urgent, customer blocking issue, I could potentially:
1) get really lucky and diagnose the problem instantaneously. Realistically, it will take me at least one hour to properly diagnose any serious bug that escaped our qa process.
2) Make a fix. Assuming this is a fairly trivial fix, this can be done in minutes.
3) Submit fix to automated testing. Autotest turns around in about one hour.
4) Make a patch. This takes about 2 hours, as it requires an autobuild of the whole project plus some version control stuff. It can be done in parallel with the automated testing if the bug is sufficiently urgent. It might be conceivable that we could bypass our standard patch mechanism to speed things up, but we've never had a sufficiently urgent bug to do this in my knowledge.
So we can turn around a patch in about 3 hours (plus fix implementation time) if it was nightmarishly urgent. A more realistic turn around would be next day, which would allow time for us to run a manual qa pass, and to do a more thorough job of documenting and testing the bug (which in the given scenario would take place after shipping the fix).
What makes this at all possible is the high level of confidence we have in our automated testing. Our auto-tests cover our application surface really, really well, so passing auto test will give you a pretty high comfort level in giving something to a customer.
"Who is the Journal of Quantum Physics going to believe?" --Stephen Hawking
Maybe the customer is being unreasonable.
Maybe the developer is being unreasonable.
It isn't possible to determine which from either person's viewpoint. You will ALWAYS think that you're right and that the other person is unreasonable.
Which is why you need criteria for bug escalation. Generating an incorrect response on 1 type of transaction for 1 specific scenario that may pop up once a year is far less important than a bug that corrupts the entire database.
And if your product is considered "mission critical", I would expect a data corruption bug to be fixed within 24 hours. Even if it is nothing more than rolling back the recent patches and re-issuing the previous version.
I'm surprised nobody has mentioned the fastest responses, which may take essentially zero time. One is: "It isn't a bug.". The other is: "Already fixed. Update your installation."
I'm an embedded developer, and when my stuff goes wrong, it can *really* do bad stuff. I've literally pushed fixed firmware to a controller running in a production scan/sort environment within five minutes of finding the bug, because it threatened to completely bring down a huge sort operation (and by huge, I mean 1 million+ pieces that day alone). I've also stayed up all night tracking down a bug crashing a device used by one of our larger (hundreds of millions of dollars per year) customers. Those, though, are the exception, and are driven by the massive financial and PR consequences of not getting it done right now. Throw caution to the wind, code and load if you are reasonable sure what's wrong and the stakes of not fixing it are high enough.
The usual bug fix cycle depends on complexity, impact, and risk. High risk of breaking things and low impact? Generally gets scheduled for the next release (4ish times per year). Low complexity and risk but medium impact? Code today, regression test the rest of the week, push this weekend. On average, mission critical bugs can get fixed in 8 hours or less around here, small to medium stuff is put on a weekly(ish) cycle with *lots* and *lots* of testing, and large stuff gets rolled to the next major release, unless it just can't wait that long.
I don't think there's enough info to even answer this one. I think the answer is entirely dependant on the importance of the program you put out. If you write rules for DoD firwalls, then you should be measuring in hours or minutes. If you write screensavers to put on OLPC... You have a little more time. And there's the inbetween, like if you write the algorithm that counts how many M&M's go into each bag.
Do not meddle in the affairs of sysadmins, for they are subtle, and quick to anger.
No offense intended.. But you should be finding these things on your own and creating patches. Not releasing it out for your customers to locate. Its as if I buy a truck from Dodge/Nissan/Toyota/Wherever and I get it home to find out that if I hit it off after driving for 30+ minutes, and try and nhit it back on, it will only go in reverse. I expect that fixed ASAP. Not 4-6 weeks.
So basically, -1 troll/offtopic is really slashdots way of saying "I hate that you thought of something before me."
Minimum turnaround.
Large product. Billions in sales go through it.
ITQA takes 10 days with no defect- 20 days with a defect.
Scheduling installation time, Sox forms, required paperwork overhead, project approval comprise the rest. Actual coding and testing probably bout 8 to 16 hours.
At a small privately held company, it would take us 1 to 3 days.
She was like chocolate when she drank... semi-sweet at first and then increasingly bitter.
does business with this company.
Depends on the complexity of the problem. But with only 1-2 mloc, problems usually aren't to complicated. If the problem is both serious and affects only a small part of the software, patches can be issued in 10-60 minutes after reporting the problem. More often though, it can take 1 or a couple of days before we fix a problem. The key to fast response it the ability to deliver small patches that don't take a lot of effort to install. Our patches are installed on a central server, and then automatically installed on each of the clients whenever the client starts a program.
We generally get fixes for real bugs out within 24 hours, unless the problem is traceable to the OS, the only factor really out of our immediate control. Even then, we do a quick evaluation to see if we can replace the OS function. Over the years, we've replaced quite a few of them, but rarely within 24 hours.
But we know our code backwards and forwards; I wrote the majority of the current codebase myself, and I can generally get to within a few lines of the problem just by a bug's description... the rest is a matter of minutes and testing. This app is very large - comparable to Photoshop in terms of feature count - but it is also very stable after 15 years of whack-a-bug and a continuous drive to make the internal structure as orderly and regular as possible.
It is my observation that the more programmers you have involved, the slower your turnaround time (for everything from bugs to features) will be. Likewise the larger the entity, the slower it will generally move. Almost every layer of management and corporate compartmenting disease will contribute to slowing down the process.
For the apps that I use that I have had the experience of reporting bugs, it is my general experience that bugs often are never fixed at all. One browser, "Omniweb", truly my favorite in terms of features, has bugs that make it essentially unusable for me. Crashing, slowing, lockups and so on - really serious problems. I've reported them, they never were fixed, in fact the software was never updated. Eventually, I just went back to firefox. Then as Leopard came out, after years of doing nothing, they released a "Leopard version" in which, perhaps, I might find those bugfixes if I looked... but as I say, I have moved on and no longer have any enthusiasm for the product. Slow bug repair (or ignoring them) is synonymous with telling your customers you really don't care what kind of experience they have with your software.
Apple, with all their emphasis on customer experience, does this too. They've had bugs in hand for very long periods where they simply don't address them. If your bug isn't something they think will affect a lot of people, it isn't likely to be fixed. I've not yet purchased Leopard, preferring not to catch early-adopter syndrome bugs myself, but when I do, I would not be the least bit surprised to find you still can't refresh a remote share that's been changed by the remote OS; that the wifi differs hugely in compatibility between PPC and Intel hardware; that mail still hoses the sent mail box based on the return address; that shell fonts are poorly rendered; that shell ANSI compatibility is still broken; that the OS still provides locked-up beachballs at the most inconvenient moments; that the OS still puts the wrong things away on the HD when RAM gets tight, and consequently becomes massively unresponsive... Basically, Apple doesn't have good control of their OS, are unable to respond to bugs in a timely fashion, so much so that they triage out bugs based on report counts, and the common patter is that Apple provides a great customer experience. So while my own experience is that bug fixes are important and can be quick in turnaround, here's Apple showing us that you can make a complete thrash out of the entire bugfix issue and still come out smelling like roses. So is a few weeks too long? Probably not, if you have a good marketing department. :-)
I've fallen off your lawn, and I can't get up.
That's why there are companies who think a minor bug fix, or a small development, changing fonts or simple features, reconfiguring servers, restoring backups etc is something that doesn't need testing, concentration, at least little bit of planning and basic things like version control. So that's quite common in the industry: customers who think they are getting their product for less money because they can force every change as an emergency. They don't realize they are making development more expensive with hacks and constant build, tests and deploys overhead. Simple concepts from lean methodologies like Scrum should be taught to anyone who plans to spend more than someone's monthly wage on software. Everyone can benefit from a healthier development cycle and software will come out better and cheaper. But there are some clients learning to get the benefits of a constant release cycle and, as the poster said, they are getting the beta development cycles for free.
I was thinking about a joke on my subject on the lines of "people only know how to buy tech on Civ", but it's less important and I'll leave it on the jokes backlog.
^[:wq!
That's not a question that can be answered here. Why would you even bother asking? Every piece of software is different. Every company has a different set of talent, different size budget, different processes and different sets of expectations.
You need to hire a lawyer. Part of my job includes crafting software licensing contracts. In your services contract, define levels of bugginess--mission critical all the way down to insignificant. Set acceptable turn-around periods for each level of bugginess: two weeks for mission-critical to months for bugs that you can work-around. (These are just examples; I'm not providing legal advice; hire your own counsel.)
A NYC lawyer blogs. http://www.chuangblog.com/
From nearly forty years of programming (yes, since the IBM 026 keypunch days), I can tell you with absolute certainty that the more that you do for management, the more that they will want from you. It is not your responsibility to bear all the punishment for the lack of foresight and resource allocation on their part.
Consider this: What would be the managerial response if you asked for a cost of living salary increase and that you needed it within 48 hours? Do you think that they would be willing to work day and night to make that happen?
Working in panic mode is not professional behavior, and it certainly is not conductive to good engineering practices. Furthermore, it is detrimental to long term company survival. Engineers who support continued unreasonable demands have only themselves to blame for enabling poor strategic planning by management.
It depends. E.g., if your software controlled a spacecraft, and there was a mission-critical failure, you would be thinking in terms of hours to find a fix - not weeks.
For the exception of the "ALWAYS", I agree with you. I have never had the attitude that the customer is ALWAYS unreasonable. Honest.
I prefer Flambe as apposed flamebait.
to begin a post with a sentence fragment which is a continuation of the subject line.
Momentarily, the need for the construction of new light will no longer exist.
Low priority issues just get dropped. Management may order us to start working on a high-priority issue a month after it was raised. For simple problems a fix could be released another month later. Many issues have to be escalated to our supplier, which usually takes a few months. Hey, we just released a bug fix to our customers two whole years after receiving it from our vendor. Security issues such as exploits are deemed low-priority.
Just change the version number and ship the patch. Then you can take all the time you need to fix it right.
been doing some Agile/XP work recently complete with Continuous Integration. we write website code and not box code, so might be a special case here, but our CI server builds in about 13 minutes. we have a production branch of the code monitored by CI so if/when we make a patch, it's immediately tested and built and ready for deployment. so if it's just an easy fix, it might be as quick as 30 minutes that we could deploy it out to the production website.
I also recall an experience I had with an open-source project in which I requested a somewhat trivial feature. the author added it to scm, sent me a patch file, and it was in the next nightly build as well. so that took about 12-24 hours.
in light of these experiences, 48 hours doesn't seem unreasonable to me, but I also agree with previous comments - it's a highly relative thing to judge. the best you can do is keep your code maintainable and try to reduce red-tape, I guess.
It seems your managers weren't able to negotiate a good contract terms, and that's why customer is able to dictate such harsh terms. Whether customer is right or wrong, let them know, that out of Quality, Time (Speed), and Cost they can pick only two.
I guess the real problem for many is the quality of the code in question and the level of documentation. If you as many OSS projects have very well written code that as a bonus is extremely well commented a patch can be issued within hours for Q&A. If on the other hand have a big pile of poorly made code that lacks both sanity and documentation, well, it will take time.
Spaghetti code can make even the easiest fix break things in totally unrelated places making Q&A a nightmare. If it takes you a couple of weeks to get a patch released something is broken. Be it management or code, it still sounds like a long time.
My own experience is that many OSS projects for some reason has extremely short patch periods and still dont break stuff all over when releasing a patch. One of the differences is the lack of centralized management. How much of the time for releasing the patch is really just waiting for approvals and such from others?
HTTP/1.1 400
The answer to your question is - "it depends". /.
It depends on too many unknowns that may take you longer than 48 hours to explain on
The overall size of your code base is a meaningless metric - it could be a beautifully architected masterpiece that is easy to tweak and modify or lots of little pieces with relatively few interactions or it could be a 20 Mloc ball of spaghetti.
What is the complexity of the specific code with the bug?
What is the source of the bug? Is it easily reproducible? Is the location of the bug known?
Is the buggy code relatively insulated from the rest of the codebase or is it central to the entire product (think my_very_specific_function_for_twiddling_diddles() vs. printf()).
If you fix the bug wrong how many other parts of the code could be affected?
What are the ramifications of screwing up the fix beyond embarrassment?
Unless you know all of these, it is arguable you are not prepared to answer if the customer is being realistic (as opposed to reasonable).
So, they want these faster? Sure, you'll bust your ass to get it done a little quicker but the thing that generally suffers the most in this situation is testing time. That means there is a much better chance of serious bugs getting out the door (probably about as bad as what you were trying to fix).
Management is going to notice when customers are screaming that the company is producing junk quickly. By now the boss has completely forgotten that he or she demanded a 4-5 week project be done in 2 days. In response they add more oversight (TPS reports and meetings with your 8 bosses) to ensure that they proper procedures are followed next time.
Now the 4-5 week project takes 8 weeks because you need approval from everyone to do anything and daily progress meetings to make sure everything is on schedule.
If you didn't come to party don't bother knocking on my door. Prince '1999'
Yes. But the right answer isn't to just say no. You and your company need to take steps to fix this problem.
You: It sounds like you have a lot of panicked patches going out. Improve your tests, expand your test coverage, consider creating custom test tools to better capture edge cases. Try to define an area or category of the app that usually leads to this situation, and consider Test Driven Development for it.
You do need to ask those within your company demanding the patch to allow you the time to get a proper fix done, not a hack job.
Your company: There are people in the world who can speak with a customer or contact a group of customers, admit it brokèand leave the customer feeling happy and well taken care of. You need to find and hire at least one such person, possibly many, and have them handle these situations.
I guess it depends on the severity of the fault but in my organisation they (customers) expect a "remedy" within 2 hours and a fix in 4 hours for a "show stopping" fault. If these times aren't met we incur financial penalties. That's mission critical for you..
"They misunderestimated me"
Basically, if you don't have the tools, you are SOL.
--
Geeks of the world unite!
In my last gig, we had a continuous build process going for all builds and unit tests for lots and lots of things. The release bits was selected from a build from the continuous build system which automatically deployed to the test servers once selected - which meant that nothing was done by hand other than selecting which build. If the nature of the fix was such that with confidence problems would be caught in unit tests, then we would limit the amount of other testing. The build time was about 2 hours (because of the unit test run) and so if we did everything right, we could get a fix out within 24 hours depending on the complexity of the fix itself. There was no "build" guy, everyone on the team was responsible for the build. We used a home grown "tinderbox" like system which meant that we had very accurate knowledge of what fixes went into what builds. The build directories and test results were automatically kept if there was a build error so it could be examined manually if there was a failure. Intermittent failures came up all the time and were snuffed out because of the ability to go back and check. On windows builds, unit tests that segv'd allowed you to drop into the debugger saving a huge amount of time not having to re-create a failure scenario. On linux builds it would core dump which allowed it to be re-examined. This system could be improved upon in many ways but it was very very cool.
http://blog.wired.com/27bstroke6/2007/11/yahoo-settles-w.html
Last Friday, 2 hours turnaround time on a minor defect fix. But then, it's for a data delivery platform, not a shrinkwrap product, and it was blocking a large client's use of the system, and the fix was simple. It passed through support reporting it, a developer and QA person each testing it, the developer fixing it, the QA person regressing it to verify it was fixed, and it was deployed, all in that two hour time frame.
So, yeah 48 hours? No, that's not unreasonable, depending on the difficulty vs. if it's a blocking defect. 48 hours turnaround for a new feature? Yeah, that's not reasonable.
..experimental patches that have passed QA muster to an ailing customer within two days. It used to be much shorter than that, but our regression test suite is more extensive now as a result of our growing featureset. And this is enterprise-level software that services core development infrastructure for major software companies.
How big is the software? How big is the bug/change?
For a new feature, 48 hours is generall way too short.
For a bug fix, the question is where is the bug? If it deals with money at all, then it needs to be fixed quickly, but it can't contain errors, so both time and correctness are important. In that case, then the size of your team becomes the determining factor. If you have a small team (
In my experience.
burrocrisy
and that would be what? Ruling by jackasses? Never has a slashdot misspelling been more apropos
If the problem causes your customer to be dead in the water, then 48 hours is too long. If its minor, then no. Imagine if your furnace broke and under warranty and they told you it would be 48 DAYS before somebody got out there. I think the sw industry as a whole doesn't take customer service very seriously.
I imagine the team works (read gets paid) to work 40 hours a week.
Would you, if compensated adequately, work 80 hours? This would about double the man hours and half the delivery date.
I have no problem working weekends, or after hours. Read my contract and you'll see the rate is about 3x what I normally charge.
This isn't a question about speed, this is a risk analysis question. Is is riskier to allow the bug to stay or riskier to run code that didn't go through a full QA cycle? At the end of the day, that's the customer's decision, not yours. It's your company's job to communicate the risks to the customer.
Having said that, I'd say that getting a build out in 48 hours shouldn't be hard. If you have proper source control and proper build procedures, you should be able to reliably rebuild the last version of your product with absolute certainty. If you have a one line change that has been code reviewed by other senior developers, you should be able to create a point release and still sleep soundly at night. (Disclaimer: ignore his paragraph for software that human lives depend on such as space vehicles, medical devices, etc.)
Eight hours is about the fastest we can go from code complete, to full regression test, to production. There's a lot of automation involved to pull that off. Most of the time is automated regression testing. Build and release to QA is 90% automated and only eats about 10 minutes. Release documentation is about another 20 minutes. Deployment to production is done manually and takes about 15 minutes. Our app is 300,000 lines of ASP.NET 2.0, C# and T-SQL. Database changes add another couple of hours to the whole process. Drop the full regression and we can go in less than an hour after code complete. We put a lot of thought into making our testing & deployment process as effortless and painless as possible, without adding any undue risks.
It really depends on the system, the problem and the people involved. The system we run where I work has changes put into it in production daily with turnaround times anywhere from an hour to several weeks depending on the change. We have some customers where a fix can be worked on and implemented over the course of weeks due to testing and they have no problem with that and we have customers that expect things to be fixed later today. The real question to ask yourself and your team is is it reasonable in your product or system to be able to turnaround something that fast?
It really depends on what kind of patch we are talking about but if it is a zero day security vulnerability I would expect it to be fixed in hours. The FOSS world is pretty much the gold standard for this sort of thing. If your software is super critical banking type stuff you should still be able to get a fix out in hours but also have an extensive set of regression tests to run the fix through so you can have some assurance that you did not break anything else.
~20 Mloc.
Fuck You.
I worked at a place with similar problems, while I was there they restructured things to rectify the situation.
Originally they had all the engineers delivering work into the main code dev stream, then when a release was needed they would branch for release and then run a battery of verification and regression tests which took several weeks to complete.
The solution was to tier the main line, developers delivered into an integration stream, the work in this stream was then regularly delivered en mass into a test stream where dedicated testers then gave it a work over to check for interaction issues between the various bits of work coming in. Once it received their approval it was then delivered into the mainline proper and another batch was brought into the test stream from the integration one. The mainline in turn was subjected to continuous automated regression testing.
The net effect of this is that you always have a high level of confidence in the quality of the mainline code. So if an urgent release is needed for some particular fix only that fix and it's interactions need to be tested, the rest of the code can reasonably be assumed to be in good working order.
Fast, Good, or Cheap.
Choose two.
More Twoson than Cupertino
This is less of an issue about economics and more of an issue of intelligence. But lets indulge. In free market capitalism, business' (you) provide services to meet customers (them) demands. How well or creatively you do that has a direct correlation to your degree of success.
So your really asking the wrong question. But you could try telling the customer that they are being unreasonable. Let us know how that works out for you. Heh.
Lieutenant: Of course I did.
Scotty: Oh, laddie. You've got a lot to learn if you want people to think of you as a miracle worker.
"The ferrets, they're every where I tell you!"
... in an interview that from a good bug report to patch issued was on average *1 hour*. I imagine that this is because they don't have the usual administration bullshit to deal with and can just "get it done." So, I'd honestly take a look at the process of getting the patch out rather than your development of it to find where the bottle-neck is (that is unless it is you guys that suck). Because, IMO, 4-5 weeks is ludicrous.
...but it depends on their expectations and the nature of the bug. Suppose there a situation where a new feature they absolutely must have in order to do business is so buggy it renders the rest of the software unusable. That's a show-stopper. If the software doesn't work at all, then you must deliver a fix ASAP.
For a new bug where you've recently delivered nothing new that's critical, i.e most of the time, IMO you're better off with a rollback than a 48-hour patch as long as there are no compatibility issues. What it comes down to is, do they want it done, or do they want it done right? A patch that breaks something else because you haven't been allowed the time to test it properly is a patch you're better off not releasing. In my experience, that happens with unacceptable frequency when you crank one out this quickly.
But they may understand that risk and be willing to put up with it for the sake of whatever they want patched. In that case, it's their call as far as I'm concerned.
On the other hand, if this is an online app, and the bug has opened up a vulnerability that has let them get hacked to the point where the goatse.cx man pops up every time they go to process an order... Yeah, that's probably one you should do in a hurry.
And the brethren went away edified.
Depends on severity of problem/risk assessment of change. Problem is, once you start factoring in risk assessment of change, you can be sure that whoever does the risk assessment will assess it as low/minimal risk to get it approved quickly. Then whoever has the problem will assign it critical severity to get it done quickly. So you need to do a real sanity check review and accept that you can't fully trust your inputs.
If you're going to be updating a few million desktops around the world, take a month to get a right. But even Microsoft has a much faster process (hotfix) for limited-scope fixes that aren't goinog onto millions of desktops for important enough stuff. So yes, being completely unable to hit 2 days on an expedite/important change is pretty bad. But, you never said how important the change really was.
If you are just doing something that isn't truly earth-shattering if it isn't perfect, if the cost of downtime isn't measured in $$$ per second, then you should seriously be able to trim down your timeline and process a bit here.
I've worked at a place with a normal-three-months-but-emergencies-as-fast-as-a-day-but-usually-two-days procedure, and in a place with a normal-one-week-but-emergencies-in-minutes procedure, and both seemed reasonable given the scale. I've never worked in a not-possible-at-all-in-2-days shop, but the normal-three-months place certainly would routinely refuse a 2 day request unless it was a critical impact (security, billing, or commission tracking problem, or current outage - nothing is more important than fixing your customer's bills or your salesmen's commissions, and if the system is already broken then the parts of your beuracracy that are in place to keep it from breaking no longer apply.)
is competition good, or is duplication of effort bad?
I think that software that takes a long time to patch is often, but not always, designed badly. Do the time sensitive parts of your code in C/C++/Asm and the rest do in a higher level language such as Python. That way the majority of your code should be pretty easy to patch quickly while still keeping the speed boosts where they matter most. An additonal benefit is that there are many languages out there that are easy to embed in your program and usually they have far more maintainers than you'd have resources to put on a project - any bugs that show up will often be found and fixed before you even know about them. Code written in high level languages often is also more resistant to many types of security issues.
:)
If you're already using a high level language and fixes take weeks then I think you may be doing something wrong.
At what price learning? At what cost wisdom? The price is a man's peace of mind, and the cost is his life.
Please upload the applicable code and how to reproduce the security glitch and slashdot can make an assessment of the feasibility of the timeline. kk thanks.
we got an urgent request from our support team
Seems it's NOT the customer asking for the quick fix, but the in-house support team.
How's this scenario:
A bug in the software is costing a customer using said software $80,000 per day.
Customer calls support and basically says 'fix this asap and all is fine, otherwise we'll be suing you in court for damages'.
30 days @ 80,000/day is a nifty 2.4 million.
If every day you slip on the patch release date past 2 days costs $80,000, what does that do to your 'reasonableness' equation? $10,000? $250,000?
If your own peple are asking for a patch in 2 days, I'd say you need to shoot for next day, as it indicates to me there's a large problem somewhere.
The customer described a program they wanted (to run on an embedded system). I estimated 3-4 months. They asked for 30 days or less. I explained what they'd get if I banged it out that fast - something that would work most of the time and not lose too much data. They then explained that the program would save them over $1,000,000 a month. If it quit working, they quit saving money, but nothing else bad would happen.
So, I saluted and said I'd try really hard for 3 weeks for the first version, then about three months longer for a version that would work all the time. Which is what happened.
Do you know the impact on this customer of not having the fix that soon? Maybe it's worth it to them...
Not all bugs are equal. Some you will be able to fix in minutes as they are little more then simple errors in your code. Others will be more complex because you designed something wrong and changing it impacts more then just the part that is bugged. And finally there will be errors that not just affect your program but the data the customer has created with your program.
You can't say a bug == 4 weeks. That is like saying, repairing a car takes 4 weeks. Is it a loose cable or have the wheels taken a detour?
To asses how long it will take to fix you have to know what the bug is. Is it an error in a line of code? Is it a function that does not quite what is expected? Is there a design flaw/oversight? Is data involved? And finally, have people come to depend on that bug as a feature? This ranges from a hotpatch in a couple of hours to something you may just have to learn to live with including having to code that 'bug' into future releases.
Now stop reading slashdot and start fixing that damned code. Geez, you would think Microsoft would have better control over its people. Oh wait, a patch in a month? Nah, too fast for an MS coder.
MMO Quests are like orgasms:
You may solo them, I prefer them in a group.
I can't answer your question but I can give you things to look at in answering it yourself. I hope these questions help you cut your fix time and make your customer a happy camper.
If you are taking several weeks turnaround time I would suggest that somebody look into what is happening during that time. How long from customer finds bug to reporting it? to diagnosis? to approval to fix it? to fix it? to build the package? to test it? to deploy it? How many times do you need to repeat one of those steps because it messes up?
Was the product built in a hurry and filled with spaghetti? When you make changes are there frequent crazy side effects? Do you respond to long release cycles by bunching up lots of fixes so that even though the customer gets a release late they get a big release? Are you making the releases too big?
My bet is that many of those items are taking way too long. For example, if you invest some money in regression testing capabilities then you can cut the testing time by a great deal, not to mention saving repeat bugs. If you automate and streamline builds then maybe you can get a reliable build out the door quicker. Of course, the disciplines of code management require equipment and tools and training/design to be successful.
Now, my next question is why is fix turnaround such a big deal? Is the product buggy? Did it launch without being complete and is now being enhanced? Are the customers extracting a pound of flesh for a late delivery by expanding scope and getting free work? Are you issuing bug fixes that result from the previous bug fixes?
A well funded and streamlined team can turn around fixes in very good time if they have the tools and the support to attack the causes for development delays. Most times managers want to shorten times without investing any effort (money) in methods that will save time. There is a common illusion out there that increased pressure will result in increased outut while in fact it typically results in the opposite.
Human factors - typically programmers, especially good ones, assume that they are the cause for their problems. Many times they are, but when the system is not optimized then no programmer regardless of how good can be productive. So everybody just looks at bad failure rates and is unable to change the situation. Frequently it is the process and methods that need to be fixed, not the programmers. Recognizing that is a major step in creating a highly performing team.
48 Hours?! HA! I wish my company had that, we (a few of our firemen) get a problem and fix it on the same day (through QA etc.) Meaning we could get a problem, and have the fix on their site in less than 2 hours total. It is actually encouraged we do this within a 24 hour period on some.
When you sold the software, your company should of sold a service agreement. This should indicate how long you have to fix bugs (depending on the nature of them) including lots of other stuff.
If they have asked for a fix in less then that time, you can try, but your customer should be made aware that if they want the fix faster, they should be paying for the priviledge.
If other people in your company are telling the customer something different, then you need to speak with your manager. The entire company should be giving the customer the same message. If that message requires you to have bug fixes out within a short amount of time, you need to inform your manager that you need extra resources (people, machines) to be able to do that. Obviously you can't get a fix out as soon as you hear about a bug, but any small bug should be fixable within a day or two, provided you have the resources to find the bug, determine the cause, implement a fix and test it within that timeframe. If you don't have the resources, your company shouldn't be selling the product (whether that is at the time the customer is buying the product or when support is answering a call) in a way that gives the customer the impression this is possible.
I don't know what your work load is, but it sounds like you don't have the resources to do this. Either get the resources or ask management to give their customers relistic expectations. I know its one of those "harder then it sounds" things, but either the company will get a bad name because they can't fix bugs, or the company will have a high staff turnover (and ultimately a bad name).
To summarise, discuss this with your manager.
My understanding of Microsoft fix timescales is something like
That makes 3 months. Is that a scheme that the rest of us should Embrace & Extend?
I'll see your Constitution and raise you a Queen.
Constantly. Isn't "don't use windows" article 1 of the Slashdot mantra???
(Typed on a Windows Vista notebook, of course.)
Prediction for end of Universe #42: Fencepost error in Quantum_bogosort.cpp
There is one thing you must never, ever do. That is to release code which has not been thoroughly tested. If you can't push a change through your normal QA process in 48 hours, then the change does not happen. Period. You don't lower your own standards and release shoddy workmanship just because a customer is demanding it. You get this change out there, it explodes in the customer's face, and you look like a moron. Not just to that customer, but to ALL your customers. And to any other professional organization your are affiliated with.
If the customer simply refuses to deal with you unless you circumvent your standard quality control, then that customer is doing you more harm than good.
If one specific customer wants a fix for one specific problem, it may be feasible to diagnose, design and code that within 48 hours. Testing is obviously out of the question, but - depending on the problem, and partly on the customer - that might not matter.
Talk to the development team and ask if it can be done. If they don't laugh at you, then talk to the customer and tell them that what they're getting in that timeframe is not a patch (with all the connotations of QA that word carries), but a "hotfix" just for them, that's not being released to anyone else unless they ask for it, and they should be aware that this fix may or may not cause unpredictable side-effects. Then everyone gets to be happy, and you don't have to compromise your patch process.
I work for a large healthcare organization and typically have very fast turn-around times (bugs often get squished within an hour). For clinical applications and other core applications, though, we're much more methodical and careful.
I often explain to the user that I can push changes out immediately, but it introduces certain risks. I then detail the risks they may face, and that if they say to go ahead anyway, at least they'll be aware of what might happen.
On an issue where we have a medical device reportable event, and patient safety may be at risk, we can issue a patch within 24 hours.
Really, it depends on your environment, and what needs to be done.
I'll use one of my web site as an example. It's all PHP and Perl, so ya, it's programming (I'm sure people will argue this).
Since I wrote all the code, I know it all inside and out. If you say "there's a problem [here]", I know exactly what file to look in, and what code to look for. I've banged out changes, tested them, and put them into production in a matter of minutes.
On a high traffic web site, we had a java applet which was being used by about 25,000 people per day. For little things, I'd change the code, test on all applicable platforms, and roll out the change in a few hours. Even then, the bosses were sometimes displeased with the time it took. Since I was careful to test, I never rolled out bad code, so I was never pushed into the long QA cycles.
Working with one company, things were a lot different. It went something like this.
1) Propose the change to your manager, with supporting documentation.
2) Manager would go to the project coordinator (i.e., customer liaison)
3) project coordinator would go to the customer
4) customer would approve the change.
Up to here was anywhere from an hour to a week. Sometimes the customer would put stipulations on the change, such as "there's a big event happening, or going to happen, don't make the change until X time."
5) document the proposed changes
6) hold a meeting with development, QA, the project coordinator, and management. Discuss the potential
changes.
1-3 days later
7) hold another meeting with the same people to rehash the changes.
1-3 days later
8) hold another meeting with the same people to rehash the changes.
9) Write the changes. Make them available to the QA team.
3-7 days later
10) Explain to the QA team that the errors they are experiencing with the fix have nothing to do with the fix, they were preexisting problems with another piece of code.
1-7 days later
11) hold another meeting with development, QA, project coordinator, and management, to explain that the error has been fixed with the supplied changes. The other problems are elsewhere.
1-3 days later
12) hold a strategy meeting to plan on how to fix the other problems.
13) fix the other problems, and break more things.
1-3 days later
14) have QA test the other changes.
14) roll back changes in step 13
15) beta test the previous changes, and notify customer
16) Customer balks at other pre-existing problems.
17) Repeat steps 5 to 15 again, until the customer gets tired of balking.
18) Implement changes.
Then start the process all over with step 1 to fix the other pre-existing problems.
The solution really is...
1) Identify the problem.
2) Gather together the appropriate staff who won't talk outside of your group.
3) Fix, internally test, and implement the resolution.
4) If anyone asks, there was no problem to start with, and you were all really working on steps 5 to 15 of the previous plan on another problem.
Funny how that works.
But, it's a matter of, is it a trivial fix, or something that requires serious rewriting? Did someone miss trapping invalid input in one line, or is it a poor coding practice through all of the code? Is it an included library that simply needs to be upgraded and recompiled?
Serious? Seriousness is well above my pay grade.
"10 Fucking days"
-- Mike Shaver.
OK, that statement has been retracted. But Mozilla have got serious security bugs analysed, fixed, QA tested on 3 different platforms, and out to tens of millions of customers within a week of the report being filed. And they're not getting paid for it.
Why doesn't the gene pool have a life guard?
The question of your turn around time is meaningless by itself. Suppose that a customer representing 90% of your business insisted that it was essential to get a bug fixed in 48 hours, or they exit, you go bankrupt. Obviously a 4-5 week turnaround is *not* acceptable in this case. Suppose a smallish customer made the same demand. 4-5 weeks may now be reasonable, depending on your evaluation of the bug. It all comes down to a simple equation: If the *cost* of action A is greater than the *cost* of action B, you choose action B. The cost of careful software management tends to be smaller than mayhem, but that may not be true for startup situations, or emergencies. So... Don't get caught up in dogma, always analyze every situation regardless of predjudice.
I worked on a few teams at Sun in the 90's that could turn the crank overnite, in as little as 8 hours. If a team cannot turn the crank from the time the developer submits the fix in less than 24 hours, the process and infrastructure is severely broken
How long is a piece of string?
I work at a software firm that handles CMS, association management, custom apps, etc. I know our customers want a fix within hours sometimes. We do our best but it will never be fast enough...they expect bulletproof apps from install (we know this will not ever be the case) but remember:
Select * FROM Users WHERE Clue > 0;
0 rows returned
(I love that shirt from think Geek...I regularly wear it on casual fridays)
There's a lot of variables here. Ultimately the question is simple: can you get the job done to the expected level of quality in the amount of time given. I will now check the anonymous box and continue to talk about my previous job.
In my previous job, I worked for a client software developer (vague, but go with it). Basically they put together software for cell phones, laptops, etc. As part of this, there was a server component that was used to update those clients periodically with new software, profiles, etc. The entire team consisted of:
2 Lead Developers - me and one other guy
1-2 Offshore developers - Bangalore for teh win
Now, given that, you may notice something missing. Anybody? Anybody? Oh wait, no QA.
See the thing is we had lots of QA people assigned to deal with the client side of things. But not a single one of them knew a damn thing about our server software. We could occasionally get QA assets, but since they didn't understand the system, you didn't get much for thorough QA.
So, as a result, most of the time, I wrote, tested and released my own code without a single other person even glancing at it. Thus, invariably we had bugs fairly routinely, especially given the contributions of our team in Bangalore. On the bright side given that I wrote most of the code myself, I could identify and fix the bugs very quickly. So I would often have a bug fixed and deployed to a customer in mere hours. I was however frustrated and pissed off much of the time because I never felt like I had any support, and thus why I'm no longer working there.
Getting back to this though, my point is, in that environment, taking more than 48 hours to fix and turn around a bug would have been overkill most of the time. If it was a lower priority issues, then we'd put them off to be fixed as part of a larger collection of minor bugs. If we'd scaled up the development team and the scope of the project and actually had QA people, it would inevitably take longer to triage and fix a bug regardless of the severity.
At major hedge fund in Greenwich we turn around major defects in two hours under threat of losing our job and our families livelihood while being screamed at by traders. Given this doesn't happen very often, but when it does you can believe it is fixed in two hours. All traditional software positions outside of finance on Wall St. are day care for adults.
By keeping the trunk reaosonably stable at all times (yes, spending a few MORE minutes looking and verifying changesets), we can put a copy of trunk on the customer's box without much risk. It's much like doing a git pull in a linux kernel tree.
Since your software continues to have bugs, I don't think you can claim that any previous patches were "bulletproof". That being the case, you don't hold a candle to most FOSS teams who get equally good patches for such exploits out in days, if not hours.
I'm in the same position -- I own and operate a small web/internet/custom software company. And certainly, things get published/pushed/shipped with bugs. Welcome to the software world taht we know and love. But when a bug creeps up -- when a bug is found by the client, or by the client's customers -- it gets fixed immediately.
And by immediately, I mean between 10 minutes and 10 hours -- if it's going to be fixed at all.
Certainly there are those minor cosmetic bugs that no one cares about -- client included. And there are those other usability bugs that have acceptable work-arounds. Those two get fixed in the next set of upgrades -- if the client ever wants upgrades.
But anything that actually affects the client's on-going business has to get fixed absolutely immediately.
And we're capable of this for a number of reasons:
- we build with "developer empowering" code, so it's easy to make small changes to significant areas.
- we don't have as many bugs and seems to be the average
- in general, much of our code promotes "self-healing" of user data
- sensitive data routes (financial, encryption, security, money, accounting) get extra care during initial development, so fewer bugs are emergencies.
As the owner of a business, I'm with the client on this one. If my web host had a bug that stopped me from writing a cgi script, I'd need it fixed pronto. If my pipe bursts, I need a plumber immediately. If my bank is closed when I need money, it's a problem. Any client whose business is affected by your bug is being very patient if they're willing to wait two days to resume their regular business operation.
You're stalling their business.
That said, client education is very important. That's why I've collected a list of almost 100 news articles of huge bugs in huge companies -- banks, NASA, various militaries, etc -- so when clients say rediculous things like "I'm paying for software, why does it have bugs", I can point to a billion dollar fighter jet, with four nuclear warheads, and say that it has bugs too. But that's not to get more time, it's just so that they understand there will be bugs, and they'll be fixed right away.
And yes, that's 24/7/52. (I take off the last Friday in July, not that anyone wishes me well for it)
Our app is a VB frontend using an Access database. Yeah, I know, I know...
But I found a noticeable bug, causing about 10% of our users (well, maybe 150-200 users) significant problems, not data loss but very poor performance.
We got a patch in 4 hours, and rolled into into the next service pack with little effort. I suspect our app is tiny, with at the time 2 full-time developers, 1 QA position, and around 8,000 users. It's highly specialized, and crucial for our clients, the product of 3 years' serious development and another 8 years before that of previous systems all the way back to DOS.
48 hours to patch a serious bug? Sounds like a big, all-hands-on-deck event. You present it in a way that implies, to me, that you don't like to send out 'good enough' patches.
It's all about priorities. We've been watching development of a web-based app to replace our Windows- base solution. F-I-A-S-C-O. A case study in how NOT to do project management, web developent, app design, everything is wrong about this. I pray that it will silently go away, but there is no practical hope of that. Someone has it on their list of things they did last year, and why they are deserving of their bonus. And they will get their bonus. We get the support calls.
Hold out for time to deliver excellence. When the business demands it, solve the problem, even if it's dirty, and then go and perform perfection. If you find that the dirty patch is considered good enough, and you;r enot allowed to go back and 'do it right', then you will have more reason to push back and resist the quick fix.
It's all a compromise. Fast, Good, Cheap. Choose any two of the three, right?
deleting the extra space after periods so i can stay relevant, yeah.
Some have already mentioned that it is not possible to answer your question without more knowledge but we all have opinions, this is /. All I can add that it depends, if you rush just for a change order there is something wrong with your management. If you take 48h to fix a problem for a 24x7 something else is a problem. I like that toilet paper answer except - I once had a paper mill as a customer, simple ? application taking care of the warehouse. The problem, 6pm 100 miles away it started crashing but it is just the warehouse application? Except, if you can't manage a warehouse you have to stop the production, if you can't manage the warehouse you can't send the paper out, nobody can track all that without a computer. So the estimated deadline was midnight and after that 3 ships would sail without paper, machinery stopped and restarted, cost XXX millions. Not 48h, not even 12h to fix - I got the system fixed 11.45pm by patching a live/running system in memory ( with help of a person in other country more experienced of that part of the system ) So, yes, I have taken my time to fix something and lazy as I am, sometimes days but (un)fortunately most my customers are/have been 24x7 - you fix it now ! So, case by case and remember to build your system they way they are easy to analyze and easy to fix.
I do Aikido training about once a week. We've got this basic excercise that's called 'Irimi-Tenkan' which actually means 'Going in (into/close to the opponent) and turning around (spinning)'. I'd say I'm about down to half a second. You half to bend your knees a little and see to it that the front half of your feet are the first to touch and the last to leave the ground. Watch out that your upper body stays vertically upright even if you're downing your first uke with a technique while doing so. That way you'll be spun 180 in an instant and ready for #2 coming from behind. Just don't be to hectic or you'll lose balance.
Bottom line:
It's all about pratice man. And take my word for it: 4 weeks and even 48 hrs is *way* to long, even for a beginner. I know snails that go faster. Really now.
We suffer more in our imagination than in reality. - Seneca
The first programming job I had was with a California insurance rating company. We were unusual for a software company in that if we actually guaranteed the accuracy of our software. If our software said your premium was going to be $600 a year, and the correct calculation was $1200 a year, we paid the difference. As you can imagine, this lead to great urgency on our part to fix bugs. If the bug was going to be an expensive one, our bug fix turn around time was 24-48 hours. If our error was a fringe case that was only going to cost us a couple of hundred dollars, we might let it sit for 2 weeks. I still wait for the day that Microsoft shows as much faith in their software that we showed in ours.
I worked for a consulting company doing custom development for various customers for many years after that. Our turn around time for bug fixes was usually 24 to 72 hours. On a rare occasion a bug would sit out their for 1 to 2 weeks.
I currently work for a company doing internal development, our department has much more control over what we do and don't do. Non-critical bugs can sit out there for months. Anything that is a security bug or one that prevents people from doing their job are usually dealt with between 24 and 72 hours.
I work in an agile development environment where our application is a network-distributed thick-client program. We are in constant communication with out end users, and when issues are reported at least 1 developer (more as needed) is placed explicity on the issue at hand. Many small issues are commonly released to our users in a matter of hours, larger problems requiring in-depth diagnosis are typically released within a small number of days, and only problems with elusive resolutions require longer than a week. It's not uncommon for us to be working on an issue less than 15 minutes after the report comes in--and the beauty of it all, all it takes for our user's to receive the updates is a restart of our application. It's wonderful to be able to call up a user, say "we've found the bug, could you restart the program and try it again" and have the problem finalized in moments. The application we are replacing with this one however was a different story. It has issues reported years ago that are backlogged to be released in future versions. There's often vastly more documentation than code for the fixes, and it's astonishing what sacrifice in quality a manager will accept as long as there's a plan to resolve it within a decade.
Do it once, and they'll expect it from you every time afterwards!
"It's the height of ridiculousness to say for those 9 lines you get hundreds of millions."
Usually we get small bugs and feature requests released within a few days. Today at 5pm we got a feature request that the customer said absolutely had to be there by tomorrow morning. Slightly optimistic I think, especially considering they're only on an evaluation license...
Where I work we generally get bugs fixed within 48 hours, anything longer and my boss gets an automated email detailing when the issue was raised, developer it is assigned to, etc. We can do this because our testing cycle is short and we only support hardware / OS / browsers of our choosing.
In my experience QA / Testing is the longest part of any bug fix. You might change one character or half the code base, but the testing department still have to go through the same process because they probably do not know the extent of the fix due to the principles of blind testing.
If the testing team are given too much info regarding the nature of a bugfix they may well tailor the tests they run to each fix. This sounds fine in theory, but it means if the developers miss something the testing team probably will too. Decent testing should be based on the original written specification rather than the implementation.
I dont read
sometimes patching dosent always work as a old dev i knoe this. i had one program with a miner bug that plaged it no matter how mutch i tryed to patch it it would not go away. a patch i orginaly figured a cuple days would up being a cuple months. couse i had to rewright the entire program for the bug to finnly be dead.
Back in the bad-old mainframe days, it was not entirely unusual to live patch (in memory, no less) the operating system on one of the Burroughs mainframe lines (Medium Systems, aka V-Series).
One customer in particular, a large finance services firm in Wisconsin, was running a very large ATM network on the system and needed a high-priority patch applied. I stopped the processor with the maintenance console and used the maintenance console to hand patch the operating system 'open' system call and started the processor back up. Less than a minute of service interruption wasn't even noticed by the branches or the ATM network. We had staff on site and developed, applied and tested the patch within 30 minutes of the bug manifesting.
Patching that system was quite easy, as it was all BCD (and addressable to the nibble).
Is it an enhancement, or a bug fix? If an enhancement, then you'll have to leave it to the marketeers who will always assume the enhancement is needed RIGHT NOW! Unfortunately, your bosses will determine the schedule for enhancements.
Bug fixes are a little easier to get a handle on. Determine what impact the bug has. Is it leaking data to unauthorized users? If so, is the data private or public? Use the egg-on-face test. If the data got out to the press, would the PR have a highly negative effect on the company? Negligable effect?
Is it corrupting data? If so, same analysis, but does the corruption of data have a highly negative impact on the application's usefulness? Negligable? Does it destroy very important customer data? Not so important?
Is it delaying data? Is the data involved very time-sensitive? Not so much?
Each of these criteria can help you determine how important it is to fix the bug. Unfortunately, it's your company's culture that will determine the timeframe. Laid-back company? Large company? Going downhill and everyone knows it? Critical fixes: 1 week, important fixes: 2 weeks, nice-to-have fixes: 4 weeks. Small company? Foaming-at-the-mouth company? Critical fixes: 24 hours, important fixes: 4 days, nice-to-have fixes: 2 weeks.
These are just guesses. Again, it depends on what your company has historically done.
And if, at the end, you still feel the deadline is unreasonable, I guess you can always just slow the work down without telling anyone ;)
Towards the Singularity.
Most defects can be fixed in 15 minutes once you know the root cause. Of course that can take days sometimes.
If you're worried about companies making mistakes that can be fixed in 15 minutes, then you probably shouldn't buy or use any more software, ever, especially from a couple of fortune 15 companies that shall remain nameless.
2 weeks is a standard practice in the industry. However, it can be shorter - much shorter - if the right things are done.
1. The customer has to be aware of extra risks. Getting it to them in hours means that some testing has to be omitted. They need to be aware of and accept the risks of a quick fix.
2. You need to have a well written code base. If you code base is fully understood, nicely modular, well documented etc. - everything that code bases generally are not - then you can find and stomp on a bug quickly. If not, it might take weeks of dump grovelling just to find the bug. If your code base is an organically grown monster rapidly achieving sentience, and management wants x-hour turnarounds, you need to tell management what resources you need to rework your code base. The upside: well written code bases have less bugs.
Prediction for end of Universe #42: Fencepost error in Quantum_bogosort.cpp
If management from the director level on down works around the clock, so will I.
-fb Everything not expressly forbidden is now mandatory.
I'm one of your customers, so I'm really getting a kick out of these replies...
At the risk of getting modded "offtopic" I will say what everyone is thinking and take a hit for the team
IS THERE ANY WAY TO BAN THIS ASSHOLE!!!! (pardon the little pun I threw in)
Goatse was funny 10 years ago but its really stale.
Make SELinux enforcing again!
I suppose it depends on what kind of request this is. I doubt it's a "wanna new pony" request for something nonessential but very time consuming, but a request for something vital like security or critical usability. Even if you can't deliver in that time frame (and odds are probably good that you can't otherwise you'd have done it rather than complain), you can still come up with something that will mitigate the damage for the customer. Then keep the customer updated as to how you are progressing in solving the problem. Devote as much resources as you deem necessary given the importance of the customer and the bug. It might not keep the customer happy, but that seems the best use of your resources.
In relation to the current state of the industry, your turn around is great. In my opinion, the industry stinks. The truth is that you can never be quick enough; the customer will always expect things sooner and sooner. I have worked in every type of shop there is from extremely "agile", since everyone has to put a term on something, to extremely structured. My best experiences were in an agile shop. The downside with this structure is that every developer needs to know everything that is going on. In my opinion, that should be a requirement anyway, but not everyone sees it this way. In the agile shop we could turn around fixes within hours or days. Because everyone felt a sense of ownership, people were less lazy about their work. If we were putting out fixes in hours, the expectation still grew -- they wanted it in under an hour. In the structured environment, all the developers felt stifled and disconnected from the application, without a sense of ownership; their input was less important. The work wasn't as strong and people rarely did anything extra. Business analysts who were not technical at all decided how the application should work, which was usually wrong. The development cycle was ridiculous. Something I could have completed in a week in my old environment took 6 months to a year. It was culture shock to the say the least. Some people think an agile environment is sloppy where the programmer should not be trusted. I say that if you can't trust your programmers, you need to hire new ones or a better lead. What is worse is documentation. In structured environments, documentation is treated as if it is the final product, rather than a means to the final product -- pampered and petted with political combing and pseudo-code. But if the industry wasn't this bad, Dilbert wouldn't be as funny as it is.
My turnaround time? Half a second. Watch, I'll do it again.
What if I do the same thing, and I do get different results?
I once worked with a developer that got an urgent request he turned it around in four hours. I stopped him before he pushed the patch to production. I forced the first peer review ever in that shop. His fix introduced a bug so severe it would have subtly corrupted all the data in the system over time. The one bug he was trying to fix would have been fixed but he would have destroyed every other working case for the code.
I stopped the patch release and as a team we provided a real fix in three days. I wasn't a popular guy with my coworker for a very long time because I stepped on his ego. I sometimes wonder if developers shouldn't have to carry malpractice insurance.
My consolation prize was that upper-management personally thanked me.
Upper management clearly got the big picture: You may look like a jerk for taking three days to fix a high priority bug, but how much more incompetent would you look for rapidly producing a cure that killed the patient?
In some cases, critical fixes help save companies from bad publicity, loss of potential
revenue, or loss of existing customers.
If you're writing code that is used by banks or other financial institutions, you have to keep in mind
that those companies have regulatory requirements. Violations of those regulatory requirements,
can lead to fines. Fines which could far exceed the cost of any software purchase price or continual
maintenance fees.
I know it may seem like an unreasonable request to turnaround a patch in 48 hours, but if
your customer is paying for that service through enterprise level contracts,
your company should be well compensated for the request.
Whether or not you, (the developer) are properly compensated, is something you'll have to take up
with your employer.
Happy Coding.
Avg. Live Expectancy of a SysAdmin, 45 Years.
If you have tests in place, it should be very reasonable. If you don't have full test coverage, you have to test everything manually, which could take a lot of time depending on the size of the code base. Write tests first is the lesson here.
My company's product is a suite of GIS applications. It's not uncommon to find bug mission-critical bug after a code push. Things often get patched, QA'd and released within 12h. It really depends on how important the bug is, in relation to value of upcoming project deadlines.
What? Me? Worry?
I often have to do fixes that are requested in 2-3hours.
7H for a reasonably proof fix.
approx 14H for a bulletproof fix.
yes that's fcking short and i never failed yet:p it includes testing and packaging, also "external human" testing for the 14H one.
welcome to my world (i think i would get bored in 2 weeks o.O)
I think you're asking an unreasonable question. The industry turnaround time is irrelevant to the particular business, software, and defect type that you're dealing with.
Like many other decisions, there is a simple balance between risk and reward here. The question is - what do you gain by doing the fix in 48 hours vs. what risk do you incur by foregoing the longer testing/QA cycle?
The answer to this question empowers you to deal with the customer more reasonably. If you're reasonably sure that this quick fix corrects the problem and does not add any new ones, and the problem is severe enough that it should not go untreated - why wait even 48 hours? Code it up and fix it.
On the other hand, if you can honestly say that the fix has major scope and brings a lot of risk/unknown with it, then you should make it clear to the customer that the possible reprocussions of rushing with this are greater than the problem you're trying to solve.
In general, bugs should be fixed as early as possible and as a programmer you should have a good sense of what's simple and safe and what's risky. You probably already know the answer to your own question, and the industry averages probably don't matter.
http://ed.markovich.googlepages.com
On IBM's zSeries boxes, there is a big difference between a test fix and a final fix. If a customer has a serious downtime bug and millions of dollars are at stake, we get them a workaround/testfix as soon as is humanly possible.
The support staff works 24x7 until the problem is identified, and once that occurs a "simple fix" which may not be the final one is shipped.
In that respect, it is not unreasonable to expect a result in 48 hrs. You resolve that customer's problem and document the temporary solution for others who may need it. (sometimes these temporary fixes involve rewriting the hex machine code in memory while the system is running... those are the fun ones!)
Once you get that out of the way, then as long as that temporary fix is stable, the development teams will take months to produce a final, official, and rock-solid fix. Since no one is sitting with downtime, it isn't a problem.
So which is it that the customer wants? A solution to their problem, or the Final Official Fix for the bug you've found?
We could release fixes once a day but we sit on them and make the customers wait for no reason other then we look release happy and unprofessional to our customers not to mention wasting an extraordinary amount of time managing builds every day.
Whenever we get release happy we cover it up by merging several releases into one in the change chronology.
By Chance do you work for Palm Inc? I know that urgent tickets pertaining to error 3000 from their major MR fix for the Treo 700P took months instead of days. Just curious.
There's no Freedom like UFP-dom
I work for a very small Telecom software company. I often times release patches in hours. We never take more than a few days to fix issues that are critical if we can help it. Occasionally our patches break something, which means another patch with a fast turnaround time. Our customers are often times depending on our software for their day-to-day operation, so quick turnarounds are often necessary. Other times we do it quickly because we can and it makes our customers happy.
I work for a large defense contractor on mission critical applications with millions of lines of code, and we have requirements (in the software spec) that mission-threatening bugs be fixed in 72 hours or less. The longer it takes, the less we get in our award fee (the "bonus" in a cost-plus contract). That includes root cause identification, patch development, regression test (at subsystem and system level) and delivery to the customer site. It's amazing how motivating money can be.
It's not a bug, it's a feature
Now, as the organisation has grown somewhat larger, there are many more developers, much more process involved in making even the most minor change. A lot more people want a say in everything which gets done. So our bugfixing time has increased exponentially. Major bugs, like "system down" still get a lot of attention and get fixed quickly, but turnaround on small easily fixed items now sometimes stretches into months.
Fast Cheap Right Pick any two...
I try to keep my dev team on a 24 hour goal. 24 hours from discovery to successful fix. Unfortunately our release process takes MONTHS to get executed after the fix has been made, but at least its available for human consumption ASAP.
The stipulation is that any fix which is beyond a certain complexity level cannot possibly be completed in 24 hours, but fortunately 90% of our fixes can (i.e. correct spelling, check for a null pointer, etc).
So tell us the barest form of the joke in a truly ugly form factor.
Then you can refine it much later after this story is no longer news. : )
My first Journal Entry ever, in 8 years! http://slashdot.org/journal/365947/aphelion-scifi-fantasy-horror-poetry-webzine
Rampant carbon sequestration destroyed the Dinosaurs' tropical paradise. I'm here to help repair the damage.
I work for a fairly large software company(about $1 Billion of revenue per year) and our turnaround time really varies. If we are talking a HUGE issue that effects lots of customers or one of our largest customers, then we are looking at 3-4 weeks once it gets sent from Support to Dev and then goes through the whole QC and testing process(example was a bug we had in our DST patch). Other issues can take over a year to get fixed, or if it is a small bug that effects one small customer, it may never get fixed. It's all about priority. 48 hours is totally unreasonable for any fix though. Someone needs to sit down with your customer and give them a wake up call.
I could generally get a fix for a small problem out in 48 hours. But then again my software may not be the complexity that your software is. If the software I was working on was say, the kernel of Oracle or UNIX, then I'd imagine 4 weeks would be good.
The vendor that supplies our company's billing system runs their software on a zSeries mainframe. They take 3-4 weeks for the more critical fixes and about 3 months for the low priority bugs.
You're already in line with industry standards. The best you might be able to do is shave off another week.
We can usually get a response in a matter of hours, but we build web applications which completely changes the playing field.
Wow. I never thought I would see such an offensive post on Slashdot. That was in every way childish and wrong.
I would like to comment this and turnaround. Some countries have laws for billing and salary systems so you better be fast otherwise someone may in worst case lose their business license, you or your customer, or at least it is very costly. About the bug in OS, if it has an effect to your business, you again better be fast. Been there, done that 2am for a system on other side of globe. IMHO the mainframe support is the best, my dealing with IBM and they were always able to deliver a PTF ( program temporary fix ) in hours at any time of day.
It depends, doesn't it always ? A fix can be urgent, but still utterly trivial to implement. Say a spelling mistake in a highly visible spot. If it's really a question of changing a single character in a string and rebuilding, I don't see why 48 hours is unrealistic at all.
If the fix is substantial, so it really requires new code, plus testing, quality-assurance etc, then 48 hours is unrealistic.
Yesterday afternoon I got a call that one feature in a new product did not work as intended. Project change 5 Minutes, verified, fixed and tested in two minutes, regression tests 45 minutes, rollout to all units in the market(!) three hours.
If there is something new or a complicated fix, I might get a day to fix it.
We are not Microsoft, we can't sit down and wait for things to happen. Our customers expect us to make things work. And thats what we do.
Too many (ie, all) software companies have got to the same point of assuming that software will be broken and that it's "reasonable" to release in that state. In fact, it's not and it is you who's being unreasonable, not the customers.
TWW
"Encyclopedia" is to "Wikipedia" what "Library" is to "Some people at a bus stop"
In all the projects I've worked on I use one question above all else: "I'm your customer, I need to ship what we have finished today, when can we have it?". if the answer to that question is anything but this afternoon then inheritently you aren't really doing agile (or not doing it very well). One of the back bones of the agile methodology is testing, and since you have to do it every day you automate it. Its this automatation that can give you confidence to release without massive cycles. But in order to get the confidence you need a decent set of tests:
1)Unit tests
2)Integration tests
3)Functional tests
4)Performance tests
Ideally you shoot for 90% code coverage from the unit tests, and the rest is about covering all the major functionality and error cases as possible. Every bug you find starts with writing a test that fails.
As a process this in itself can drastically reduce your turnaround time for a fix and give a team the confidence to release within a few hours. Because if the regression suite runs then you know you haven't broken anything important. Automate more and you should see your turn around time reduce.
Teams doing waterfall support models tend to take a minimum of a couple of weeks to release because they need to do a big block of testing (functional and performance) for all the new functionality and regression of the old software. A common way around that is for emergency quick fixes they branch from the current production and do the fix there with a different track in their team, which gets it out fast. That doesn't work so well for features however because they don't have the same sort of testing effort available and it just slows them down.
Teams doing agile don't need the big testing cycle, and in the worst case need to go to the end of the iteration to have the fix ready (which is normally 1-4 weeks long). They tend to be faster at fixing and don't normally need to resort to going back to the last production, because they do some form of release every iteration anyway.
Sounds to me like a bit more automation and reliance on it might help you greatly. With 20 MLOCS that is going to take a while for the coverage rate to get high. Still every little bit will help.
I work for a major db company, and we have two policies covering this. Policy 1 is that if the customer finds a real show-stopper (as in they have to suspend business operations) we will ship a hotfix as soon as possible. That might mean within a few hours, depending on the nature of the problem. This fix will not be tested before it's released (though we'll certainly QA it as soon as we can) because that would delay the release, and however bad the fix is it couldn't be any worse than what they already have - that's pretty much the definition of a Sev 1 defect. All other fixes will be released in point releases aproximately every 3 months, all nicely tested and packaged.
In contrast there's policy 2. That states that we'll do the above, roughly, but we'll also release a hotfix every 4-6 weeks, fully QA'd, that contains as many fixes of any importance as our major customers tell us to include. All versions of the software will be supported for as long as our major customers tell us they will be. Any functionality required by major customers will be backported from the version it was introduced in to the version they have. All customers are considered major.
Can you guess which one we operate?
4 - 5 weeks is rediculous to be honest, if you automate the build - test - packaging you and probably reduse the time taken here to effectivly 0. look at some tools to make your life easier, team foundation server if you're a MS shop, Cruise control and JUnit if you're not. Once you get used to having a full set of unit tests run for you every time you check in a fix to the main tree you'll never go back to a waterfall aproach. We aim to get a fix built, tested and ready for release in 2 - 4 hours from the point at which the programmer is given the task.
--
It depends on where in that sequence you spend the time. If all the time is going into build/package and shipment then your customers are right to demand a speed-up.
If your QA time was, say, one day, then I would look for a turn-around time (on the critical issues) of 48hrs plus the time to find the root cause and code a fix which could be anything from an hour to a year. If the customers don't know the root cause then they can't expect such a quick response.
Yes, I appeciate that you may need a lot longer than a day for QA. But if QA is a large part of the 4-5 weeks then maybe you need to work on this?
I'd just like to put the timescales mentioned here into perspective from a different type of software system. My experience of development has been providing online services in the form of web applications, so that means our customers don't need to install anything. That means our release process is different from the one mentioned in the question: it sounds a lot easier and quicker.
But any errors we introduce can impact the business severely, and in these cases 48 hours is just too long to fix a critical error. In such cases we need to take a view on how much the error is affecting our customers. If a high percentage of customers are affected then it needs to be fixed ASAP, but if only a small percentage is affected then a longer time can be taken to fix the error.
In our case it comes down to how much business we'll lose by not fixing the error.
http://devproj20.blogspot.com/2007/11/how-fast-is-your-turnaround-time.html
I think it really depends on the Bug. In a Web Environment i dont see any Problem to realise it. With our Navigationsoftware it is alot harder but sometimes possible. What has to be done, has to be done ;)
I'm amazed you still have customers. For really subtle low-level bugs, that will affect large parts of the system, ok. For run of the mill bugfixes, a week should be plenty of time. Unless of course you stuck with some dumb OOPs architecture. In which case all I can say is hahahahaha....
need a free COBOL editor for Windows?
Depends also, on the level of integrity required. If your software is a drawing package and the bug is that you cannot save drawings, then maybe you should have a patch on the internet in 48 hours, but if you have an aircraft flight control system, then maybe release in four weeks (analyse problem, design fix, risk review of knock-on effects of changes, code review of changes, update of test simulators, review simulator updates, write silumation tests, review simulation tests, test changes, review test results, write test reports, get clearance for flight trials, write flight trial tests, perform flight trials, analyse flight test data, write reports, submit for certification, etc), would be seen as impossibly fast. What is to be lost if your quick fix fails?
Unicorn Setu. "Eagles may soar, but weasels don't get sucked into jet engines".
1. You reply to FP.
2. You try to get karma by replying to FP.
Quit ignoring that by getting the first reply to the first post, your post cannot be buried any more than it is. You deserve to DIE.
I work on a soft that we release periodically. It take about 2 weeks to test and validate. So, the shortest period between 2 release is about 1 month. That is the normal bullet-proof release delay.
Sometimes our customer and sponsor asks for a special new feature very important, and wish to have it released tomorrow.
First, we schedule a meeting to have the feature explains in detail - half of the request end here as our soft is heavily configurable and we find a way to do about the same thing with the current version or at least help the customer enough for him to wait the next official release.
Next, if we have to develop something, we explain that we will do the best possible but that we can't validate the soft so there is a regression risk. This is done during a meeting where one of us explain in detail what changements are involved by the patch and where the risk is pricesly. Half of the request end here with a "no go" as the sponsor think the risk is too high.
For all the other case, we develop something very quickly and the result can be anywhere from very good to very bad and this depends only on the developer skills.
The battle of sales vs. Support will always exist. To cite an example, just last week one of our clients need more upload space. We removed the quota for the acccount because sales said yes. Just yesterday, the server crashed because the hard drive filled up. What should of happened, in my opinion, sales shpuld have asked: can we support this? We then would have said: not at this time, but, give us (three) days to get a new file server up so we can. If the client then wanted to leave, to me thats unreasonable. The client would then have to research similar companies, set up a new account, transfer everything, etc. But we could have given them more space in three days. Personally I think we all need to be more patience
They just don't understand your side of the issue. All they see is "I have a problem" and "You need to fix it" without knowing anything technical. You need to say "48 hours is not reasonable and here is why" and list them out. If the customer still wants it, let them know it won't be "bulletproof" and they need to accept that.
3-4 for weeks seems about average here... or at least it was. I submitted a few bugs shortly after our programmer left (which was over a month ago). I checked up on them the other day and they've yet to be looked at.
Generally a priority 1 bug would get turned around in 1 to 2 weeks assuming;
developer resource available to look at it
it was fairly straight forward to diagnose & fix
didn't require business analysis input
testing resource available to test it.
So if the bug is causing your customers significant pain then 48 hours may not be unreasonable assuming it is possible to identify and code the fix in that timescale.
"I deny nothing, but doubt everything." Lord Byron
I pretty much get paid to push out 'mostly' bug-free code very fast. It's not unusual for me to post fixes within hours (or less) of a bug report. My clients understand what this means and accept that bugs are going to get through that, mostly (one of my biggest ones - and I mean huge corporation that you would know the name of - refuses to supply me a QA person no matter how much I bitch nor will they let me speak to users when I'm trying to decode their specs)
It's just the way it is depending on where you work and what the applications do. At one of my clients, my software pretty much runs his business - everyone there uses it all the time. If there's a problem, I pretty much turn it around as quickly as possible, period. It's what he pays me (quite well) to do.
Is it the right way to do things? hell no. Is it the way a lot of corporate america works? hell yes. I'm lucky if I HAVE specs on paper to work from half the time - and it's never actually complete.
It's pretty much this way with most of the coders I work with (admittedly a small circle) - we come in to get the job done, mostly when other people have screwed it up. I usually have to deal with a lot of resentment from employees coders but they can't deny I get the job done - and that I wouldn't be there if they were.
is probably that you are posting on slashdot rather than actually working on the problem.
Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
Lots of comments already, but heres my take. Regardless of how long it takes *other* people to do it, it will take you about this long:
- The amount of time to understand what the problem is plus
- The amount of time to identify who can fix the problem plus
- The amount of time to communicate the problem to the people who can fix it plus
- The amount of time to find a load of the customer's software and equivalent hardware plus
- The amount of time to verify the problem with the build plus
- The amount of time to find the correct version of the software for the build plus
- The amount of time to fix the bug in that version of the software plus
- The amount of time to verify that the bug is actually fixed plus
- The amount of time to build a full production build of the fixed load plus
- The amount of time to run regression on the fixed load plus
- The amount of time to fix problems discovered in regression plus
- The amount of time to document any user visible changes plus
- The amount of time to package the load for the customer plus
- The amount of time to get the new load to the customer
While there is a lot here (hope I didn't forget anything), almost everything is under your control. Estimating each of the steps for a typical problem will tell you how long *you* can currently take. If you want to shorten the time, you probably can. But it will cost time and money (of up front work before a problem is discovered) to get the time down.
As you will probably find, even if you fix the bug in 0 time, there probably is a lot of time being taken up by other steps. So the question of "How can we reduce this number" should not be taken as a question of your programming competency. Instead it is an opportunity to clean up other areas. For instance:
- Reduce code ownership so as to allow more people to handle problems in various areas. This will reduce the amount of time it takes to route the problem to the programmer.
- Keep each programming task in normal development down to only a few hours (1 or 2 days max). This will increase the availability of development staff.
- Engage in training for first line support to streamline the process of taking technical details.
- Engage in good configuration management practices and maintain builds of every active load in the field
- Maintain a lab containing all the equipment usually used in the field
- Train programmers and testing personnel how to set up all the equipment quickly
- Maintain high availability of testing personnel for verification of field issues
- Keep a log of all the software (with versions) run by all important customers
- Maintain low complexity software (i.e., value low complexity in the code over other factors). Refactor complex code.
- Invest in a build system that can easily and quickly build a load on demand
- Ideally have the build system runnable on every developer's machine so that they are testing with real builds
- Maintain a full automated regression test system
- Maintain a continuous build system with automated regression system
- Have automated notification of build and regression system failures
- Have an automated packaging system
- Have a network (i.e. internet) distribution system for new builds to the customer
Mostly these are the obvious ways to improve your turnaround time. But probably you aren't doing all (or possibly even any) of these things. So there's always room for improvement. Since you have an intimate knowledge of your work environment, you can probably find all sorts of other ways to speed up the process. But all of these things are costly, so chances are management will choose not to do them. Then it's their choice to be slow.
20 years in customer support, but bulk of that in Tech Support and the most likely group to send development such requests.. They don't care if you have to keep a team at the office round the clock to get 3 weeks of dev in within two days. They don't care that they are being pushy. They care that you released a product without realworld QA, that your code is impacting their business, and that getting a straight answer out of the CS group can be damn near impossible. Those "unreasonable" customers are responsible for the rather high salaries you all make coding in the dark.
I work in a company of 4 people.
My projects are ALL MINE. I rarely require assistance from the other guys. I am the Network Admin, Database Admin, Coder, Tester, Q.A., Documenter -- everything for my projects.
When there is a problem or a simple request, I can usually get the work done before the end of the day. When there is a request for a totally new procedure, it depends on the complexity of the procedure and whether or not I have a similar procedure already in place.
My work requires web sites, WSDLs, cron jobs, UNIX, Linux, MS, Backups, Help pages, Automation, SQL, Perl, PHP, C++, Shell scripts, VB, and anything else I determine is necessary. I have the luxury of using what I need. I have never had to go to a two hour meeting with 10 managers to get a permission to use a $50 software package.
So a short answer is multiply by a fact of 3 for every level of development (Q.A., DB design, Web interface, etc) and multiple that by 10 for every level of management. If only one person is the end all and be all (and is not too overworked, and LOVES his job) then only multiply by 1.
- I live the greatest adventure anyone could possibly desire. - Tosk the Hunted
Do you want code modified and compiled on the developer's machine, emailed straight to the customer, with no guarantee that it will work, it's been tested, changes are documented, changes were checked back into source control, (etc)? we'll get that "fix" to you in 3 hours man!
But if you want all that silly "software engineering" stuff. . . well, that takes a little longer. Even with well staffed, smoothly-oiled organizations.
These are my friends, See how they glisten. See this one shine, how he smiles in the light.
Of course it is unreasonable to expect a bullet-proof fix in 48 hours. The thing that we look at in my department is downside vs upside. Downside = risk for additional bugs, upside = good customer relations, fast elimination of current issue.
Any code with that short of a turnaround time is a risk for bugs but if there is no workaround (or if the workaround is unwieldy) for the bug out there, it may be the best way.
The other issue to watch for is the "you did it before" problem. You need to protect your department from having these kind of demands put on it for minor issues, as these demands will lead to reduced code quality.
Coding Blog
The questions should be asking, "how much time should we spend in up-front work to reduce our error counts?"
You'll never improve quality in a test-driven environment. Quality comes from proper analysis of the problem space, and proper analysis of the design. This leads to isolation and minimal impact of requirement changes.
BTW, the posters who responded with the "why are you wasting time on Slashdot?" type responses are obviously ignorant of the meaning of knowledge work. To quote Timothy Lister, "People under time pressure don't think faster."
Why is the fix so urgent? It's because the custoemr is being hosed by the defect that's in the software in production. The test process is (should be) designed and improved over time to be a proven way to turn out code that won't mess put your customer into a desperate situation. So you do this 48 hour fix, and before long there's another customer that's got a desparate situation caused by this fix. And it may not be this fix that causes the problem, or it may not be the same customer that has a problem with this fix.
The software development organization needs to take a step back and release changes only after they are tested. Alternatively, the development organization needs to be clear with the customer that this is an unstable version, and don't be too surprised if it messes you up a little. The reality is there's only two scenarios you can realistically offer your customers: either take the stable version that we only update once per quarter or whatever; or take the leading edge version that can have lots of changes and customizations, but realize that you may be bitten by issues with it. Should issues pop up, we'll do our best to get the changes to you rapidly, but the changes will go through our test process.
I bought Red Hat 5 when it was first released for a customer demo. The demo was keyed to me getting a large contract to install mail servers virtualized and clustered. Problem, after creating virtual hosts I was unable to register them for updates at rhn. As the first release of RH-5 was pretty well riddled with flaws there were plenty of updates (about half the total installed packages). Working with Red Hat support and several members of their upper management I was able to get the issue resolved in just over 90 days. Lost the contract, thank you Red Hat. I will never buy, sell, or even remotely suggest your product again to any current or future client. And that my friends is what happens when you let your customers dangle.
Others have talked about how hard or easy the fix might be, and so on. This is about Quality Assurance and the processes that organizations use to attempt to achieve it.
I've worked in environments where a single line fix that isn't an emergency, takes seven weeks. I've also worked in environments where such a fix takes five minutes. In my experience, the quality of such patches is very similar.
In some environments where an emergency break fix takes ten days. The application is broken for at least that amount of time. No amount of persuasion that the fix in hand is better than production will speed it up. The idea is that the process for getting something into production includes so much testing, that things simply aren't broken in production. This approach is problematic. Testing generally does not test everything, despite claims to the contrary. Not a single such environment i've worked for has required code coverage testing. Only one environment in twenty required a code review where another developer looked at the code. Quality gates essentially consisted of a manager asking the developer if certain steps were taken, and if the developer says they did it, then the approval stamp was given. This is worse than useless. It takes time to get the attention of the appropriate manager, while adding no value.
These quality gate environments often also have built silos where a systems admin must perform deployment. A database admin must perform all database changes. And so on. Each of these groups are nameless and faceless. Communication to them is equivalent to email, that is to say, one way. Despite considerable chances for miscommunication, the admin teams are considered infallible. When it turns out that they aren't, and they end up changing the environment, breaking the applications, it often still requires the full cycle to apply fixes.
Another quality gate often enforced is documentation. I used to think that programmers didn't like to write documentation, and therefore avoided this work. However, it turns out that the required documentation typically has no well defined audience. No one ends up reading it, because it has so little to say that they need to know. The developers themselves don't value it, because it is often out of date with respect to the code. Some documentation has required sections where huge volumes of screen shots, ER diagrams, class diagrams, and so on are included that are automatically generated. So any effort towards documentation is considered to be a waste of time.
If you suffer from some of these symptoms, the recommended route is to change the processes. Drop managerial approvals. Include real peer reviews - code reviews. Consider implementing automated regression testing. Pull admins out of isolated silos and into development teams. Require only targeted documentation omitting content that is automatically generated. If possible, get the customer to verify all changes before launch.
-- Stephen.
I work for a smaller commercial software company and quite frequently do fixes with even shorter turn-around times even down to 15 minutes if it's a data corruption type of bug getting worse at that very instant.
However, this is for fairly heavily customised software - for each project we take a new copy of the codebase and so changes made in an emergency for one customer do not have any effect on any other. The system is also designed for easy patching so we can usually put in an emergency fix with no downtime and without the operators even knowing. A knock on of the level of customisation is that our front-line support in emergencies is handled by the engineers that wrote the system, aided by a direct link to the client's site.
With increasing levels of 'bespokeness' of the software, you'll always see more bugs but they'll be easier and faster to fix. MS beta software goes out to tens of thousands of users for testing and still the final product has bugs that no-one ran into before. Fixing a bug in this sort of software generally involves reading a vague description from a non-technical user being interpreted by a semi-knowledgeable front line support guy. If you can work out what's going on, you still need to check the change isn't going affect the other users using the software in different ways.
I (nostalgically) thought it was part of the 10-year anniversary celebrations...
The complexity of the system and the magnitude of the problem play an important part in this.
If it is a simple fix in a well understood part of the package, 48 hour turn around would be reasonable. This might be necessary if someone made a mistake that QA missed because testing requires special conditions unique to the customer. (Bullet proof testing is virtually impossible once the complexity of the system reaches a certain point. You can't test ALL the branching options a client is likely to encounter unless the system is VERY simple.)
If the sales people claim that something is a simple fix, and it isn't, try to document what the so called 'simple fix' requires in terms of modification and testing. You'll have to do the research anyway to do the fix, so you might consider presenting the preliminary results of your impact analysis and say that this is only the first pass.
At my previous job, the Unix admin group had a contract to fix problems within 48 hours. We almost always completed tasks within 2 hours. Because of this, whenever we would take, say, 10 hours to finish a task, we'd start getting phone calls and emails asking us WTF. We'd have to gently remind them of the 48 hour clause, although this never made the client happy.
I recall only one instance where it took us longer than 48 hours. We informed the client right away that what they needed done was going to take at least a week. They shrugged and said, "okay".
It always takes longer than you expect, even when you take into account Hofstadter's Law. --Hofstadter's Law
Man I would love to get 4-5 weeks to code and test. Instead of the 4-5 hours i get now to push out critical patches. so my question is.... Are you hiring??? And which planet do you work on?
The company I work for deals with getting medications to patients in nursing homes. If there is a bug it means I stay up and work until the bug is solved. It does not matter if it is 1am that I get a call about a software flaw. I'm up and awake until the bug is resolved so that we do not kill a patient by giving them the wrong drug.
I think it depends on the nature of the bug. I was sitting down for coffee the oether day with a field guy going over issues from a customer. Most were enhancements, one was a bug. It stopped them from using a feature. I could visualize the code problem, found it fixed it while at coffee. Checked it in, called into work for someone to hit the automated build button. Within the hour we had generated a new installer and delivered to the customer (a close customer) as a BETA from them to QA.
If the bug had been anything other than a real stupid, should have been caught in QA bug, I could not have done this. But it was simple to me and a huge deal to the customer.
But quick turn around can sometime make you look great to the customer. (The ones that pay our nice salaries)
Well, let me - an "unreasonable" customer - tell you a story.
.... turns out I was right.
This past weekend I purchased something from an e-tailer - one of the largest in the computer business.
I was torn between shipping to my house or my parents as I'm heading there for Christmas and didn't know which would be easier. I checked the shipping to both places and determined to have the components sent here so that I could save money and install the OS and such. Well, when I actually placed the order - after checking to ship it to me - it chose to ship it to my parents place AND charge the larger fee. So what did this "unreasonable" customer do?
I called and had them make the correction by hand but they refused to adjust the shipping rate. After explaining that I should get credited the difference they told me they had no proof. I sent them screenshots and STILL they were defiant. They said that it showed the same rate on their systems no matter what they did - to which I replied, "something must be wrong with your systems then." So, I tried again with a fictitious purchase to verify the steps and took screenshots all along the way. They STILL wouldn't credit the amount. All of this took hours...
I sent multiple complaint requests and was actually harassed by some of the employees UNTIL one decent fellow wrote back and asked for the screenshots. I worked with him for another 40 minutes or so, trying this and that, and documenting it all.
Conclusion: They had some bad Javascript code that wouldn't look for special characters in the identifying addresses. This would cause the system to bork and not process the changes.
So after about 4 hours of free tech support FOR THEM, name-calling, being made to feel like a total loser
I'm still waiting for the credit to post for my shipping and almost feel like sending them a bill for my services.
"unreasonable customers" - hmphhhh
We have done turnaround in 48 hours before, for stupid mistakes that are clearly more horrible than the fix.
Six score characters.
Brevity being wit's soul
I have enough space.
If you released a bug that a customer deems necessary (jumping, yelling and screaming) to be fixed in 48 hours, I see 2 possibilities:
A. You are not tuned in to the critical needs of your customer(s). You may need to take a hard look at your feedback chain from customer to sales to requirements to engineering to sqa...
B. The customer is an exception and is using your product in a way not used by the majority of your customers:
i.This can be because they were sold the product incorrectly (retrain your sales group;), or
ii. because they are a demanding customer you want to please (read $$$; refocus your requirements and testing;)
See Competing On The Basis Of Speed for some ideas on how to cut down that 4-5 week figure, and other advantages you get as the overhead comes down.