How Fast is Your Turnaround Time?
petrus.burdigala writes "I work for a mid-sized commercial software company (~20 Mloc) and we are frequently challenged by our supervisors to get fixes around the clock. Overall, we manage to get a 'bullet-proof' patch in about 4-5 weeks (from coding->QA->Build/Packaging->shipment), which I consider not so bad. But the other day, we got an urgent request from our support team to come up with a decent fix in 48 hours. I think they're a tiny bit unrealistic. So I wanted to get feedback from my peers: are we doing that bad? It takes months for other software vendors to issue zero-day exploit fixes, are our customers being unreasonable?"
You have to serve the client who is paying the bills - and we had a very vocal one (Nik*). We had a running joke about the release d'jour. But it wasn't a joke. We literally would push a new build to them every day which contained minor bug fixes. It was maddening! But no one had the balls to stand up to the 800lb gorilla, so the madness continued. As a side-note, they were acting as a beta tester and anyone in the software business knows what that can mean.
It depends upon the nature of the problem and the competency of the developers.
If you know enough of the code tree you can tell when first reproducing and examining the failure whether it is a one off mistake or a larger procedural fault.
Single instance stupid errors (doh! moments) can be rectified and put through testing fairly quickly, however if your initial examination uncovered a larger problem then obviously the process will take longer (if at all - consider workarounds).
If the original dev/test team has been replaced over time this becomes a more difficult issue and every bug must go through complete verification simply because the extent or ramifications of the code modification will not be known.
In some instances we have had fixes out of the door the same day an issue was noticed, in others months go by before a final fix is put in place.
liqbase
But the other day, we got an urgent request from our support team to come up with a decent fix in 48 hours. I think they're a tiny bit unrealistic.
Well, we really can't answer that question with knowing how big the problem is. If it's an embarrassing typo on a dialog box, then 48 hours is reasonable. If it's a windows vista security patch, then 48 days would be unrealistic.
-Grey
Silver Clipboard: Time Management Tips
It depends on what you're maintaining and how complicated it is. I've gotten fixes out in 2 or 3 minutes. That doesn't mean I'm fast and you're slow, though. "How fast is your turnaround?" is like "how long does it take to write a computer program?" It's hopelessly vague.
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
Sometimes, customers are unreasonable and if they are, they should be treated with respect and the problem explained to them. Yes, they may be incredulous, but if you hold your ground (if they're being unreasonable), treat them with respect, they will come around.
The fact that the parent was moderated down just shows me that the arrogance, contempt, and stupidity in corporate America is alive and well - especially in IT.
I prefer Flambe as apposed flamebait.
With a little simplification, you have four parameters: Difficulty, quality, speed and available resources. Whenever you fix three, the fourth follows (with some unvertainity). It is well known, that there is a limit on how much you can improve the speed with more resources. So there is an upper limit on speed already. The second problem that difficulty is unknown when starting such a task. There is no fix for that.
So if these people fix speed and available resources, and difficulty is fixed by the task, quality is determined by these factors. Period. There is no arguing with hard, real limits. If they do also want to specify the result quality, then they have to leave speed open. Again, there is no way around that limitation. In fact they should be happy if the team manages the required quality at all in reasonable time. Not all teams do.
Maybe thisn will be an argumentation that is inderstandable for people with a business background. Engineers should already know this.
Software engineering is engineering. Engineering tasks in general have minimal time requirements. Look at structural engineering: Nobody would try to design and build a full-custom bridge in a week. Instead it takes up to a decade, depending on difficulty. And you can generally not speed things up by increasing the team size.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
A patch (IMHO) is a bug fit to existing code. Given the resources we should be able to get a PATCH out in a week. However, if you need a new version of the software to address the issue. Then we're talking longer development/testing/QA times if which case 4-5 weeks would not be unreasonable. Bugs should be fixed as soon as they are spotted. If their is need for a whole rewrite then you may want to talk to your staff
Ask not what you can do for your country. Ask what your country did to you
I know I'm going to end up baiting some developers, but I work for a specialized ASP and see a ton of third party software from a perspective few get...
Normally, the smaller the company the more agile. No surprise. They also get patches out faster too. Also no surprise.
When we look at vendors of equal size, the ones who are really quick at sending out patches are in that situation because their software is more buggy, and they have a *lot* of practice. It never fails.
In response to your question, I would suggest that you should look more at the frequency of patches and less at the duration. Sure, it might not be as fast as your support group wants, but if you start reflexivly sending out patches every time someone yells, then your overall product will suffer since you can't possibly do the proper QA to ensure THAT patch you just whipped up doesn't break something else.
That brings me to the age old choice:
Pick 2 of the following:
Speed
Quality
Cost
48 hours is tad bit tight. However, I've turned things around in a similar amount of time.
But, the old adage is true: you get what you pay for:
When faced with unreasonable deadlines in the past, I usually voice my opinion once, and just do the best I can. Your higher-ups are probably already quite stressed at this point, and adding stress to the situation doesn't do anything for your career or theirs. Rather, if you make the point that you're doing the impossible, you might just have a little bit more bargaining power when it comes time for raises.
But on the flip side of the coin, if management doesn't learn, and you find yourself constantly asked to do the impossible, you might want to consider employment elsewhere...
The society for a thought-free internet welcomes you.
You really have to supply some more detail to get any useful answer. And what is ~20 Mloc? About 20 million locations?
/.? Do you make internet-facing apps, and an urgent request means your customers just formed a spamming bot-net? Are in the medical/health care field, and an urgent request means folks might die?
What kind of software? What classifies an urgent request? Do you make games, and an urgent request means your bug just made front page
I think a better question is, how do you classify bugs? How do you make that trade-off between fixing a bug ASAP and taking the time to make sure the bug fix is done right?
Who is involved in the decision process? Is it just the technical & regulatory folks? Do you pull in business folks to help gage customer impact? Do you pull in sales and support to see if they can push a work around before the final fix is ready?
Those are all better questions than, "How fast do you do this task of unspecified scope."
I love the parent's subject-line analogy.
I'd add, it depends on product, the complexity of the codebase, the extensibility, modularity, readability, and extensibility of the codebase (eg, if it's highly modular it's easier to test a fix that's limited to the module/plugin)
I'd suggest that weeks sounds too long for an in the wild update without a security patch - or published workaround limiting your exposure. (eg, "use this method to restrict the IPs that can access it to trusted ones.") But that isn't me saying you're developing too slow, it's me saying that if it's going to take you that long you need to either find alternate solutions or create a architecture that allows you to quickly make access-limiting patches and layered security.
Actually, I'd make that more broad - if they want faster response to patches, what they need to do is to invest a lot on a highly modular, pluggable architecture so you can MAKE rapid changes. It's really a question of how much investment they want to make.
We routinely do same day fixes to certain kinds of things... but certainly the complex things take longer. And I think we're pretty unusual in that regard.
Looking for freelance Actionscript (Flash/Flex) or ColdFusion work and/or freelance developers. Email me, put Slashdot
*15 minutes.
It's bad enough that they directly state they're not really testing patches with a 15 minute turnaround, but the fact that they're making mistakes that can be fixed in 15 minutes speaks loudly as well.
--
Maybe the customer is being unreasonable.
Maybe the developer is being unreasonable.
It isn't possible to determine which from either person's viewpoint. You will ALWAYS think that you're right and that the other person is unreasonable.
Which is why you need criteria for bug escalation. Generating an incorrect response on 1 type of transaction for 1 specific scenario that may pop up once a year is far less important than a bug that corrupts the entire database.
And if your product is considered "mission critical", I would expect a data corruption bug to be fixed within 24 hours. Even if it is nothing more than rolling back the recent patches and re-issuing the previous version.
That's why there are companies who think a minor bug fix, or a small development, changing fonts or simple features, reconfiguring servers, restoring backups etc is something that doesn't need testing, concentration, at least little bit of planning and basic things like version control. So that's quite common in the industry: customers who think they are getting their product for less money because they can force every change as an emergency. They don't realize they are making development more expensive with hacks and constant build, tests and deploys overhead. Simple concepts from lean methodologies like Scrum should be taught to anyone who plans to spend more than someone's monthly wage on software. Everyone can benefit from a healthier development cycle and software will come out better and cheaper. But there are some clients learning to get the benefits of a constant release cycle and, as the poster said, they are getting the beta development cycles for free.
I was thinking about a joke on my subject on the lines of "people only know how to buy tech on Civ", but it's less important and I'll leave it on the jokes backlog.
^[:wq!
From nearly forty years of programming (yes, since the IBM 026 keypunch days), I can tell you with absolute certainty that the more that you do for management, the more that they will want from you. It is not your responsibility to bear all the punishment for the lack of foresight and resource allocation on their part.
Consider this: What would be the managerial response if you asked for a cost of living salary increase and that you needed it within 48 hours? Do you think that they would be willing to work day and night to make that happen?
Working in panic mode is not professional behavior, and it certainly is not conductive to good engineering practices. Furthermore, it is detrimental to long term company survival. Engineers who support continued unreasonable demands have only themselves to blame for enabling poor strategic planning by management.
Even if the bug is obvious, it doesn't mean that your fix
1)Works
2)Works correctly for all corner cases
3)Does not have unintended side effects
4)Didn't accidently include some other changes you were working on before, which are not ready for production.
You still need to QA. Attitudes like yours are why the quality of software is so poor.
I still have more fans than freaks. WTF is wrong with you people?
This is mid-sized company here. Think about big time companies like Apple, Adobe, and especially Microsoft...companies who may sometimes take MONTHS to get fixes out the door!!! For companies like these, the community produces a fix before the company does... Moral is: lay off the guy a bit. I agree, the bug should be fixed ASAP, but instead of pointing fingers...think about the bigger picture and how even billion-dollar companies get lazy and complacent with software patching.
Even if the bug is obvious, it doesn't mean that your fix
1)Works
2)Works correctly for all corner cases
3)Does not have unintended side effects
4)Didn't accidently include some other changes you were working on before, which are not ready for production.
Yes but there are minor bugs where the fix works, where 2 and 3 simply do not apply and where 4 can be easily avoided by doing a cvs diff. The 15 minute example is for extremely minor bugs. They are fixed because they can be. Sometimes, very minor bugs slip through the cracks. It happens to everyone.
And for things that are not so minor, we can often fix and release in 8 or so hours, doing the requisite QA. If you know your code base well enough, you know the scope and reach of the code you are fixing.
Exploits should be a high concern for any company
Which is exactly why exploit fixes must go through STANDARD QUALITY CONTROL. What the fuck good have you done if by fixing one exploit you introduce ten bugs and two new exploits? I don't care how urgently the customer needs it. I'm not going to give them something I haven't tested. That's insane. If they don't like it they can shop elsewhere.
I work for a large healthcare organization and typically have very fast turn-around times (bugs often get squished within an hour). For clinical applications and other core applications, though, we're much more methodical and careful.
I often explain to the user that I can push changes out immediately, but it introduces certain risks. I then detail the risks they may face, and that if they say to go ahead anyway, at least they'll be aware of what might happen.
Yeah, but everything is relative. If you're fixing a Hello World application and you can't get it turned around in 5 minutes there's a problem. But if you're fixing an international banking application bug with security implications, 48 hours is definitely unreasonable.
At the risk of getting modded "offtopic" I will say what everyone is thinking and take a hit for the team
IS THERE ANY WAY TO BAN THIS ASSHOLE!!!! (pardon the little pun I threw in)
Goatse was funny 10 years ago but its really stale.
Make SELinux enforcing again!