Security Patch Creation at Microsoft

devonshire writes "Officials at the Microsoft Security Response Center have provided a detailed look at the process used to create security patches. From the time the first vulnerability data is received from grey hats to the time a bulletin is shipped, it's a pretty interesting look at how they handle the information flow and patch testing and why it takes so darn long to release an IE update."

Comments

Typical corporated programming

[guruevi]

Instead of just believing the people that there is a problem, they have to test it out and develop a plan and then reprogram the piece. I hate that. In my company they have implemented such system too and if you have a problem you have to wait a month before it is planned in (if it is accepted by a group of non-technical managers) and then another month before it is fixed making a problem sometimes last for over 6 months and after an endless amount of pointless meetings there is finally some kind of fix. Programmers in corporation are under a lot of (time) pressure and that is not good as it makes them make mistakes. But they have to be able to make quick fixes (as is with most Linux projects) without any corporate meetings or managers.

Re:Typical corporated programming

[Atrax]

Your company just seems to have a problem of balance. Your company may have a slow process, but equally they'd be insane to lean too much the other way and just let the techies spin out patches willy-nilly without fear or favour.

Striking a balance is the trick, and non-technical managers will tend towards the extremely cautious end of the scale without their caution being necessarily grounded in a realistic appraisal of the problem. They don't realy understand it, so they go slowly and have accountability at every step.

Sounds like you might want a shorter chain of command, with technically knowledgable managers making the calls.

How you get that to happen, well, I really don't know. A new CEO might be a start (it's worked at my old company)

Re:Typical corporated programming

[Tune]

Either you have no idea about how (software) project management works or you have seen some worst-in-class examples at your company. Testing and reproducing a bug is *very* important. Bypassing that step is a guarantee to waste valuable programmer's time on non-issues. In a healthy organization with averagely skilled testers, this part of testing takes a couple of hours at most.

Next is bug fixing. This is by far the most variable and unpredictible part, requiring the best of any programmer. It may take minutes or it may take weeks. Besides good programmers, good process can be of great help here.

Finally comes the release testing, which is what the article is talking about. This phase is essential: *never* trust a programmer if he says its "fixed and I tested it". Generally, programmers are simply incapable of testing their own stuff. I know as a programmer. Release-testing takes a considerable, but predictable amount of time, assuming the programmer did a good job. Skipping this phase will sooner or later lead to disasters like the recent Netscape 8 release.

Now I agree with your complaint on workload and lack of tech-savvy managers, but it's nonsense to say that the process as a whole sucks.

UDP Floods

[Anonymous Coward]

I don't think there's a single service on a windows box that can withstand a UDP flood. This has been known to be an effective DoS method for years...roommate using all the bandwidth with bittorrent? Playing Doom3 in the middle of the night with the volume jacked up?

Send a UDP flood to ANY of the services which are actively listening by default, problem solved. Where's the triage team on that one? I guess 99.9% resource consumption isn't a vulnerability in their eyes.

From the article:

[guruevi]

It's not easy to test an IE update. There are six or seven supported versions and then we're dealing with all the different languages. Our commitment is to protect all customers in all languages on all supported products at the same time, so it becomes a huge undertaking. 1: languages shouldn't be a problem, that is (hopefully) not completely split up throughout the source code is it? aargh!!!! 2: I know only a 3 SUPPORTED IE versions (IE 5, IE 6 and IE 7)

Re:From the article:

[Vo0k]

1: languages shouldn't be a problem, that is (hopefully) not completely split up throughout the source code is it?

You'd be surprised. Very surprised.
Things are far more screwed up than you'd think. An article on development of a new OS release would come in handy, but putting things shortly, somewhere between 60 and 80% down the way with the development of the new OS, the code is branched into "local versions" which are independently developed by corresponding local Microsoft divisions. Bugfixes, features etc are usually shared, but only "usually", and the final code base varies wildly. There's no simple way to "translate" a version of Windows, or port features from one to the other. That's why each language has separate service pack and the service packs for them show up at wildly varying intervals - each team has to roll their own. That's why e.g. people in Poland used german version of WinNT instead of polish one on mission-critical positions - because it's more stable. There's way more to "local versions" than plain "local language files". The design is consistent thorough the system, but the code behind it may be completely different, even if it's not really localization-related.

Re:Testing is only a priority on closed source app

[Renegade Lisp]

As to users giving you feed back. HA! The best I get is once in a while someone tells me that something crashes. I might die of shock if someone sent me fixed source code.

Remember what ESR wrote about this? "If you treat your users as if they were your most valuable resource, they will respond by becoming your most valuable resource."

In other words, I think this is all about community-building, and I grant you that this may be beyond what you can do as a single developer who simply shares some code with the world. Still, I have found ESR's statement to be quite true in my own projects, and it only takes a small effort to express this attitude in the e-mails you send to your bug reporters.

Re:Testing is only a priority on closed source app

[Anonymous Coward]

I find it strange that open-source application authors never, themselves, sell their product as well. Why wasn't the creator of WINE the founder of TransGaming or CrossOver Office?

Oh give the man a break...

[Kjella]

We have to make sure it doesn't break the Internet [web access provided by IE, which as far as our customers go means breaking the Internet]

The Internet wouldn't be broken as such, but I doubt the users would see it that way. To them, it doesn't matter if it is the browser, the connection or the servers (massive worm?) that is broken. They can't do what they want, hence it is broken. It is as simple as that.

Kjella

Re:Testing is only a priority on closed source app

[Dancin_Santa]

As an Open Source developer, I'm not in this for the money. If I were, you can bet the project would be Closed Source.

Rather, I want this project to be open and usable for all. To that end, I license it under the GPL and anyone is free to use it.

So my users are partners with me. They are not my guinea pigs. Though I maintain control over the project, there is no set-in-stone law that no one else may fork the project. In fact, they are encouraged to, if they feel it necessary.

I release the patches, and they accept them or reject them, depending on their own circumstances. I don't rule them with an iron fist. I consider them my Knights of the Round Table where they all have the right to say what they want and none is any greater than the other.

So maybe you think that users are passive slugs, but I'd rather give them the benefit of the doubt.

Re:Testing is only a priority on closed source app

[Atrax]

Was I talking about IE? Was the OP? Surely we were debating the patch process in general, not specifically IE?

Besides which, a hell of a lot of corporates consider their intranet (extranet/web) apps 'critical'. IE (or other browser) is a major component in that mission-critical situation, wouldn't you say?

Re:Testing is only a priority on closed source app

[noidentity]

are you seriously suggesting you'd just release a brand new patch into the wild without even cursory testing?

You can always release a patch to the patch if any problems are found with it :)

But seriously, it makes most sense to correct most bugs (that will be caught in the short-term) before a wide release, where there is a single copy of the source, rather than after release, where there are as many copies as there are users.

With open-source anybody is free to provide this service. If the author only has the time/motivation to do barely-tested releases, why reject his code? Someone else with the desire can do testing and make releases to a wider audience that are more stable, and users can choose between the two options (or more). These can even form without any direct arrangement between the various parties.

Grey hats?!? WTF

[thomasj]

Some people submit a vulnerbility report to the brickwall called Microsoft Support. Then after 6 months they release a security opdate. And now they call the submitters "Grey hats"? What do they call themselves? The "Pink hats"?

Re:The Market Cycle

[xtracto]

But then again, you are making money by SELLING A SERVICE not by making a program.

I dit not spend my 4 Unviersity years learning how to rightly develop computer systems just to go out and be a seller... or a service provider.

I would had studied Economy or public relations.

Re:Hahaha.

[multi io]

If they accidentally deliver a patch to IE that makes the browser send 256 requests per second to randomly chosen servers, something that's indistinguishable from "breaking the Internet" will happen.

your logic is bullshit.

[spacepimp]

I think you are responding more with anger than with logic. Firstly, whoever did your deployment of firefox, should have tested it before he went to everysingle machine and deployed the update, this is called quality contorl/damage contorl. secondly it is very easy to remoe firefox, and install whichever version you need. From what i gathered in your statement, you are claiming you have never had any down time or senseless tech cycles put towards removing spyware or malware on any of your computers. I do tech support as a consultant for about 20 small businesses. this is by far the most common phone call i receive, "my computer is broken i cant get past these pop up adds, internet explorer keeps crashing its really slow and i cant get my work done" now there are some malwares and spywares you cant get rid of, i've reimaged machines after several hours of attempting to remove some of the newer variants. now let me ask you where did you save the time, and money? was it from the extended hours of firefox, in a deployment cycle, (seriously this should take moments to install and uninstall)? If you think i am exaggerating call Dell or any other computer support company and ask them the number one call they receive, it isnt that firefox isnt working its their entire os, to which they respond put in your restore disk, so they can keep a profit margin. Im not a fanboy, but i do see the weakness behind Internet Explorer, and the fact that microsoft didnt update a thing until they lost ground to Firefox (ie: they had to protect their name) seriously redo your math, and figure out, where your costs lie, if you think the only response is hiring a unix/firefox coder to analyze and fix firefox code, then your techs are incapable, or just plain idiotic,or you should cease doing your own tech cos you are doing more damage than good.but i suppose you just pass the costs along to your customers, as is the american way.

Re:The Market Cycle

[ziggy_travesty]

1) You completely dodged the parent's point about selling products v. services.

2) Your "once upon a time" nonsense reads just like any other fairytale in that it is make-believe. The software industry was born when demand was created by the advent of PCs. It had nothing to do with a mythical band of hand-holding programmers. Keep selling your install services and numbing your mind. I'll keep selling software products.