Security Patch Creation at Microsoft

devonshire writes "Officials at the Microsoft Security Response Center have provided a detailed look at the process used to create security patches. From the time the first vulnerability data is received from grey hats to the time a bulletin is shipped, it's a pretty interesting look at how they handle the information flow and patch testing and why it takes so darn long to release an IE update."

Comments

Next week's headline:

[Anonymous Coward]

New Windows worm circumvents Microsoft patching process

Typical corporated programming

[guruevi]

Instead of just believing the people that there is a problem, they have to test it out and develop a plan and then reprogram the piece. I hate that. In my company they have implemented such system too and if you have a problem you have to wait a month before it is planned in (if it is accepted by a group of non-technical managers) and then another month before it is fixed making a problem sometimes last for over 6 months and after an endless amount of pointless meetings there is finally some kind of fix. Programmers in corporation are under a lot of (time) pressure and that is not good as it makes them make mistakes. But they have to be able to make quick fixes (as is with most Linux projects) without any corporate meetings or managers.

Re:Testing is only a priority on closed source app

[Atrax]

are you seriously suggesting you'd just release a brand new patch into the wild without even cursory testing?

who's going to want to install it? when everyone is a guinea pig, a certain reluctance to jump in first may manifest itself.

IE is the internet?

[gd2shoe]

"This is exactly why it can take a long time to ship an IE patch. We're dealing with about 440 different updates that have to be tested [for different versions]. We have to test thoroughly to make sure it doesn't introduce a new problem. We have to make sure it doesn't break the Internet."

? ? ? ? ? ?

Pick me, pick me!

[Infinityis]

I know the process!

1. Identify holes in current software
2. Release patches that only fix some of the holes
3. Start charging for tools to take care of the rest of the holes
4. Profit!

(If you're from Indonesia, no problem, the software will only cost $1 anyways)

Re:From the article:

[N3Roaster]

You missed the funniest bit:

This is exactly why it can take a long time to ship an IE patch. [snip] We have to make sure it doesn't break the Internet.

So, the next time someone tells you, "The Internet is broken," you can just blame Microsoft for putting out an IE patch too quickly.

The Big Blue E

[value_added]

"It's not easy to test an IE update .... We have to make sure it doesn't break the Internet."

Sometime a joke doesn't need a punch line.

Re:Testing is only a priority on closed source app

[Dancin_Santa]

As an Open Source developer, I'm not in this for the money. If I were, you can bet the project would be Closed Source.

Rather, I want this project to be open and usable for all. To that end, I license it under the GPL and anyone is free to use it.

So my users are partners with me. They are not my guinea pigs. Though I maintain control over the project, there is no set-in-stone law that no one else may fork the project. In fact, they are encouraged to, if they feel it necessary.

I release the patches, and they accept them or reject them, depending on their own circumstances. I don't rule them with an iron fist. I consider them my Knights of the Round Table where they all have the right to say what they want and none is any greater than the other.

So maybe you think that users are passive slugs, but I'd rather give them the benefit of the doubt.

Re:Pick me, pick me!...Alternate Patch Process

[darkPHi3er]

Customers Complain About a New Security Hole. The number of complaints reaches Management's "Action Threshold". The Patch Process is started.

1. First, blame the customers' other software packages for the insecurity.

2. Then, blame the customers' for failing to apply services and hot fixes in a timely fashion.

3. Security Focus (or another of the Sec/Priv sites) calls up and threatens to "out" the hole if it isn't fixed.

4. Accuse the complaining entity of having a "partisan" agenda against your company, initiate "Four Corners" Stall -- while you try to figure out if you actually CAN patch the damn thing

5. As the news of the new exploit makes it into IRC and the UGs' Forums you issue an indignant press release stating that it has never been proven that the new exploit has even been used and is principly "theoretical".

6. As your Patch Team frantically works to get the patch out, explain that even though the chances of this exploit being used WERE previously slender or non-existent, now that some details of the exploit have been malicously leaked, HEAVY SIGH, you'll now go ahead and fix it.

7. Issue the patch, take credit for being "Right on Top" of security issues, explain how much money and time you are spending to counter the effects of the "Few Bad People" on the Internet.

8. News start to come in that your patch has broken a number of somewhat older apps -- explain that Users have a responsiblity to use "current" software products and refer them to Sales.

9. News of another exploit comes in --GOTO 1

BTW, this is pretty much AN INDUSTRY STANDARD APPROACH

In Commerical Software, it's the FEW companies that DON'T do some version of this that are the (delightful) and unfortunately RARE exception.

Liars

[cperciva]

Quoth the article:

We respond immediately to the initial vulnerability report and provide the researcher with contact names, e-mail addresses and phone numbers. We make it clear we want to work closely with the researcher to pinpoint the problem and get it fixed. We commit to providing [researchers] with a progress report on the Microsoft investigation every time they ask for one

My experience directly contradicts this on all points.

When I reported the hyperthreading security flaw to Microsoft, I was provided with the first name of the person who was responsible for dealing with it ("Christopher"), but I was not provided with his last name, phone number, or any e-mail address (apart from the generic secure@microsoft.com address which I used to report the problem). Later the issue was transferred to "Brian" -- again, no last name, no email address, and no phone number.

Over the following two months, I heard from three independent third parties that Microsoft was "very concerned" about this issue, and had "several people" looking at it; but they never made it clear that they wanted to work closely with me -- in fact, they ignored all my attempts at co-operation.

Finally, prior to releasing my paper, I sent several emails to Microsoft asking about their progress and asking for a vendor statement for my web site; again, they did not respond.

Re:Liars

[cperciva]

Despite what the article says, what do you think Microsoft owes you in this case?

Nothing. However, I do believe that they owe the public, and their shareholders, the truth about how they handle security issues -- which, judging by my experience, they did not provide in the linked news article -- and I believe that they should take every opportunity available to improve their security, including working with the people who report security issues to them.

You are an academic nobody in their eyes, despite any delusions of grandeur you may possess.

Maybe; or maybe not. I'm not just an academic who happened to stumble across a security problem; I'm also a FreeBSD deputy security officer. I may not have quite as much experience at dealing with security issues as they have, but I don't think I'm a complete "nobody" in security circles either.

Re:Testing is only a priority on closed source app

[xtracto]

Although teorethicaly it is possible to sell OSS, it is not proffitable.

Why would someone want to buy something he can download for free in other place?, if people tend to "download for free" something that they CAN NOT (by law) use for free??

Of course, now you will tell me that RedHat, Mandrake, etc etc are making buisness with OSS, but the truth is they are making buisness SELLING SERVICES, not the software.

Now, I am a programmer (well, I was a programmer before I started my PhD), I really like to program, when I was in the University I was a Linux advocate (although when I was in High School I was a FreeBSD advocate... can you imagine I bought FreeBSD without really knowing what was it... then when it arrived I spent like 3 weeks installing it, I was like 13 or something).

But, after I finished the University I had written some programs which I wanted to sell, hell I DO know how to program...

I put them like shareware on the internet, it was cool, but I also wanted to "contribute" to the OSS, in the "real world" (i.e. outside the net in my life) I was trying to get a job, As I lived in Mexico that was no easy task, so all my income was from my shareware programs and some money my parents gave me.

But I WANT to program for a living, and that is NOT possible with OSS, only people who have a name and are at the top position in this "OSS" power hierarchy can do it.

There where possibilites of open sourcing my programs and then proffiting with the "customer" services, of course the money I would get there was going to be a hell less than the money I won with my shareware (which was not a lot of course) and besides I DID NOT studied any kind of administration or client service degree I AM A FUCKING PROGRAMER and I want to program because THAT IS WHAT I KNOW HOW TO DO!!

So no, it is not possible to live selling OSS, it MAY be possible to live selling a service but not by pure development.

And of course it is possible to get hired in a company which develop open source as a branch (IBM, Sun, Mandrake, etc) and you could say that you earn your living with OSS... but the one that is paying you is the company.

Nowadays I am making my PhD outside Mexico (no, not in the US, in Europe). I have a wider view of this OSS, and althouh I understand it is great for acadamey (in fact I OSS it every day) It is NOT right for the commercial developer... And now as I have seen the Programming buisness is very crowded I have decided to enter the academy buisness, that way when I return to my country with a Europe degree I would be able to enter and teach somewhere at least...

And, I will be able to use and create OSS (of course as a side project JUST FOR FUN). At the end, that is why the OSS projects propsere, people do them JUST. FOR. FUN.

Re:Right

[DogDude]

I do run a business, in fact. And yes, I could pay somebody a small fortune to review patches for me. With most applications, TCO is already down the toilet just with the time it would take to *find* somebody who could do it, never mind actually paying the person. Case in point... the last Firefox upgrade broke all of our machines (Firefox quit working on all of my machines... I hope that was all that was effected). IE has never done that. Insignficiant program, true, but what am I supposed to do... hire somebody to review each of Firefox's releases to tell me whether or not they'll work? Am I supposed to spend, what, $10-20K to have a Unix programmer come in to analyze the latest Firefox build and tell me where the problem is? That's insane. Instead, we simply removed Firefox from all of our machines, and went with IE, which was already properly tested before being pushed out to users. Much cheaper. Much simpler. Much quicker time for me to get back to the core of my business (which trying to get broken web browsers to work).