Slashdot Mirror
Security Patch Creation at Microsoft
devonshire writes "Officials at the Microsoft Security Response Center have provided a detailed look at the process used to create security patches. From the time the first vulnerability data is received from grey hats to the time a bulletin is shipped, it's a pretty interesting look at how they handle the information flow and patch testing and why it takes so darn long to release an IE update."
New Windows worm circumvents Microsoft patching process
Instead of just believing the people that there is a problem, they have to test it out and develop a plan and then reprogram the piece. I hate that. In my company they have implemented such system too and if you have a problem you have to wait a month before it is planned in (if it is accepted by a group of non-technical managers) and then another month before it is fixed making a problem sometimes last for over 6 months and after an endless amount of pointless meetings there is finally some kind of fix. Programmers in corporation are under a lot of (time) pressure and that is not good as it makes them make mistakes. But they have to be able to make quick fixes (as is with most Linux projects) without any corporate meetings or managers.
Custom electronics and digital signage for your business: www.evcircuits.com
I don't think there's a single service on a windows box that can withstand a UDP flood. This has been known to be an effective DoS method for years...roommate using all the bandwidth with bittorrent? Playing Doom3 in the middle of the night with the volume jacked up?
Send a UDP flood to ANY of the services which are actively listening by default, problem solved. Where's the triage team on that one? I guess 99.9% resource consumption isn't a vulnerability in their eyes.
are you seriously suggesting you'd just release a brand new patch into the wild without even cursory testing?
who's going to want to install it? when everyone is a guinea pig, a certain reluctance to jump in first may manifest itself.
Screw you all! I'm off to the pub
Microsoft's non-security is well organised. :-)
It's not easy to test an IE update. There are six or seven supported versions and then we're dealing with all the different languages. Our commitment is to protect all customers in all languages on all supported products at the same time, so it becomes a huge undertaking. 1: languages shouldn't be a problem, that is (hopefully) not completely split up throughout the source code is it? aargh!!!! 2: I know only a 3 SUPPORTED IE versions (IE 5, IE 6 and IE 7)
Custom electronics and digital signage for your business: www.evcircuits.com
"This is exactly why it can take a long time to ship an IE patch. We're dealing with about 440 different updates that have to be tested [for different versions]. We have to test thoroughly to make sure it doesn't introduce a new problem. We have to make sure it doesn't break the Internet."
? ? ? ? ? ?
I won't join Slashcott. OTOH, If Beta goes live, I just won't be back until it's fixed. Sorry Dice.
I know the process!
1. Identify holes in current software
2. Release patches that only fix some of the holes
3. Start charging for tools to take care of the rest of the holes
4. Profit!
(If you're from Indonesia, no problem, the software will only cost $1 anyways)
Well, Microsoft does have Automatic Update working for them. They may have slower patch creation times, but they can push the created patch to you much more quickly. If you were a corporate executive, would you say that you'd rather immediately install an externally verified patch, or take your own company's time and resources to verify the patch? Sure, for large, computer-intensive operations like air traffic control or medical care, you'd need to verify the patch either way. But if it just means that a secretary wouldn't be able to play Solitare, and especially if your company doesn't have any individually-designated "Computer Security" positions, I think you'd install the patch right away. Also, it'd be ill-advised for an open-source shop to not regression-test patches before release anyway. I don't want to see the size of your Bugzilla database.
I mod down pathetic posts.
Microsoft makes security patches? And tests them too?
unable to resolve function slashdot.sig(), aborting...
Microsoft is adding a patch to a pair of jeans, but it's difficult because after all the previous patches the pair of jeans looks like a spherical ball of patches 10 feet in diameter.
Again, you keep saying how good OSS is compared to CSS. Now tell me, honest, if you write an application and someone tells you they can sell it for $100/copy and give you 50% of each. Would you still make it open-source? What you said is true, but I'm tired of everyone bragging about how "cool" OSS is. Yes, it's cool, but writing it isn't...
We have to make sure it doesn't break the Internet.
Don't worry, guys, no matter how badly you screw up it won't hurt the Internet - because the Internet doesn't run on Microsoft boxes. Hard to imagine, I know, but true.
Isn't the writing of Open Source software the whole point?
If no one wanted to write it, OSS wouldn't even exist.
Remember what ESR wrote about this? "If you treat your users as if they were your most valuable resource, they will respond by becoming your most valuable resource."
In other words, I think this is all about community-building, and I grant you that this may be beyond what you can do as a single developer who simply shares some code with the world. Still, I have found ESR's statement to be quite true in my own projects, and it only takes a small effort to express this attitude in the e-mails you send to your bug reporters.
Linux distro's have automatic updates too and the distro maintainer assumes the role of testing the application with the new patch applied.
The GP was only half-right by saying that 'a patch can be released right away and users can compile in the new sources themselves' is a strength of OSS. In reality only small numbers of users do this themselves, most simply get it through their distro's auto update feature after its been tested and qa'd by the distro maintainers.
Pre-canned Evolution Links for all those Slashdot holy wars.
real OSS projects actually have an organizational structure.. a closer knit group of users associated with the project will test and comment (or fix) problems they see with code. when the code seems to be good, it is released to the public as an actual release.
So what's different about that compared to the pre-release testers employed by Microsoft? not a lot, it may seem. Besides, my reading of the OP's post didn't indicate this was the meaning at all.
The fact is, going back to the OP's harebrained scheme, that no-one is going to apply a patch to a critical environment unless it's been through major testing. Sure, your l33t box under your desk which you rebuild every week anyway? patch it with whatever you like, but a production database server pushing out data to thousands of clients? I want that bastard tested thoroughly before the patch ever hits the net.
Screw you all! I'm off to the pub
We have to make sure it doesn't break the Internet [web access provided by IE, which as far as our customers go means breaking the Internet]
The Internet wouldn't be broken as such, but I doubt the users would see it that way. To them, it doesn't matter if it is the browser, the connection or the servers (massive worm?) that is broken. They can't do what they want, hence it is broken. It is as simple as that.
Kjella
Live today, because you never know what tomorrow brings
Which is basically a fancy way of saying you're going to treat your user base as guinea pigs and let them test your patch for you.
Hopefully any "issues" they have will not have been fatal...
Any sect, cult, or religion will legislate its creed into law if it acquires the political power to do so.
"It's not easy to test an IE update .... We have to make sure it doesn't break the Internet."
Sometime a joke doesn't need a punch line.
As an Open Source developer, I'm not in this for the money. If I were, you can bet the project would be Closed Source.
Rather, I want this project to be open and usable for all. To that end, I license it under the GPL and anyone is free to use it.
So my users are partners with me. They are not my guinea pigs. Though I maintain control over the project, there is no set-in-stone law that no one else may fork the project. In fact, they are encouraged to, if they feel it necessary.
I release the patches, and they accept them or reject them, depending on their own circumstances. I don't rule them with an iron fist. I consider them my Knights of the Round Table where they all have the right to say what they want and none is any greater than the other.
So maybe you think that users are passive slugs, but I'd rather give them the benefit of the doubt.
"It's not easy to test an IE update .... We have to make sure it doesn't break the Internet."
.... We have to make sure it breaks Firefox and Opera."
Here I fixed it for you.
"It's not easy to test an IE update
Better
Was I talking about IE? Was the OP? Surely we were debating the patch process in general, not specifically IE?
Besides which, a hell of a lot of corporates consider their intranet (extranet/web) apps 'critical'. IE (or other browser) is a major component in that mission-critical situation, wouldn't you say?
Screw you all! I'm off to the pub
Customers Complain About a New Security Hole. The number of complaints reaches Management's "Action Threshold". The Patch Process is started.
1. First, blame the customers' other software packages for the insecurity.
2. Then, blame the customers' for failing to apply services and hot fixes in a timely fashion.
3. Security Focus (or another of the Sec/Priv sites) calls up and threatens to "out" the hole if it isn't fixed.
4. Accuse the complaining entity of having a "partisan" agenda against your company, initiate "Four Corners" Stall -- while you try to figure out if you actually CAN patch the damn thing
5. As the news of the new exploit makes it into IRC and the UGs' Forums you issue an indignant press release stating that it has never been proven that the new exploit has even been used and is principly "theoretical".
6. As your Patch Team frantically works to get the patch out, explain that even though the chances of this exploit being used WERE previously slender or non-existent, now that some details of the exploit have been malicously leaked, HEAVY SIGH, you'll now go ahead and fix it.
7. Issue the patch, take credit for being "Right on Top" of security issues, explain how much money and time you are spending to counter the effects of the "Few Bad People" on the Internet.
8. News start to come in that your patch has broken a number of somewhat older apps -- explain that Users have a responsiblity to use "current" software products and refer them to Sales.
9. News of another exploit comes in --GOTO 1
BTW, this is pretty much AN INDUSTRY STANDARD APPROACH
In Commerical Software, it's the FEW companies that DON'T do some version of this that are the (delightful) and unfortunately RARE exception.
Ten quid, she's so easy to blind. And not a word is spoken...
My experience directly contradicts this on all points.
When I reported the hyperthreading security flaw to Microsoft, I was provided with the first name of the person who was responsible for dealing with it ("Christopher"), but I was not provided with his last name, phone number, or any e-mail address (apart from the generic secure@microsoft.com address which I used to report the problem). Later the issue was transferred to "Brian" -- again, no last name, no email address, and no phone number.
Over the following two months, I heard from three independent third parties that Microsoft was "very concerned" about this issue, and had "several people" looking at it; but they never made it clear that they wanted to work closely with me -- in fact, they ignored all my attempts at co-operation.
Finally, prior to releasing my paper, I sent several emails to Microsoft asking about their progress and asking for a vendor statement for my web site; again, they did not respond.
Tarsnap: Online backups for the truly paranoid
...purely political.
Microsoft wants to give you one "bad news" per month. Predictable, patch time is "low" meaning the time between release and installation is low. It is easy for IT staff to work that way, you can schedule it.
OSS will give you a patch per issue, patch time is near instant, but they keep coming at you all the time, whenever you can't afford to waste time installing them. That is why you need a distro to keep you patched at all times.
The rest? Bullcrap. The security patches for Linux don't cause more regression issues than Windows. Like Microsoft, they do audits but instead of one "catch 'em all" release, they do several. In short, it is to make Windows look good.
Kjella
Live today, because you never know what tomorrow brings
You can sell OSS.
Thanks for mentioning the pros of Open Source. I agree, but that's not the point.
Even OSS developers do some testing before they release their code. At least for the larger (multi-developer) code bases. Quality is essential if you don't want to scare your users/co-developers away. And quality is only partially a result of programming skills.
Now you may point at the difference in emphasis between informal release-testing and formal QA in the legal sense. But it's just rediculous to assume that OSS solves everithing to the point where you just merge & release everthing you type and/or every patch submitted to you without even looking at it.
--
It is impossible to make anything foolproof because fools are so
ingenious.
If Debian isn't the epitome of an Open Source project that's overly obsessed with quality releases, at the expense of frequent releases, I don't know what is.
I write code to accomplish what I intend, and I succeed. I don't need to test. What needs testing is other peoples' crappy code that my code depends on. I'm looking at you, GW BASIC maintainers!
For 90% of people, the web is the internet.
For 88% of them, the internet is IE.
Which means that 79.2% of people think that the internet is IE.
My Journal
are you seriously suggesting you'd just release a brand new patch into the wild without even cursory testing?
:)
You can always release a patch to the patch if any problems are found with it
But seriously, it makes most sense to correct most bugs (that will be caught in the short-term) before a wide release, where there is a single copy of the source, rather than after release, where there are as many copies as there are users.
With open-source anybody is free to provide this service. If the author only has the time/motivation to do barely-tested releases, why reject his code? Someone else with the desire can do testing and make releases to a wider audience that are more stable, and users can choose between the two options (or more). These can even form without any direct arrangement between the various parties.
Thanks for posting that, I'd mod you up if I had points. Which, typically, I don't.
Open source doesn't eliminate the need for testing, but it can make it easier, and specifically make it easier for knowledgable users to fix bugs themselves and contribute back. As for the testing release issues, it wouldn't be much more trouble for closed-source systems to release nightly builds to the world to test, just less tempting to test.
The fact that users can fix bugs themselves, though, is not an excuse for releasing buggy software. By all means give users who want a bleeding-edge release access to your newest and greatest (but maybe not quite fully-tested) code, but don't go around releasing such code as your official version. Give it some time, test it a bit, before putting that out. Just because people can bug test and fix their own software doesn't mean that they should be made to.
OSS can make testing easier, but it does not, as you point out, remove the need for it. For anything above a "hobby" project, for things you actually expect people to use, it's just irresponsible not to undergo at least some testing. Overuse of "caveat emptor" just makes OSS look unprofessional -- which is fine, but it could cause problems when trying to break into more corporate grounds. The people who say both that companies should use more OSS, and that OSS doesn't need to be tested, really need to re-evaluate at least one of those viewpoints.
I sense I'm ranting, so I'll stop.
clearly, there are many different types of software users... from those that actively contribute to it's code, to those that test out the latest versions and report bugs, to pure users that just want to use your tool to get their own stuff done.
most users fall in the last category and they'll quickly jump ships if your stuff is too buggy/unusable and/or there's something better out there user-wise... case in point, firefox, where the majority of the 30+ million downloaders were not open-source contributers but rather software users that found something better.
but hey, if you're just interested in chucking untested code out there for your "partners" then more power to you... this "passive slug" will be supporting more serious projects.
Although teorethicaly it is possible to sell OSS, it is not proffitable.
Why would someone want to buy something he can download for free in other place?, if people tend to "download for free" something that they CAN NOT (by law) use for free??
Of course, now you will tell me that RedHat, Mandrake, etc etc are making buisness with OSS, but the truth is they are making buisness SELLING SERVICES, not the software.
Now, I am a programmer (well, I was a programmer before I started my PhD), I really like to program, when I was in the University I was a Linux advocate (although when I was in High School I was a FreeBSD advocate... can you imagine I bought FreeBSD without really knowing what was it... then when it arrived I spent like 3 weeks installing it, I was like 13 or something).
But, after I finished the University I had written some programs which I wanted to sell, hell I DO know how to program...
I put them like shareware on the internet, it was cool, but I also wanted to "contribute" to the OSS, in the "real world" (i.e. outside the net in my life) I was trying to get a job, As I lived in Mexico that was no easy task, so all my income was from my shareware programs and some money my parents gave me.
But I WANT to program for a living, and that is NOT possible with OSS, only people who have a name and are at the top position in this "OSS" power hierarchy can do it.
There where possibilites of open sourcing my programs and then proffiting with the "customer" services, of course the money I would get there was going to be a hell less than the money I won with my shareware (which was not a lot of course) and besides I DID NOT studied any kind of administration or client service degree I AM A FUCKING PROGRAMER and I want to program because THAT IS WHAT I KNOW HOW TO DO!!
So no, it is not possible to live selling OSS, it MAY be possible to live selling a service but not by pure development.
And of course it is possible to get hired in a company which develop open source as a branch (IBM, Sun, Mandrake, etc) and you could say that you earn your living with OSS... but the one that is paying you is the company.
Nowadays I am making my PhD outside Mexico (no, not in the US, in Europe). I have a wider view of this OSS, and althouh I understand it is great for acadamey (in fact I OSS it every day) It is NOT right for the commercial developer... And now as I have seen the Programming buisness is very crowded I have decided to enter the academy buisness, that way when I return to my country with a Europe degree I would be able to enter and teach somewhere at least...
And, I will be able to use and create OSS (of course as a side project JUST FOR FUN). At the end, that is why the OSS projects propsere, people do them JUST. FOR. FUN.
Ubuntu is an African word meaning 'I can't configure Debian'
"Oh, it's ok, we'll release a patch instantly and the users can review/compile it themselves."
I don't know about you, but I have things I actually want to _use_ my computer for - I don't want to have to review any code changes for patches/upgrades/new versions and check them before I do an install.
Not that I even have the technical know-how to do that for the vast numbers of programs out there.
My Journal
It's closed source; Closed architecture; Closed development processes. They could be throwing code together like monkeys and making all this stuff up for the PR value. Who knows?
I'm not trying to flame-bait here, either. These are the simple facts. Flame-baiting would be saying something like: Haven't they always boosted their value via PR and under-delivered? Or: Doesn't Microsoft lie like sacks enough for you to notice?
That would be flame-baiting. But I'm not flame-baiting.
got laid in high school, do you think there'd be a Microsoft?
Of course not.
You got to spend a long time stuffed in your own locker with your underwear wedged up your ass before you start thinkin "I'm going to take of the world with computers! you'll see, I'll show them."
Your Average Joe
Once upon a time, musicians gathered in groups and performed on street corners -- just for fun. Often they'd drop a hat, so passers by could show their appreciation. Sometimes they could put on whole performances, rent space and charge admission. Once in a while, they could play for their king and make real money.
Then the record industry was born. Now a song could make a musician a steady stream of money, for many years. However, after decades of "success" the public saw through this sham and invented ways of putting the right perspective on the value of music and performance. And the musicians returned to being performers because the former era was over.
Actually, that's not how the story ends because the rich benefactors of the record industry used their money to create laws to enforce their way.
Once upon a time, computer programmers gathered in groups to share ideas and collaborate on projects -- just for fun. Often they would solve some incredible problem and get recognition for it. Sometimes they'd get paid hourly to solve a specific problem. Once in a while they'd get real funding.
Then the software industry was born. Now an application could make a programmer a steady stream of money. However, after decades of "success" the public saw through this sham and invented ways of putting the right perspective on the value of software and applications.
Actually, that's not how the story ends. It'll be a while before we get to the end.
I sell lots of open source software. Very little of this software have I written. It's easy for a software-savvy person to download and install OSS applications. It's difficult for the majority of the people on the planet to understand how to download and install any application. That's what I charge for.
You probably wouldn't believe how many times a week I'm asked to install CSS applications. These are packaged products that should be easy for anyone to install. Yet your average business owner and their entire staff are intimidated by the prospect of having to install any application (OSS or CSS) -- they'd rather visit the dentist.
Think about it: For CSS applications, the end user often pays twice.
Can a programmer with 20+ years of experience make good money with OSS? I do.
The fact is that no-one is going to have a critical environment that uses IE
Really? Thats great news! I'll just go and tell my boss that none of our web based apps are "mission critical". He'll be thrilled that we don't have to worry about them anymore.
"I realise this is not a very popular opinion but it's the truth, and there for needs to be said" -Bill Hicks
This is why concise, clear and well documented modular programming is a winner. Even Firefox suffers this. It's a huge mess of code that a handful of people could even be bothered to read...
In microsofts case everything has to be implemented upon layers of undocumented C++ classes to which the average microsoft employee [let alone third party developer] can't decode.
Tom
Someday, I'll have a real sig.
Some people submit a vulnerbility report to the brickwall called Microsoft Support. Then after 6 months they release a security opdate. And now they call the submitters "Grey hats"? What do they call themselves? The "Pink hats"?
:-) = I am happy
:^) = I am happy with my big nose
C:\> = I am happy with my OS
Um, the GPL doesn't say that you have to give your code free to everyone on the planet.
It says that you have to give your code free to anyone you sell the binary to... *if* the person ask for the code.
so a company using internal GPL'd code does NOT mean that their code will be avaliable to their competitors, unless they sell their product to their competitors.
multifariam.net -- yet another nerd blog
He's close, but not spot on; customers demand quality software, but are forced to deal with faulty programming and broken applications. Customers wait for 'quality' patches, and deal with the associated trouble of a system that's broken-in-the-meantime. But hey, we've got fade-out windows and drop shadows, and some really neat animated assistants, so I really shouldn't complain?
perl -e "eval pack(q{H*},join q{},qw{70 72696e74207061636b28717b482a7d2c717b343 637323635363534323533343430617d293b})"
Dilbert: As part of your ISO 9000 certification do you have a defined pathing process and what is it?
Elbonian Balmer: We hold a village meeting where we boast of skill and curse the devil spawned enduser.
Elbonian Gates: Sometimes we Juggle.
Elbonian Balmer: The at the last second we slam out some code and go roller skating.
What always cracked me up about that joke is that it is defined process and therefor meets the requirements of iso 9000 certification.
Some drink at the fountain of knowledge. Others just gargle.
Maybe you can't but others certainly can, and if you are so inclined, you can learn.
Also the fact that code is open makes the authors more careful, i would think. If I am going to publish code with my name on it, I would hope it doesn't suck.
Closed source could have terrible code style and use all sorts of hacks, and no one would know. Or it could be perfectly written using Hungarian notation. With OSS, if you have bad code, people who can read it, will, and tell others.
Besides, if you want, just do a search for printf and gets in the code -- you might find some bugs w/o having to write a thing.
sigh. Why is it that when people can't figure out how something is done, they simply say "it CAN NOT be done"?
Firstly, let's get something clear: hardly anyone makes money simply selling software. A perfect example is databases -- for all but the high-end database projects, a free database works just as well (sometimes better) than a commercial, closed-source DB. Yet, people still buy MS-SQL server, and Oracle, and the like for even small projects. Why? They are buying the support of MS and Oracle: not just the telephone support but the "this large company has vetted my software" support. They are buying trust and service.
Now that that's clear, let me explain that I make money by selling OSS solutions, and that RedHat and Novell make money from my work. I contract as an OSS developer/integrator. I sell my development ability and support. But, my clients buy Linux from Novell or RedHat; they are getting support from me, so why would they buy these OS, when they can be had for free?
The answer is simple: people (and to a greater extent, corporations) see value in something they've paid for. If something happens to me, they know someone will stand behind the product. They know that someone they've paid is working on security patches and improvements. And, ultimately, they know the product is less likely to be abandoned.
So, when my clients buy Linux from RedHat, they are buying exactly the same thing as when they buy Windows from Microsoft: trust. Trust that the software has some degree of quality, trust that it will be patched and maintained, and trust that it will continue to be available. With OSS, however, they get the bonus of knowing that migration to another vendor will be relatively painless because the vendors of OSS software have access to each other's code.
It is possible to make money with OSS, but it is a lot harder to start your own OSS business. People don't like buying software (closed *or* open) from one-person organizations.
We may not imagine how our lives could be more frustrating and complex—but Congress can. – Cullen Hightower
I think you are responding more with anger than with logic. Firstly, whoever did your deployment of firefox, should have tested it before he went to everysingle machine and deployed the update, this is called quality contorl/damage contorl. secondly it is very easy to remoe firefox, and install whichever version you need. From what i gathered in your statement, you are claiming you have never had any down time or senseless tech cycles put towards removing spyware or malware on any of your computers. I do tech support as a consultant for about 20 small businesses. this is by far the most common phone call i receive, "my computer is broken i cant get past these pop up adds, internet explorer keeps crashing its really slow and i cant get my work done" now there are some malwares and spywares you cant get rid of, i've reimaged machines after several hours of attempting to remove some of the newer variants. now let me ask you where did you save the time, and money? was it from the extended hours of firefox, in a deployment cycle, (seriously this should take moments to install and uninstall)? If you think i am exaggerating call Dell or any other computer support company and ask them the number one call they receive, it isnt that firefox isnt working its their entire os, to which they respond put in your restore disk, so they can keep a profit margin. Im not a fanboy, but i do see the weakness behind Internet Explorer, and the fact that microsoft didnt update a thing until they lost ground to Firefox (ie: they had to protect their name) seriously redo your math, and figure out, where your costs lie, if you think the only response is hiring a unix/firefox coder to analyze and fix firefox code, then your techs are incapable, or just plain idiotic,or you should cease doing your own tech cos you are doing more damage than good.but i suppose you just pass the costs along to your customers, as is the american way.