Slashdot Mirror
Microsoft Exploit Predictions Right 40% of Time
CWmike writes "Microsoft today called its first month of predicting whether hackers will create exploit code for its bugs a success — even though the company got its forecast right just 40% of the time for October. 'I think we did really well,' said Mike Reavey, group manager at the Microsoft Security Research Center (MSRC), when asked for a postmortem evaluation of the first cycle of the team's Exploitability Index. 'Four of the [nine] issues that we said where consistent exploit code was likely did have exploit code appear over the first two weeks. And another key was that in no case did we rate something too low.' Microsoft's Exploitability Index was introduced last month."
That's great, guys, but don't you think being proud that you were right about your code being exploited is... backwards? That's like being proud you correctly predicted you would get stabbed while walking through a ghetto wearing gang colors.
Then again, this is Microsoft. They probably throw an office party every time something compiles without errors.
=Smidge=
A little heavy on the false positives but no false negatives so it allowed more efficient targeting of the risk areas. Also good enough to provide useful feedback.
Well, that's like saying, after you block all your email from getting through, "We rated all the spam accordingly, and let none of them through".
How about, we just guess, a rough fucking guess, that any "remote code execution" or "run with elevated privileges" exploit or hell ANY GOD DAMN FUCKING BUG YOU FIND, needs fixing, right Microsoft?
...
Any engineer who says that "40% is pretty good predicting" is incapable of writing good software, or managing a project, or, even, applying the scientific method.
Hint: 40% is worse than guessing.
How we know is more important than what we know.
Interestingly what they are saying here is that they think that
a) Hackers are smarter than they actually are
b) Microsoft code is easier to exploit than it actually is
So the perception is that Microsoft is better than their prediction, but the implication of that is that Microsoft think they are rubbish.
Maybe all these years of "Microsoft sucks" posts on Slashdot have actually come from the MS security team.
An Eye for an Eye will make the whole world blind - Gandhi
Actually that was John Cleese, even posting anon you should give credit where its due.
Most of its correct though, even if it is as old as your mom :)
...
Only 40%, which is already "a success", but they can improve this score, and this would become a triumph !
Actually that was John Cleese, even posting anon you should give credit where its due.
Actually it originated with One Alan Baxter of Rochester and expanded by other people on Usenet. So if you do give credit where it's due give it where it's actually due.
Microsoft Exploitability Threat Level Indicator is a series of color codes starting from Dazzling Arctic White to Heart of Dick Cheney. Though exact number of these colors is considered a secret, from the past announcements we deduce there are at least 22 million of them.
For PRNewswire, copy edited by Anurag Chakraborty in Bangalore and supervised by Robert Zimmermann in Pittsburgh.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
Research also shows Slashdot editors verify submission figures 112% of the time.
"But our metrics said we were save!"
"Windows 7 now ranked no 1 in unexploitability with 1723 unexploitability points ahead of Debian"
Am I the only one who thinks this talk starts we people don't understand the matters that they are dealing with?
sounds like a yes or no question... won't flipping a coin give you a 50% success rate?
So ... Are they admitting that bugs in their software that are being targeted by crackers are there by design? Or just incompetence?
If they know their software is filled with bugs, why not just fix them and be done with it before it's released.
"Suppose you were an idiot...and suppose you were a member of Congress...but I repeat myself." Mark Twain
there is so many to chose from...
That only puts it slightly below random guessing. Great work guys!
...is the same as being wrong 60% of the time.
Doesn't look so impressive when you look at it this way.
Smalley as their Senator? Is there anything more pathetic than a bitter old self-hating Jew?
While i do not work for Microsoft (but i do work for another multinational corporation), it seems to me many of the exploits that Microsoft 'finds' in their products are emailed to them. For those who have not heard of the standard story before:
1. Regular Joe the geek finds an exploit on a product
1a. If Joe is malicious, he will exploit this right away. However, this seems to happen very rarely as most people have at least a bit of a conscious.
3. If Joe is not malicious, he will keep the exploit to himself, do some research, and email the details to the company.
4. Joe waits for a reply and a fix. Since the exploit seems so serious, he is at least expecting a fix within a month or so.
5. Joe waits one or two months, without getting much more than a standard response.
6. Joe grows tired of waiting. He will start surfing to see how other people deal with these issues
7. Joe publishes hints of the vulnerability. If he leaves a good trace back to himself, the company or the law might get involved.
7a. Some script kiddie or someone else might pick up on the vulnerability and start exploiting it.
7b. The company might actually have a fix ready by now (6 months to two years later).
7c. Nothing happens.
8. Joe either is thrown in jail, sued to extinction, or forgets about the whole thing.
Any way you stick it, Joe is trying to do the company a favour, but only can get crap in return.
From the company perspective, it is certainly possible to predict exploits if the company just gathers all the information available to them.
Posting anonymously because I know too many "Joe"'s.
...predicting exploitation of vulnerabilities.
This studies which vulnerabilities should be prioritised for patching. It is not studying which code will have a vulnerability as some here seem to think.
I use Linux anyway.
Granted, they're doing better than guessing... but in reality, I only care that they get it right on the risks that count. They could be 1 for 10, if the harm that the single exploit would cause was more than the sum of the other 9, and be doing decent.
For instance, if they patched the priv. escalation to SYSTEM that has a broad surface area (think, say, remote IIS exploit) over 9 exploits that require physical access and can only get guest access. If someone else has physical access to your box, it's no longer yours, anyways. Risk assessment has to account both for the opportunity and consequences of a given security hole.
If I mod you up, it doesn't necessarily mean I agree with what you've said, sorry.
Is this code part of Microsoft's code base?
Yes - it is exploitable
No - it is probably not exploitable*
* This accounts for the 60%
/bow to your knowledge
Afaik that was where it was from, thankyou :)
...
Microsoft is ignorant.
If hackers can exploit source code THEY HAVEN'T SEEN, wait until more real source is revealed.
Microsoft is an ignorant company sitting on top of a shell of a business.
60% of the time it's wrong, every time.
Ahh, here we go.
http://www.snopes.com/politics/satire/revocation.asp
More exciting than reading about how badly microsoft can classify security bugs eh? :)
ps. NO FIREFOX, I WILL NOT CAPITALISE THE "M" IN mICROSOFT!
...
Thank God for a sensible post! Better than that guy eating shit.
That means that forty percent of the time, it works every time.
Sorry guys, I know it's off topic, but I couldn't help it. :)
Well, I for, one, welcome our new stiff-upper-lipped, bland food eating, emotionless British overlords!
My blog
Microsoft is now bragging about the fact that they predicted 40% of their bugs would be turned into exploits?
I realize that Windows is a complex hunk of crap...errr...operating system, but wouldn't they be better served trying to find and correct these issues rather then just releasing them into the wild and keeping their fingers crossed?
Their attitude is sort of like pointing the gun at your foot and firing five times, and bragging that you only hit two of your toes.
This is why, every day when I arrive at work, I log into this XP box and ask myself why my organization continues to put up with this garbage.
Joe Dougherty, Florida, USA
The words I thought I brought, I left behind. So, never mind.
We would be delighted to become subjects of the crown again, but we doubt that her majesty could afford all the cameras that her subjects are so accustomed to.
PS. The Irish make better beer than you do, and soccer still sucks.
This is my sig.
I've decided to start my own Exploitability Index & my main selling point will be that I will be right 50% of the time compared to Microsofts mere 40%.
Wanna fight ? Bend over, stick your head up your ass, and fight for air.
So microsoft predicted these with some accuracy, and used this to give priorities to it's developers.
Jolly good, but apart from perhaps knowing that microsoft is trying to do a good work, why should I care about the process? This is a microsoft internal thing.
I can't bear anymore stupid executives bullying to the world about their stupid internal work. Do your homework and shut up. I really don't care how they do their work, just do it.
P.S. in microsoft developers are definetly not doing good work...Marketing instead has always been doing a great work there.
With the exception of points 7. and 9. it all seems quite reasonable.
Maybe one day you'll learn to drive on the right side.
And vinegar is acceptable on salad only, not potatoes.
By the way, I live in Continental Europe and my ancestors, at the time you were wearing animal furs and piling rocks in bizarre patterns, were building aqueducts.
So, in the end, Her Majesty, please
1. learn to drive
2. learn to cook
3. understand that fox hunting isn't a sport
4. stop using that absurd currency that is the pound sterling
Kudos to Microsoft for choosing to toss the coin rather than to spin it. My research team argued for months with the board that spinning a penny instead of tossing it results in heads only about 30% of the time.
"I guess I'm gonna fade into Bolivian."
Has there ever been a Microsoft bug that hackers have left alone?
We've been through this 'a million times' since DOS; there are literally more than a million active viruses out there, with another 100,000 per month. 40 percent chance of an exploit being used seems kinda low, doesn't it?
--- For a good time mail uce@ftc.gov
What REALLY happened is this: Every security hole that MS discovered on its own, was exploited BUT we are supposed to be happy because in 40% of the cases MS correctly predicted that it would be exploited.
Let me translate. I left fish out on the kitchen counter, EVERY single time it was eaten by my cat, but don't worry, I predicted he would correctly 4 times. Ain't I a genius.
This prediction ain't just bad by MS, it is idiotic. EVERY SECURITY HOLE WAS EXPLOITED. As you would expect it to be. Then claiming that you are smart by claiming you saw this coming 40% of the time is meaningless.
Coin tossing. If flip a coin and predict 40% of the time that it will land face side up, nobody would assume you have any special powers, it would just be down to randomness. Harmless and useless. But this is not what MS predicted. MS predicted that 40% of the time a coin flipped into the air, would come back down. That ain't a good guess, it shows that MS has a fundemental lack of understanding software security. Then again, that is hardly going to come as a suprise to anyone who has been following them for the last 2 decades.
MMO Quests are like orgasms:
You may solo them, I prefer them in a group.
Predicting something will change the outcome in this case.
Not only will hackers know the most exploitable bug but they will also know the least likely to be updated.
Yet another bloat to innocent users.
FFS can someone make them a feature usefulness vs crap predictor..
Oh, we can drive on the right. We just choose to drive on the left so we can defend ourselves with swords. Comes in useful oh-so-often.
As a Brit, I'd be happy to get rid of Association Football, but replacing it with American Football? Ice Hockey would be my choice. Or Rugby Union. Or even Cricket.
I have to admit that whenever I've had food cooked by an American, it has been the blandest rubbish I have come across. Modern British cooking for me, any day.
What's absurd about pounds? Unless you're referring to pounds, shillings and pence.
With such low standards of good performance, it is no wonder that the software coming out of Redmond lately has been so horribly poor.
Hackers are missing 60% of opportunities to exploit Microsoft code.
"This month, we're going to predict whether evil hackers will exploit bugs in our code. What do you predict?"
Steve Ballmer: "No."
James Allchin: "Yes."
Mike Reavey: "Yes."
Jim Gries: "No, I fixed all the bugs."
Sarah Ford: "I dunno. I'd say no; I'm confident in Microsoft."
Val Mazur: "No."
Rui Chen: "Well, the possibility is there, but they'll never prove that they did, so it's the same as no."
Kathleen Dollard: "Of course I will! er --I mean, THEY will. Yes."
Michel Fournier: "How am I supposed to know? How many people said Yes so far? Oh, okay, then I'll say yes."
Bill Gates: "No. Of course I count as part of Microsoft! Write my vote down. No."
Mike Reavey: "Okay, so, what was the right answer? Oh, umm... we were 40% correct. That's not too bad --there's been improvement."
404555974007725459910684486621289147856453481154 in hex is "You sank my Battleship?"
[GPG key in journal]
That might have been funny when it was posted after the 2000 election, or even after the 2004 election (really, what happened there? Dubbya stole the election in 2000 and you promptly actually elect him the next time round?!).
I for one think that the election this time round was carried out according to the rules; the biggest budget^H^H^H^H^H^H^H^H^H^H^H best candidate won.
They can predict exploits in their own software. Well paint me yellow and call me a phone directory!
How can a PR team for one of the largest corporations in the US seriously release a statement like this? What kind of company fails so badly that they can only predict 40% of exploits in their own [proprietary] software?
If a major car (or car part) manufacturer "accurately" predicted that 40% of their automobiles would explode and burn their owners alive due to a fuel system defect....would people still buy their cars? Oh right...firestone.
Then their probability will increase to 50%! j/k.
Correct the headline.
And vinegar is acceptable on salad only, not potatoes.
What about potato salad?
Microsoft Security Research Centre is a success as a disaster agency? A bit harsh, but I suppose so...
40% accuracy with no false negatives on Microsoft software is easy. Mark them all as top priority show stoppers and there you have your 40%.
They tried to predict if a hole will be exploited or not. Those are two outcomes. If you were to guess you would end up with a 50% chance of guessing right.
And they were only 40% right and 60% wrong?
Well, roundabouts (aka traffic circles) would make a lot more sense than four way stops or traffic lights in a lot of locations. However, "yeild" doesn't seem to be in the American psyche anymore.
I think it's about time to go fully metric, though. It's a much simpler system. Practically all of the rest of the world uses metric. The sciences and military use metric. Alas, "yeild" doesn't seem to be in the American psyche anymore.
When our name is on the back of your car, we're behind you all the way!
Alas, "yeild" doesn't seem to be in the American psyche anymore.
It's also not in any dictionary that I'm aware of either. Yield is though. Sorry, couldn't resist :)
The ones I worry about are the 12 year old bugs that have had exploit code for 8 years already and only now gets fixed - maybe.
Excuse me, but please get off my Pennisetum Clandestinum, eh!
4/9 =/= 40%.
Does anybody else think it's really funny that Microsoft's predicting abilities are better than their patching abilities?
I might know what I'm talkin' about, but then again, this is Slashdot...
And a third party finds a bug and reports it to them. Now would be good to predict the severity of the bug, so that the more exploitable ones can be fixed first?
You mean, unlike the press which tends to overblow every report of a vulnerability on Linux and/or FireFox, although in reality the "vulnerability" only work in a few very rare cases where a complex mix of condition. Plus a very gullible and cooperative user who will go through a long process in order to reach the point ...
Thats exactly what they are doing, and they are able to get the severity 40% of the time right, with no false negatives (that not a single severe one has been classified as a low priority one).
So, now, do you think this is bad or wrong or something?
It wouldn't be bad, if it weren't for microsoft's software quality being so bad, that simply calling every bug as critical would be a 100% sensitive 99% specific test.
Today's news almost sounds as if sponsored by Captain Obvious.
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
It's also not in any dictionary that I'm aware of either. Yield is though. Sorry, couldn't resist :)
/Me hangs head in shame, and yields to your greater spelling prowess, but is humbly constrained to point out that the duplication of the error was due to copy/paste, and not necessarily a faulty knowledge of "yield"'s proper spelling.
[Sigh]
When our name is on the back of your car, we're behind you all the way!
Call us when they rated something TOO HIGH, or OVERESTIMATED the number of exploits, not the other way around.
(boggle)
No sig today...
Well, roundabouts (aka traffic circles) would make a lot more sense than four way stops [...]
I ran into a few 4-way stop signs on a recent holiday to the US. Exactly how *is* the stalemate of 4 cars arriving at the same time supposed to be resolved ?
http://dispatch.com/live/content/local_news/stories/2008/10/28/ajudgerule.html?sid=101
878659 - yep its prime.
Yup. Trust me, we got it. But given Microsoft's track of security and to get back to the metaphor of grand parent's post :
Yes. Eventualy from time to time the coin will magically manage to get stuck somewhere and not land down on one side. By saying it will always land on its said, surely they will have a good ratio.
In other words : Microsoft's security is so bad that simply calling all bugs critical is a 100% sensitive 99% specific test. You'll rarely have a false negative that way.
They simply auto-congratulate themselves because sometimes they might predict when the coin won't hit the ground.
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
Why by yielding to the right of course! So while everyone is yielding to the right the people in the biggest hurry can slam on the gas simultaneously and meet in the middle. Hopefully thus increasing the average IQ by a small fraction.
Seriously though, I've never had a problem with that. Where it's unclear who was the right of way I just coast into the intersection a bit more slowly than usual so that it's clear who came to a stop first.
I watch Top Gear. Isn't that enough?
Flappinbooger isn't my real name
When four drivers arrive simultaneously at a 4-way stop, the appropriate order shall be determined as follows:
(1) Largest Truck/SUV
(2) Most guns in gun rack
(3) Most teeth
If the appropriate order cannot be determined by the above methods, then the Yuppie Exception is brought into play:
(1) Whomever holds the most complicated Starbuck's drink
(2) Largest age difference between parent and first born
(3) best gas mileage.
First in, First out. If 2 cars arrive at the same time, the person "on the right" goes first. If all 4 arrive at the same time? On the odd chance that it happens, the person with the largest gun, tallest 4x4 or shiniest belt buckle goes first. (just kidding, a little brit humor for you)
To be honest, I don't know, and I don't think I've ever experienced it. I've been at 4 way stops with cars in all directions, but there was always a clear order of who went next, it's basically clockwise or in order of arrival.
Flappinbooger isn't my real name
The car to the left actually has the right of way, seeing as how it is more difficult to see a vehicle on your right, than your left. In the UK, I assume the opposite is true since the drivers side of the vehicle is on the right.
"And we have seen and do testify that the Father sent the Son to be the Savior of the World"
1 John 4:14
Sure, it's easy to sit back and try bashing MS... but neither Teh Lunix nor Apple even try.
Apple ignores problems and sues people who disclose exploits... and Teh Lunix desperately prays that their anti-MS FUD campaign will magically make them safe, which it never does.
So at least MS is trying. That's more than can be said of their do-nothing, know-nothing competition.
Yeah by allowing homeless people to register to vote they basically rigged the election.
I mean who the fuck wants homeless people to be allowed vote, society has made it clear that they aren't wanted
No one seems to be looking at this from the opposite angle.
If I'm writing malware that's going to need to exploit Windows, this gives me an easy chart of which exploit I should pick -- the ones with the lowest patch priority, of course.
Warning. If you love Microsoft, don't read. Your delicate sensibilities might be hurt 40% of the time.
Engineering sector 99.999 Microsoft 40.000 Sounds about right.
Having to work for a living is the root of all evil.
$49,999 for knowing where to write the 'X'. That's how I see this, anyway. Sounds to me like the crackers were the ones that only scored 40%.
Selling source closed software will be illegal in future.
Slashdot = Sarcasm
If it was only the homeless, then fine. But you have Acorn all over the place paying people to vote with the only requirement to vote being your name and last 4 digits of your social.
Add to that the top Ohio elections official who doesn't want to verify the identity of new voters. http://www.ohio.com/news/break_news/31101144.html
You end up with dead people voting and all kinds of fraud.
http://www.newsnet5.com/news/17859950/detail.html?rss=nn5&psp=news
878659 - yep its prime.
It's laughable because Microsoft is forever telling the world how THIS time their OS is more secure than ever.
No sir. In North America, when two or more vehicles arrive at a four way stop at the same time, the car to the right has the right of way.
When our name is on the back of your car, we're behind you all the way!