Slashdot Mirror
Microsoft Exploit Predictions Right 40% of Time
CWmike writes "Microsoft today called its first month of predicting whether hackers will create exploit code for its bugs a success — even though the company got its forecast right just 40% of the time for October. 'I think we did really well,' said Mike Reavey, group manager at the Microsoft Security Research Center (MSRC), when asked for a postmortem evaluation of the first cycle of the team's Exploitability Index. 'Four of the [nine] issues that we said where consistent exploit code was likely did have exploit code appear over the first two weeks. And another key was that in no case did we rate something too low.' Microsoft's Exploitability Index was introduced last month."
Actually that was John Cleese, even posting anon you should give credit where its due.
Actually it originated with One Alan Baxter of Rochester and expanded by other people on Usenet. So if you do give credit where it's due give it where it's actually due.
Hint: 40% is worse than guessing.
No - from TFA:
The index, launched last month, rates each vulnerability using a three-step system.
Random guesses would be expected to yield 33% success.
This comment is for entertainment purposes only. Any similarity to real insight or information is purely coincidental.
What REALLY happened is this: Every security hole that MS discovered on its own, was exploited BUT we are supposed to be happy because in 40% of the cases MS correctly predicted that it would be exploited.
No. What happened was this - MS spotted 18 potential security holes. 9 of them were regarded as more serious. A company that focussed on protecting against those 9 would not have been affected at all and would have had less disruption than a company that protected against all 18.
They are offering this as a means to tell their bug fixing department and other companies which areas to prioritize.
What REALLY happened is this: Every security hole that MS discovered on its own, was exploited BUT we are supposed to be happy because in 40% of the cases MS correctly predicted that it would be exploited.
I know we don't RTFA but please at least RTFS.
'Four of the [nine] issues that we said where consistent exploit code was likely did have exploit code appear over the first two weeks. And another key was that in no case did we rate something too low.'
So no, at least according to the summary not every security hole was exploited. If you're going to claim otherwise at least provide some links to an article; hopefully one supporting your claims although that's not always necessary for the +5 informative.
In fact I just actually bothered to RTFA, just to make sure, and it said that no exploit code appeared for the low ranked vulnerabilities.
Nick
If you actually want a correct coin analogy, its that every time they called heads (heads = bug will be exploited), it showed up heads 40% of the time. Every time they called tails (tails = bug won't be exploited), it showed up tails 100% of the time. Now, since there were 18 coin flips (bugs), they were right 13 times (4/9 were correctly called as heads, 9/9 were correctly called as tails). Thats 13/19. They had about a 68% success rate.
I don't understand how the article got the math completely wrong or how people aren't seeing the extremely obvious flaw in the math.
Actually, they'd have to flip a coin for every bug – and their current statistic, "40% of the bugs we identified as exploitable were exploited", would probably look great compared to the percentage they'd get by flipping a coin.
Basically, you're looking at this wrong. Microsoft correctly predicted 40% of the exploitable bugs, but they also correctly predicted the non-exploitable ones which wouldn't be exploited.
Suppose (and I don't have actual numbers, so I'll make up hypothetical ones) Microsoft finds 100 bugs, and 5 of them appear exploitable. 2 of those are actually exploited (40%). However, you should take into account all the non-exploitable bugs that weren't exploited: Microsoft correctly predicted 95 non-exploitable bugs and 2 exploitable ones, which is 97%. They were incorrect only on the 3 bugs that they thought would be exploited and weren't (using these hypothetical numbers).