Slashdot Mirror
Microsoft Exploit Predictions Right 40% of Time
CWmike writes "Microsoft today called its first month of predicting whether hackers will create exploit code for its bugs a success — even though the company got its forecast right just 40% of the time for October. 'I think we did really well,' said Mike Reavey, group manager at the Microsoft Security Research Center (MSRC), when asked for a postmortem evaluation of the first cycle of the team's Exploitability Index. 'Four of the [nine] issues that we said where consistent exploit code was likely did have exploit code appear over the first two weeks. And another key was that in no case did we rate something too low.' Microsoft's Exploitability Index was introduced last month."
A little heavy on the false positives but no false negatives so it allowed more efficient targeting of the risk areas. Also good enough to provide useful feedback.
>if it comes up heads, its exploitable. Tails its gonna be ok.
In this case, wouldn't there be as many false negatives as false positives?
My turnips listen for the soft cry of your love
No, it means that they were able to cut the field of their immediate focus nearly in half while not missing any issues. For such a complex system without any precise mathematical model, that's pretty good.
In this case, flipping a coin is statistically likely to let an unaddressed issue through, and that's a big no-no for applications like this.
Hint: 40% is worse than guessing.
No - from TFA:
The index, launched last month, rates each vulnerability using a three-step system.
Random guesses would be expected to yield 33% success.
This comment is for entertainment purposes only. Any similarity to real insight or information is purely coincidental.
40% accuracy in predicting with no false negatives? There are plenty of distaster agencies around the world who would be incredibly pleased with that kind of accuracy
Slashdot crowd *loves* MSFT bashing doesnt it.
Ok lets see... Some company (say Canonical or MSFT) builds a huge software and releases it. And a third party finds a bug and reports it to them. Now would be good to predict the severity of the bug, so that the more exploitable ones can be fixed first? Thats exactly what they are doing, and they are able to get the severity 40% of the time right, with no false negatives (that not a single severe one has been classified as a low priority one).
So, now, do you think this is bad or wrong or something?
If the steps are sequential, it's less than 33%. The correct figure is 12.5% (50 percent of 50 percent of 50 percent).
They build enough security holes in their applications to do meaningful statistics on the monthly number of exploits in the wild.
So, now, do you think that that is not a reason for criticism on their internal software testing?
It's like running your own car into a pole, providing the mechanic with your estimate of the damages and claiming you were right when he only overcharges you by 60%.
In Soviet Russia meme tires of you!