Slashdot Mirror

← Back to Stories (view on slashdot.org)

Microsoft Exploit Predictions Right 40% of Time

Posted by timothy on from the statistics-94pct-nonsense dept.

CWmike writes "Microsoft today called its first month of predicting whether hackers will create exploit code for its bugs a success — even though the company got its forecast right just 40% of the time for October. 'I think we did really well,' said Mike Reavey, group manager at the Microsoft Security Research Center (MSRC), when asked for a postmortem evaluation of the first cycle of the team's Exploitability Index. 'Four of the [nine] issues that we said where consistent exploit code was likely did have exploit code appear over the first two weeks. And another key was that in no case did we rate something too low.' Microsoft's Exploitability Index was introduced last month."

9 of 182 comments (clear)

  1. Congratulations? by Smidge204 · · Score: 3, Insightful

    That's great, guys, but don't you think being proud that you were right about your code being exploited is... backwards? That's like being proud you correctly predicted you would get stabbed while walking through a ghetto wearing gang colors.

    Then again, this is Microsoft. They probably throw an office party every time something compiles without errors.
    =Smidge=

    1. Re:Congratulations? by iammani · · Score: 5, Insightful

      Slashdot crowd *loves* MSFT bashing doesnt it.

      Ok lets see... Some company (say Canonical or MSFT) builds a huge software and releases it. And a third party finds a bug and reports it to them. Now would be good to predict the severity of the bug, so that the more exploitable ones can be fixed first? Thats exactly what they are doing, and they are able to get the severity 40% of the time right, with no false negatives (that not a single severe one has been classified as a low priority one).

      So, now, do you think this is bad or wrong or something?

    2. Re:Congratulations? by MrMr · · Score: 4, Insightful

      They build enough security holes in their applications to do meaningful statistics on the monthly number of exploits in the wild.
      So, now, do you think that that is not a reason for criticism on their internal software testing?

    3. Re:Congratulations? by LordKronos · · Score: 3, Insightful

      Sure, if you have unlimited resources and can devote an infinite number of people to fixing everything, that would be great. However, if you have finite resources available and have to devote them to fixing up certain areas, how do you know where to devote your attention? If you can come up with a methodology for predicting such a thing, put it to the test, and get decent accuracy in your predictions, then wouldn't that be useful for confirming for you how you should devote your limited resources?

      There is nothing unique in what they are doing. I mean, look at the auto industry, for example. They don't just randomly assign engineers to try and make random things safer. They do studies, try to figure out what are the most dangerous aspects of a vehicle, and then assign engineers to work on those specific things.

      Fortunately for the auto industry, it's a little easier to do your predictions pre-release, since the "attack vectors" are more limited and well known (there are typically only so many ways you can get into an accident, so it's easier to model a majority of those cases). This allows them to be proactive in fixing flaws. Unfortunately, the attacks vectors in software are a bit more numerous, and you often have to take a more reactive approach. What Microsoft is doing here is trying to model things to see how reasonable it would be to devote resources in certain ways to be proactive.

      So again, in what way is this bad?

  2. That's not too bad by 91degrees · · Score: 5, Insightful

    A little heavy on the false positives but no false negatives so it allowed more efficient targeting of the risk areas. Also good enough to provide useful feedback.

  3. Re:This is why Microsoft software sucks by Mateo_LeFou · · Score: 4, Insightful

    >if it comes up heads, its exploitable. Tails its gonna be ok.

    In this case, wouldn't there be as many false negatives as false positives?

    --
    My turnips listen for the soft cry of your love
  4. Re:This is why Microsoft software sucks by Anonymous Coward · · Score: 5, Insightful

    No, it means that they were able to cut the field of their immediate focus nearly in half while not missing any issues. For such a complex system without any precise mathematical model, that's pretty good.

    In this case, flipping a coin is statistically likely to let an unaddressed issue through, and that's a big no-no for applications like this.

  5. Re:Still not getting it. by Roland+Piquepaille · · Score: 3, Insightful

    or hell ANY GOD DAMN FUCKING BUG YOU FIND, needs fixing, right Microsoft?

    Any goddamn bug doesn't need fixing asap the same way. Software always has bugs, even really good software, so it's a matter of prioritizing which bugs are show-stoppers, which are less problematic and which are minor.

    The problem with Microsoft is their habit of releasing bananaware: they ship green software that matures at the customers, at the expense of the customer of course who essentially pays to become a beta-tester for Microsoft. In other terms, when other reputable software shops iron out most bugs in-house before releasing their products, Microsoft just removes show-stoppers and let its customers report all the other bugs.

  6. Re:This is why Microsoft software sucks by Anonymous Coward · · Score: 4, Insightful

    If the steps are sequential, it's less than 33%. The correct figure is 12.5% (50 percent of 50 percent of 50 percent).