Slashdot Mirror

← Back to Stories (view on slashdot.org)

Microsoft Exploit Predictions Right 40% of Time

Posted by timothy on from the statistics-94pct-nonsense dept.

CWmike writes "Microsoft today called its first month of predicting whether hackers will create exploit code for its bugs a success — even though the company got its forecast right just 40% of the time for October. 'I think we did really well,' said Mike Reavey, group manager at the Microsoft Security Research Center (MSRC), when asked for a postmortem evaluation of the first cycle of the team's Exploitability Index. 'Four of the [nine] issues that we said where consistent exploit code was likely did have exploit code appear over the first two weeks. And another key was that in no case did we rate something too low.' Microsoft's Exploitability Index was introduced last month."

19 of 182 comments (clear)

  1. Congratulations? by Smidge204 · · Score: 3, Insightful

    That's great, guys, but don't you think being proud that you were right about your code being exploited is... backwards? That's like being proud you correctly predicted you would get stabbed while walking through a ghetto wearing gang colors.

    Then again, this is Microsoft. They probably throw an office party every time something compiles without errors.
    =Smidge=

    1. Re:Congratulations? by iammani · · Score: 5, Insightful

      Slashdot crowd *loves* MSFT bashing doesnt it.

      Ok lets see... Some company (say Canonical or MSFT) builds a huge software and releases it. And a third party finds a bug and reports it to them. Now would be good to predict the severity of the bug, so that the more exploitable ones can be fixed first? Thats exactly what they are doing, and they are able to get the severity 40% of the time right, with no false negatives (that not a single severe one has been classified as a low priority one).

      So, now, do you think this is bad or wrong or something?

    2. Re:Congratulations? by MrMr · · Score: 4, Insightful

      They build enough security holes in their applications to do meaningful statistics on the monthly number of exploits in the wild.
      So, now, do you think that that is not a reason for criticism on their internal software testing?

    3. Re:Congratulations? by TheCycoONE · · Score: 2, Insightful

      No, it's like correctly predicting that you'll get stabbed 17 minutes after entering the ghetto, by 6 gang members dressed in red.

      Not at all. It's much more like guessing that you will be stabbed 6.8 minutes after entering a ghetto by 8-9 gang members dressed in red, then actually being stabbed after 17 minutes by 6 gang members wearing pink.

    4. Re:Congratulations? by LordKronos · · Score: 3, Insightful

      Sure, if you have unlimited resources and can devote an infinite number of people to fixing everything, that would be great. However, if you have finite resources available and have to devote them to fixing up certain areas, how do you know where to devote your attention? If you can come up with a methodology for predicting such a thing, put it to the test, and get decent accuracy in your predictions, then wouldn't that be useful for confirming for you how you should devote your limited resources?

      There is nothing unique in what they are doing. I mean, look at the auto industry, for example. They don't just randomly assign engineers to try and make random things safer. They do studies, try to figure out what are the most dangerous aspects of a vehicle, and then assign engineers to work on those specific things.

      Fortunately for the auto industry, it's a little easier to do your predictions pre-release, since the "attack vectors" are more limited and well known (there are typically only so many ways you can get into an accident, so it's easier to model a majority of those cases). This allows them to be proactive in fixing flaws. Unfortunately, the attacks vectors in software are a bit more numerous, and you often have to take a more reactive approach. What Microsoft is doing here is trying to model things to see how reasonable it would be to devote resources in certain ways to be proactive.

      So again, in what way is this bad?

    5. Re:Congratulations? by mobby_6kl · · Score: 2, Insightful

      No, the criticism of either their coding practices or QA has nothing to do with a new and fairly efficient way to prioritize bug fixes. They already have the software with all the holes built in. Now they should deal with what they have in the best way possible, don't you agree?

    6. Re:Congratulations? by orclevegam · · Score: 2, Insightful

      Actually in this example it would be undercharging. They predicted more exploits would happen than actually did, which given the nature of the predictions I'm happy with. Had they predicted that only only 1 of the exploits was likely to be used and 6 of them were instead then I'd be more ticked at them. Of course what would make me fscking ecstatic is if MS actually managed to create a piece of software with less than 100 security flaws (and calc, notepad, and paint don't count).

      --
      Curiosity was framed, Ignorance killed the cat.
    7. Re:Congratulations? by Miseph · · Score: 2, Insightful

      Um, no, I know we're all desperate for this to be some terrible mistake on MSFT's part, it just isn't.

      This is more like the car company saying: We have found 10 ways that we think our cars can be sabotaged, and we have released free snap-on repair kits that are intended to counter those possibilities, and will distribute them to all customers who request them. As it turns out, only 4 of them have actually been used by saboteurs, but we nonetheless recommend installing all 10 kits just to be safe.

      Yes, how irresponsible of them, finding and eliminating ways for dedicated deliberate attackers to gain access faster than those attackers can actually accomplish it.

      --
      Try not to take me more seriously than I take myself.
  2. That's not too bad by 91degrees · · Score: 5, Insightful

    A little heavy on the false positives but no false negatives so it allowed more efficient targeting of the risk areas. Also good enough to provide useful feedback.

  3. This is why Microsoft software sucks by QuantumG · · Score: 2, Insightful

    Any engineer who says that "40% is pretty good predicting" is incapable of writing good software, or managing a project, or, even, applying the scientific method.

    Hint: 40% is worse than guessing.

    --
    How we know is more important than what we know.
    1. Re:This is why Microsoft software sucks by Mateo_LeFou · · Score: 4, Insightful

      >if it comes up heads, its exploitable. Tails its gonna be ok.

      In this case, wouldn't there be as many false negatives as false positives?

      --
      My turnips listen for the soft cry of your love
    2. Re:This is why Microsoft software sucks by Anonymous Coward · · Score: 5, Insightful

      No, it means that they were able to cut the field of their immediate focus nearly in half while not missing any issues. For such a complex system without any precise mathematical model, that's pretty good.

      In this case, flipping a coin is statistically likely to let an unaddressed issue through, and that's a big no-no for applications like this.

    3. Re:This is why Microsoft software sucks by Anonymous Coward · · Score: 4, Insightful

      If the steps are sequential, it's less than 33%. The correct figure is 12.5% (50 percent of 50 percent of 50 percent).

  4. Re:Still not getting it. by c_forq · · Score: 2, Insightful

    Wow, have some anger issues there? This isn't about not fixing bugs, this is about prioritizing bug fixes. Anything this large is going to have massive amounts of bugs (I can't count the times I've updated packages in Ubuntu, and the OS-X bug fixes come by the hundreds per .x release). Microsoft, just like Apple and Canonical, has limited resources to fix said bugs (and actually Apple and Canonical get some free work done for them, due to use of open source packages).

    --
    Computers allow humans to make mistakes at the fastest speeds known, with the possible exception of tequila and handguns
  5. 4/9 = 40%? by Anonymous Coward · · Score: 1, Insightful

    Research also shows Slashdot editors verify submission figures 112% of the time.

  6. Re:Still not getting it. by Roland+Piquepaille · · Score: 3, Insightful

    or hell ANY GOD DAMN FUCKING BUG YOU FIND, needs fixing, right Microsoft?

    Any goddamn bug doesn't need fixing asap the same way. Software always has bugs, even really good software, so it's a matter of prioritizing which bugs are show-stoppers, which are less problematic and which are minor.

    The problem with Microsoft is their habit of releasing bananaware: they ship green software that matures at the customers, at the expense of the customer of course who essentially pays to become a beta-tester for Microsoft. In other terms, when other reputable software shops iron out most bugs in-house before releasing their products, Microsoft just removes show-stoppers and let its customers report all the other bugs.

  7. Re:Being right 40% of the time... by dubl-u · · Score: 2, Insightful

    Doesn't look so impressive when you look at it this way.

    Depends on the payoff.

    It's not good if you're betting even money on coin tosses. But if you're a venture capitalist, it's great. The general rule for tech VCs is that 7 bets out of 10 will fail, 2 will do ok, and 1 will be a big success. If that 1 success is buying 10% of Google in the very early days, your 70% failure rate is still pretty awesome, because you're still up billions of dollars.

  8. What do ya mean 40 percent? by WheelDweller · · Score: 0, Insightful

    Has there ever been a Microsoft bug that hackers have left alone?

    We've been through this 'a million times' since DOS; there are literally more than a million active viruses out there, with another 100,000 per month. 40 percent chance of an exploit being used seems kinda low, doesn't it?

    --
    --- For a good time mail uce@ftc.gov
  9. More fail from MS by foldingstock · · Score: 2, Insightful

    They can predict exploits in their own software. Well paint me yellow and call me a phone directory!

    How can a PR team for one of the largest corporations in the US seriously release a statement like this? What kind of company fails so badly that they can only predict 40% of exploits in their own [proprietary] software?

    If a major car (or car part) manufacturer "accurately" predicted that 40% of their automobiles would explode and burn their owners alive due to a fuel system defect....would people still buy their cars? Oh right...firestone.