Microsoft Exploit Predictions Right 40% of Time

CWmike writes "Microsoft today called its first month of predicting whether hackers will create exploit code for its bugs a success — even though the company got its forecast right just 40% of the time for October. 'I think we did really well,' said Mike Reavey, group manager at the Microsoft Security Research Center (MSRC), when asked for a postmortem evaluation of the first cycle of the team's Exploitability Index. 'Four of the [nine] issues that we said where consistent exploit code was likely did have exploit code appear over the first two weeks. And another key was that in no case did we rate something too low.' Microsoft's Exploitability Index was introduced last month."

Comments

Congratulations?

[Smidge204]

That's great, guys, but don't you think being proud that you were right about your code being exploited is... backwards? That's like being proud you correctly predicted you would get stabbed while walking through a ghetto wearing gang colors.

Then again, this is Microsoft. They probably throw an office party every time something compiles without errors.
=Smidge=

Re:Congratulations?

[David Gerard]

Indeed. I swear, I called it: it's easier to predict the holes when you release them yourself.

After what was expected to be an unusually quiet Patch Tuesday, Microsoft has released eight patches for applications with an insufficient number of security holes. "Our market is the enterprise," said Microsoft security marketer Jonathan Ness. "Information technology professionals know that Windows is the greatest IT job creation scheme in history. Without Patch Tuesday, there's no reason for the experienced IT worker to spend his time hiding out in the server room watching progress bars and getting over his hangover. Also, you can't tell people a virus ate their mail, you actually have to get it back for them."

Re:Congratulations?

[Roland Piquepaille]

That's great, guys, but don't you think being proud that you were right about your code being exploited is... backwards?

Well, they're not proud of making exploitable code (if they were, there would have been a giant endless party at Microsoft for the last 20 years), they're proud of predicting when/how fast their code will be exploited.

That's like being proud you correctly predicted you would get stabbed while walking through a ghetto wearing gang colors.

No, it's like correctly predicting that you'll get stabbed 17 minutes after entering the ghetto, by 6 gang members dressed in red.

Re:Congratulations?

[gEvil (beta)]

I can think of a few ways they can get that number up. Of course, none of them would be good for the consumer. But when has Microsoft put the consumer above having numbers that it can tout?

Re:Congratulations?

[Sockatume]

If you're sailing in a yacht made of cake with sails of tissue paper, with pegs for both legs and hooks for both hands, it's useful to know where the leaks in your boat are.

Re:Congratulations?

[iammani]

Slashdot crowd *loves* MSFT bashing doesnt it.

Ok lets see... Some company (say Canonical or MSFT) builds a huge software and releases it. And a third party finds a bug and reports it to them. Now would be good to predict the severity of the bug, so that the more exploitable ones can be fixed first? Thats exactly what they are doing, and they are able to get the severity 40% of the time right, with no false negatives (that not a single severe one has been classified as a low priority one).

So, now, do you think this is bad or wrong or something?

Re:Congratulations?

[MrMr]

They build enough security holes in their applications to do meaningful statistics on the monthly number of exploits in the wild.
So, now, do you think that that is not a reason for criticism on their internal software testing?

Re:Congratulations?

[TheCycoONE]

No, it's like correctly predicting that you'll get stabbed 17 minutes after entering the ghetto, by 6 gang members dressed in red.

Not at all. It's much more like guessing that you will be stabbed 6.8 minutes after entering a ghetto by 8-9 gang members dressed in red, then actually being stabbed after 17 minutes by 6 gang members wearing pink.

Re:Congratulations?

[iammani]

Hmmm I dont have statistics about number of security holes in MSFT apps vs say adobe acrobat/flash or any close sourced software.

But given that they are closed source, I would tend to think they are doing ok.

And yes I am playing a devils advocate here, though I do hate their bloatware

Re:Congratulations?

[Zxarr]

Wait, aren't we supposed to use car analogy's on /.??

Re:Congratulations?

[NoisySplatter]

It's like running your own car into a pole, providing the mechanic with your estimate of the damages and claiming you were right when he only overcharges you by 60%.

Re:Congratulations?

[hairyfeet]

Okay.....It is like predicting you will get hit by a VW Bug crossing the street,and instead a Mac Truck nails you before you even get off the curb and drags you twenty feet. With a four out of ten pretty much the only thing they got right was they were going to get hit and it would hurt.

Re:Congratulations?

[argStyopa]

Yes, this IS bad or wrong or something.

Wouldn't it make MORE sense to perhaps spend the human/technical resources FIXING the most exploitable bugs rather than standing around with a beer in hand saying 'yep, that's going to explode for sure'.

*BOOM*

'See? I told you so.'

Re:Congratulations?

[iammani]

Wouldn't it make MORE sense to perhaps spend the human/technical resources FIXING the most exploitable bugs rather than standing around with a beer in hand saying 'yep, that's going to explode for sure'.

Yes it indeed would, and thats exactly what they have done and the story is about the review of the practice that happened at the end of the month (read during a review of what became an exploit and what got fixed at the right time)

Re:Congratulations?

[LordKronos]

Sure, if you have unlimited resources and can devote an infinite number of people to fixing everything, that would be great. However, if you have finite resources available and have to devote them to fixing up certain areas, how do you know where to devote your attention? If you can come up with a methodology for predicting such a thing, put it to the test, and get decent accuracy in your predictions, then wouldn't that be useful for confirming for you how you should devote your limited resources?

There is nothing unique in what they are doing. I mean, look at the auto industry, for example. They don't just randomly assign engineers to try and make random things safer. They do studies, try to figure out what are the most dangerous aspects of a vehicle, and then assign engineers to work on those specific things.

Fortunately for the auto industry, it's a little easier to do your predictions pre-release, since the "attack vectors" are more limited and well known (there are typically only so many ways you can get into an accident, so it's easier to model a majority of those cases). This allows them to be proactive in fixing flaws. Unfortunately, the attacks vectors in software are a bit more numerous, and you often have to take a more reactive approach. What Microsoft is doing here is trying to model things to see how reasonable it would be to devote resources in certain ways to be proactive.

So again, in what way is this bad?

Re:Congratulations?

[TheP4st]

To me it seem as most of the time there is a greater love for pointlessly bashing twitter and his sockpuppets than MS in the MS article threads.
Effectively making the threads hopeless to read, for an example of what I mean have a look at http://tech.slashdot.org/article.pl?sid=08/11/13/210255 with your prefs set to show all comments.

*Curiously awaiting the mod results*

Re:Congratulations?

[sjames]

Based on their success rate, they should flip a coin instead, then they'll be at 50%. That's what everyone's laughing at.

Re:Congratulations?

[davetv]

rofl

Re:Congratulations?

[Anonymous Coward]

Actually, in that case I think it's more useful to know where the lifeboat is!

Re:Congratulations?

[mobby_6kl]

No, the criticism of either their coding practices or QA has nothing to do with a new and fairly efficient way to prioritize bug fixes. They already have the software with all the holes built in. Now they should deal with what they have in the best way possible, don't you agree?

Re:Congratulations?

[__aaqvdr516]

If I had mod points you'd have them, but next time don't go so in-depth with the auto industry. You may have gone over some peoples heads by not explicitly calling them 'car makers'.

Re:Congratulations?

[V!NCENT]

Slashdot crowd *loves* MSFT bashing doesnt it.

Dude, get over the fact that you fail at switching away from the worst OS ever concieved that did more damage to computing in general than people could have ever thought.

Why are you being such a crybaby by defending a sucking commercial product created by a commercial company that tries to lock you in, drain you and by that fucking you in any way it possibly can?

Windows was, still is, and will always be the worst OS ever concieved and if every application worked on every platform than you (and everybody else) would never ever use it. As a matter of fact, the only reason people switch between Windows versions is because they upgrade to a faster computer that has the latest piece of digitally incarnated shit on it preloaded.

Re:Congratulations?

[Smidge204]

How about the requisite Slashdot car analogy instead?

A car manufacturer states that they predict some quantity of their cars will be stolen due to design flaws (locks and latches, windows, security systems etc).

Then they proudly announce that their theft predictions are right a little less than half of the time. "Hey everyone! Not as many of our cars were stolen as we thought would be! Isn't that great news?"

Does this really inspire confidence in the manufacturer for you?
=Smidge=

Re:Congratulations?

[gazbo]

Statistics. You fail it hard.

Re:Congratulations?

[PJ1216]

If you actually want a correct coin analogy, its that every time they called heads (heads = bug will be exploited), it showed up heads 40% of the time. Every time they called tails (tails = bug won't be exploited), it showed up tails 100% of the time. Now, since there were 18 coin flips (bugs), they were right 13 times (4/9 were correctly called as heads, 9/9 were correctly called as tails). Thats 13/19. They had about a 68% success rate.

I don't understand how the article got the math completely wrong or how people aren't seeing the extremely obvious flaw in the math.

Re:Congratulations?

[PJ1216]

13/18. That was a typo. But I know if I don't correct it, I'll be called out on it.

Re:Congratulations?

[RussellSHarris]

Actually, they'd have to flip a coin for every bug – and their current statistic, "40% of the bugs we identified as exploitable were exploited", would probably look great compared to the percentage they'd get by flipping a coin.

Basically, you're looking at this wrong. Microsoft correctly predicted 40% of the exploitable bugs, but they also correctly predicted the non-exploitable ones which wouldn't be exploited.

Suppose (and I don't have actual numbers, so I'll make up hypothetical ones) Microsoft finds 100 bugs, and 5 of them appear exploitable. 2 of those are actually exploited (40%). However, you should take into account all the non-exploitable bugs that weren't exploited: Microsoft correctly predicted 95 non-exploitable bugs and 2 exploitable ones, which is 97%. They were incorrect only on the 3 bugs that they thought would be exploited and weren't (using these hypothetical numbers).

Re:Congratulations?

[MrMr]

You mean that Microsoft is too small to maintain their own code?
Who cares about prioritization? If they have a monthly batch of fresh flaws and their don't fix at least as many within a month they are fighting a lost battle anyway.

Re:Congratulations?

[orclevegam]

Actually in this example it would be undercharging. They predicted more exploits would happen than actually did, which given the nature of the predictions I'm happy with. Had they predicted that only only 1 of the exploits was likely to be used and 6 of them were instead then I'd be more ticked at them. Of course what would make me fscking ecstatic is if MS actually managed to create a piece of software with less than 100 security flaws (and calc, notepad, and paint don't count).

Re:Congratulations?

[orclevegam]

They're off either side, they're the ones labeled "OS X", "Linux", and "Unix".

Re:Congratulations?

[maxume]

Why the arbitrary period? If they get 20 bugs in October and 10 in November, fixing 15 a month will work out fine (and be cheaper than having the resources available to deal with their worst months).

Re:Congratulations?

[orclevegam]

"We predicted that 9 cars would be stolen this month, but only 4 were! Of the remaining 6, 3 stopped working, 2 will probably be stolen next month, and the last one exploded."

Re:Congratulations?

[orclevegam]

Doh! Math fail... change that to only 3 were.

Re:Congratulations?

[V!NCENT]

Mac OS doesn't work with nearly as much hardware as Windows does, or else it'd probably be about as stable as Windows and therefore not any 'better.'

Yeah my mobile phone OS also doesn't work on a Nokia. It's not like Windows Mobile works on an iPhone either... Your point?

Linux distributions are way beyond the capabilities of the average computer user.

Which is untrue because I have seen avarage users doing productive work for their jobs on Ubuntu.

Windows works. [...]

Default drivers work? No. Security works? No. OpenGL works? No. Rendering web pages correctly with the Windows browser works? No. DVD playback works? No. Compatible with older hardware? No.

Wow, I guess nothing works unless you fix everything with external software.

[...]Maybe not for you.[...]

Think about the last time an avarge user liked Windows.

[...]The fact that a product isn't well suited for a power user[...]

What fact? Ubuntu is suited for the avarage user

[...] does *not* make it the worst product in the world.

The fact that it is made to earn money instead of evolving computing in a digital age '*does*' make it suck. Oh and Linux is not a product; it is a profitless project.

The company may have had a bad influence on the computer world, but its OS isn't as bad as people make it out to be.

You're right; it's even worse. Where shall we start? Shall we start with the NSA backdoors, the DRM, the artificialy high mininal system requirements, the ad-ridden Live Messenger, the crash of the NYSE after paying money to the NYSE so they could downgrade to Windows, the attempts at screwing the OLPC project so children in Africa couldn't be part of the digital age, errr...? There are just too many points to write down here.

Most problems that people comment on are issues that rarely will ever effect the normal user.

Like what issues? The BSODs in the first year of Vista? The remote exploits? The virusses and the malware? The system requirements? I'm not really sure what you mean here...

Stop bashing something because it wasn't designed for you, but for someone else.

Ah, so contrary to Windows, Fedora was designed for me? Cool, but very untrue. Ubuntu is designed for the avarage user. Windows is clearly not. You see, Windows is the only OS that is actually designed solely for programmers and for money. That way they could get everyone to code for Windows, but those days are now gone.

You're not the center of the universe.

That's because the universe doesn't have a center, otherwise...

Re:Congratulations?

[RichiH]

I don't like MS either, but you are doing a wonderful job of sidestepping what GP said. Pretty much exactly what he complained about in the first place.

Re:Congratulations?

[sorak]

In fairness to MSFT, it could have some useful applications in prioritizing. History has shown us that a software company obviously can't fix every bug, so, a more efficient way of knowing in how many person hours to sling in which direction may prove useful.

Re:Congratulations?

[garry_g]

Then again, this is Microsoft. They probably throw an office party every time something compiles without errors.

So you're saying M$ is being cheated out of having office parties due to inferior Devel^h^h^h^h^hCompilers? :)

Re:Congratulations?

[dontmakemethink]

There's a term for this, it's called "quality control". It used to be performed *before* distributing a product to market. The term for evaluating quality after distribution is called "damage control", and this software is akin to a nurse performing triage on patients the hospital injured.

Re:Congratulations?

[jonaskoelker]

That's great, guys, but don't you think being proud that you were right about your code being exploited is... backwards?

I think it's even worse that they're proud that they're right about their code being exploited when they did worse than chance. They were more wrong than right, and they claim they did well.

FTFA (edited but staying true to the point):

Of the nine October vulnerabilities marked "Consistent exploit code likely," four did end up with exploit code available. None of the nine tagged "Inconsistent exploit code likely" had seen actual attack code. Microsoft correctly called the four bugs last month tagged with "Functioning exploit code unlikely."

So they got eight right, out of twenty-two: 8/22. If we give them one third credit for the maybe-exploits not having full-blown exploits, they're still only at 11 out of 22.

Here's my security advice: if you want to know whether a bug will be exploited, flip a coin.

I'm better than Microsoft, and I only charge the coins you use to flip for each bug :p

Re:Congratulations?

[jonaskoelker]

So, now, do you think this is bad?

I think it's pretty bad that without thinking, just by flipping coins, you can do better than them: http://tech.slashdot.org/comments.pl?sid=1029297&cid=25763435

Re:Congratulations?

[HappySmileMan]

Except for the 5 completely unpatched exploits for a month.

Of course if the 15 they fix are very severe, and the 5 left unfixed are only locally exploitable or affect and/or only a small amount of machines then it's only slightly humiliating, which I guess is what they're trying to do now.
It's not the best solution, and clearly it's only 40% effective but it IS an improvement.

Re:Congratulations?

[10101001 10101001]

they are able to get the severity 40% of the time right, with no false negatives (that not a single severe one has been classified as a low priority one).

If hypothetically, 40% of vulnerabilities are critical, then guessing critical 100% of the time would lead to a 40% success rate and no false negatives. This, btw, is the entire reason the rainbow colored terror warning level used by the US after 9/11 was so lucidrous. If one doesn't know the actual threat but the risk from guessing too low is termination, one will *always* overguess. That's not a good thing. That just creates an environment of hyperfear.

People bash Microsoft for many reasons. It doesn't help that Microsoft's work to monopolize the market place means that the many people who work in IT are left to deal with exploits in Microsoft products regularly. In short, the amount of complaining over a company is mostly proportional to having day-to-day experience with products from that company; this is because 90% of everything is crap, so almost everyone has to deal with crap. The fact that Microsoft doesn't manage to fit in the 10% only hightens the point that the free market doesn't do a good job at choosing "the best" product.

And this goes back to the situation with the US. The US has a hyperinflated ego as a superpower. Microsoft has a hyperinflated ego as a software company. People enjoy watching those with a hyperinflated ego suffer when they act less to humble themselves and more to carry on with their ego. I don't think Microsoft over predicting its own vulnerabilities is paramount to humbling themselves, for I have the strong belief that they will use those predictions to justify their ego-intended actions. Only when Microsoft starts signs of moderation in their actions consistently will I believe such a program is part of the solution.

Re:Congratulations?

[HappySmileMan]

As far as I know they tried to figure out how long it would be until an exploit was released for it and/or it was reported on a security site, then they could check those results by searching on exploit/security sites.

Of course if theres a group of 5 or 6 people who made an exploit for that bug and DIDN'T publish it, they couldn't tell and therefore their numbers would be slightly off, but if it was exploited on a large scale it would have been discovered anyway.

Re:Congratulations?

[maxume]

You are doing the same thing with the arbitrary periods. It doesn't matter if a few bugs go un-patched for a month (or two!) as long as you and your customers can live with the mean time to patch severe bugs. The particular bucket that the bugs fall into when lumped into months isn't very important.

Re:Congratulations?

[sjames]

Those laughing (including me) just find it funny to put that much effort into figuring out which bugs can be ignored rather than just fixing the bugs.

Re:Congratulations?

[gazbo]

Really? Until everyone pointed out how wrong you were, you claimed that what you found funny were the statistics.

Re:Congratulations?

[Miseph]

Um, no, I know we're all desperate for this to be some terrible mistake on MSFT's part, it just isn't.

This is more like the car company saying: We have found 10 ways that we think our cars can be sabotaged, and we have released free snap-on repair kits that are intended to counter those possibilities, and will distribute them to all customers who request them. As it turns out, only 4 of them have actually been used by saboteurs, but we nonetheless recommend installing all 10 kits just to be safe.

Yes, how irresponsible of them, finding and eliminating ways for dedicated deliberate attackers to gain access faster than those attackers can actually accomplish it.

Re:Congratulations?

[Foofoobar]

Hmmm.... my vote is for 'or something'.

Obviously you cannot separate your passionate love for Steve Ballmers trouser snake from the need for good engineering. Should 40% be acceptable for detecting parachute defects? How about life raft defects? Is this a severe comparison? Well when healthcare systems and life support systems and financial systems and even a INTERNATIONAL SPACE STATION run on their software, you'd think 99.9% would be the only goal they would be happy with and anything short of that would be unacceptable.

Instead, they send a message to their staff that 40% if a perfectly acceptable goal. Shoot for 40% everyone, thats where we set the bar.

That's why everyone is making fun. Not at their goal, but at their acceptance of it as acceptable. Pathetic.

Re:Congratulations?

[sjames]

I find those funny as well given the sample population and number of trials is too small to draw any real conclusions yet. My 'deep analysis' was just as high in quality as any other for now.

It's amazing really. I make more or less a throw away comment and it gets treated like it was meant to be a doctoral thesis.

Re:Congratulations?

[MrMr]

Are you deliberately misreading this?
If the average discovery rate of bugs in the time interval you *don't* find acceptable (a month according to the Microsoft update cycle, but you may know better) is larger than than the average resolution rate the end result is unacceptable. If that rate is lower, as in your example, there is no need for prioritarization because all bugs are fixed in time anyway.

That's not too bad

[91degrees]

A little heavy on the false positives but no false negatives so it allowed more efficient targeting of the risk areas. Also good enough to provide useful feedback.

Still not getting it.

[Barny]

And another key was that in no case did we rate something too low

Well, that's like saying, after you block all your email from getting through, "We rated all the spam accordingly, and let none of them through".

How about, we just guess, a rough fucking guess, that any "remote code execution" or "run with elevated privileges" exploit or hell ANY GOD DAMN FUCKING BUG YOU FIND, needs fixing, right Microsoft?

Re:Still not getting it.

[c_forq]

Wow, have some anger issues there? This isn't about not fixing bugs, this is about prioritizing bug fixes. Anything this large is going to have massive amounts of bugs (I can't count the times I've updated packages in Ubuntu, and the OS-X bug fixes come by the hundreds per .x release). Microsoft, just like Apple and Canonical, has limited resources to fix said bugs (and actually Apple and Canonical get some free work done for them, due to use of open source packages).

Re:Still not getting it.

[Roland Piquepaille]

or hell ANY GOD DAMN FUCKING BUG YOU FIND, needs fixing, right Microsoft?

Any goddamn bug doesn't need fixing asap the same way. Software always has bugs, even really good software, so it's a matter of prioritizing which bugs are show-stoppers, which are less problematic and which are minor.

The problem with Microsoft is their habit of releasing bananaware: they ship green software that matures at the customers, at the expense of the customer of course who essentially pays to become a beta-tester for Microsoft. In other terms, when other reputable software shops iron out most bugs in-house before releasing their products, Microsoft just removes show-stoppers and let its customers report all the other bugs.

Re:Still not getting it.

[Khuffie]

In other terms, when other reputable software shops iron out most bugs in-house before releasing their products, Microsoft just removes show-stoppers and let its customers report all the other bugs.

You mean, like Apple's Leopard release? Or Apple's iPhone 3G release? Or Apple's mobileme release?

I fail to see how Microsoft has a reputation of releasing 'bananaware' whereas Apple doesn't. I don't recall hearing about major, crippling bugs when Office 2007 came out (one of their biggest apps), and regardless of what you hear on Slashdot, Vista was actually a solid enough release and most of the issues were due to bad drivers that manufacturers didn't bother updating a year beforehand when they had betas and release candidates. (Not saying that neither had bugs, they did, but they were in no way 'beta' software.)

Re:Still not getting it.

[azrider]

Vista was actually a solid enough release and most of the issues were due to bad drivers that manufacturers didn't bother updating a year beforehand when they had betas and release candidates.

No, the problem was that MS changed the underlying layers between the betas, RC's and the RTM. Since that was happening, the manufacturers held off until they had a stable platform to shoot at.

Re:Still not getting it.

[Khuffie]

No, the problem was that MS changed the underlying layers between the betas, RC's and the RTM. Since that was happening, the manufacturers held off until they had a stable platform to shoot at.

Really? Care to cite proof of them changing the underlying layers between the RC and RTM? Or explain how some manufacturers were able to get drivers working for Vista properly before RTM?

Re:Still not getting it.

[Blakey Rat]

The problem with Microsoft is their habit of releasing bananaware: they ship green software that matures at the customers, at the expense of the customer of course who essentially pays to become a beta-tester for Microsoft.

I find it funny that that's a "problem" with Microsoft, and a stated goal of open source software. (Release early, release often.) Frankly, I'd be ecstatic if some open source projects followed Microsoft's example and removed the show-stopping bugs before release.

Re:Still not getting it.

[Blakey Rat]

Forget it, this is one of those Slashdot bullshit claims you see around here all the time which never has any kind of supporting evidence. One of the other ones is how DRM in Vista "slows down your computer."

It's trouble enough to get vendors to support their products *at purchase*, and yet people have problems believing that they don't support products for new OSes that come out after the purchase? I have no idea where people get that idea.

I bought a USB wifi card that didn't support Vista out-of-the-box... a WEEK ago. From an extremely well-known home network company with a good reputation. People really have problems believing this company, which barely supports an OS that's been out OVER A YEAR, wouldn't write drivers for a future OS? Those people are delusional.

This is why Microsoft software sucks

[QuantumG]

Any engineer who says that "40% is pretty good predicting" is incapable of writing good software, or managing a project, or, even, applying the scientific method.

Hint: 40% is worse than guessing.

Re:This is why Microsoft software sucks

[gbjbaanb]

Dear MS. I have a foolproof way of enhancing and improving upon your algorithms to determine the exploitability index.

if it comes up heads, its exploitable. Tails its gonna be ok.

I estimate you will increase your predictive capabilities by a whole 10% using this method.

Re:This is why Microsoft software sucks

[Mateo_LeFou]

>if it comes up heads, its exploitable. Tails its gonna be ok.

In this case, wouldn't there be as many false negatives as false positives?

Re:This is why Microsoft software sucks

[Anonymous Coward]

No, it means that they were able to cut the field of their immediate focus nearly in half while not missing any issues. For such a complex system without any precise mathematical model, that's pretty good.

In this case, flipping a coin is statistically likely to let an unaddressed issue through, and that's a big no-no for applications like this.

Re:This is why Microsoft software sucks

[rugatero]

Hint: 40% is worse than guessing.

No - from TFA:

The index, launched last month, rates each vulnerability using a three-step system.

Random guesses would be expected to yield 33% success.

Re:This is why Microsoft software sucks

[iammani]

Hint: 40% is worse than guessing.

Hmm, lets see...

If they were guessing the answer of an yes or no question, I would agree with you, getting an yes or no wrong 60% of the time is pretty bad.

But I doubt MSFT would find such a measurement for exploits useful. I would think that, they probably, would guess the probability of an exploit code being created. Like, there is a 90% probability that an exploit code would run amok in the internet.

And getting this probability right 40% of the time, is not bad at all, sounds pretty significant to me.

Re:This is why Microsoft software sucks

[mdmkolbe]

40% is worse than guessing only if you have only two choices (e.g. heads or tails). If you have more choices it is a bit better than guessing.

MS was predicting not just whether exploits would appear but the kinds of exploits that will appear. Depending on how specific (e.g. there will be a buffer overrun in module XYZ) or general (e.g. there will be an exploit in Windows *somewhere*) they were about the kinds of exploits, 40% could be either pretty good (i.e. they were insightful) or pretty bad (i.e. they chose the obvious things). In either case they would still be better off than pure random chance.

Re:This is why Microsoft software sucks

[abigsmurf]

No it isn't. Unless of course you assume that for every bug hackers flip a coin and go "heads, I'll write an exploit for this".

40% accuracy in predicting with no false negatives? There are plenty of distaster agencies around the world who would be incredibly pleased with that kind of accuracy

Re:This is why Microsoft software sucks

[Raynor]

Actually 40% is quite good considering, as others have mentioned, that 33% would be the random chance.

it is also worth noting that they have 40% prediction of KNOWN threats.

I would bet there are about as many undiscovered exploits re: these updates, which could drive up or down the percentage.

If I can predict the stock market by +7% over random guessing, that is pretty damn good predicting.

Re:This is why Microsoft software sucks

[Anonymous Coward]

If the steps are sequential, it's less than 33%. The correct figure is 12.5% (50 percent of 50 percent of 50 percent).

Re:This is why Microsoft software sucks

[rugatero]

If the steps are sequential...

They're not - they are three discrete levels of severity.

The term 'three-step' used in the article is a little misleading.

Re:This is why Microsoft software sucks

[poot_rootbeer]

Hint: 40% is worse than guessing.

I'm assuming you meant "worse than flipping a coin". But this was not a heads/tails judgment; it was "for this given defect, is it Highly Likely, Somewhat Likely, or Not Very Likely that it will be exploited"?

Re:This is why Microsoft software sucks

[elgatozorbas]

My understanding is that they (implicitely) assessed thousands of potential exploits. Of these, thousands minus 20 were classified as safe and 20 as dangerous. All guesses from the "safe" category were correct and 8 out of 20 from the "dangerous" category were correct. If all those thousands minus 20 assessments would be taken into account, their statistic would be much better. Even more: it would be fishy if all of the 20 potential exploits would have occurred.

Re:This is why Microsoft software sucks

[Actually, I do RTFA]

Any engineer who says that "40% is pretty good predicting" is incapable of writing good software

It's ZERO false positives, and many false negatives. FFS, elevating 2.25 issues to "immediate priority, someone will exploit this soon" status for every one real issue seems damn good to me.

Shows the confidence they have in themselves

[MosesJones]

Interestingly what they are saying here is that they think that

a) Hackers are smarter than they actually are
b) Microsoft code is easier to exploit than it actually is

So the perception is that Microsoft is better than their prediction, but the implication of that is that Microsoft think they are rubbish.

Maybe all these years of "Microsoft sucks" posts on Slashdot have actually come from the MS security team.

Re:Shows the confidence they have in themselves

[Raynor]

No. What they say is:

You should fix this bug first, since we believe it is the most likely to be exploited.

You can save these for later, since we don't believe it will be immediately exploited.

There is, however, something to be said for hackers referring to this list to find "unlikely" bugs to exploit.

Re:Shows the confidence they have in themselves

[JasterBobaMereel]

So Microsoft thought their code was exploitable and said so, and it was, and instead of doing something about it they just congratulated themselves on predicting it!

Now here's an odd idea rather than predicting if something is exploitable and then publishing it, why not just not write code that is easily exploitable....!

and note the 40% is only the exploits they know about ....so even that is suspect....

Re:Shows the confidence they have in themselves

[maugle]

b) Microsoft code is easier to exploit than it actually is

Wait... how does that work?

In progress..

[mat]

Only 40%, which is already "a success", but they can improve this score, and this would become a triumph !

Re:In progress..

[iammani]

Only 40%, which is already "a success", but they can improve this score,

Ahh you are a manager arent you?

I wish my manager was as optimistic.

Re:In progress..

[duguk]

This was a triumph?*

No... this was a Triumph!

* sorry, as much as I love portal... it's getting old!

Re:In progress..

[mat]

No, I'm George Bush, and now I can say to Microsoft guys: Mission accomplished !

Re:Attention U.S.citizens

[91degrees]

Actually that was John Cleese, even posting anon you should give credit where its due.

Actually it originated with One Alan Baxter of Rochester and expanded by other people on Usenet. So if you do give credit where it's due give it where it's actually due.

Exploitability Threat Level Announcement.

[140Mandak262Jamuna]

Nov 14, Redmond, Washington. Today Head of Vistaland Security of Microsoft, Mr Ima F Anboi announced that Microsoft has raised the Exploitability Threat Level from Light Purple to Sunset Yellow. He urged the users to continue their normal activities and not take precipitous actions.

Microsoft Exploitability Threat Level Indicator is a series of color codes starting from Dazzling Arctic White to Heart of Dick Cheney. Though exact number of these colors is considered a secret, from the past announcements we deduce there are at least 22 million of them.

For PRNewswire, copy edited by Anurag Chakraborty in Bangalore and supervised by Robert Zimmermann in Pittsburgh.

4/9 = 40%?

[Anonymous Coward]

Research also shows Slashdot editors verify submission figures 112% of the time.

Re:4/9 = 40%?

[LordKronos]

Research also shows Slashdot readers read the articles less than 100% of the time:

FTA:

All told, Microsoft correctly predicted eight out of October's 20 vulnerabilities' exploitability, an accuracy rate of 40%.

and in the previous paragraph:

Of the nine October vulnerabilities marked "Consistent exploit code likely," four did, in fact, end up with exploit code available, said Reavey, for an accuracy rate of 44%.

Wow, and I didn't even have to read the article to respond to you. Simply clicking on the link and spending 2 seconds telling the browser to search for "40%" and then reading one single sentence was enough. But I know, that's a lot to ask.

Curious

[tuxgeek]

So ... Are they admitting that bugs in their software that are being targeted by crackers are there by design? Or just incompetence?
If they know their software is filled with bugs, why not just fix them and be done with it before it's released.

Re:Curious

[iammani]

If they know their software is filled with bugs, why not just fix them and be done with it before it's released.

Ahh that wouldnt be interesting, would it be? Microsoft of course wants to release them with lot of bugs. Thats how they get all the free media coverage with MS fixing the bugs while the Open Source community simply does not fix bugs

/sarcasm

Re:Curious

[maxume]

They are trying to make sure that God kills enough kittens to stave off the eventual cat invasion.

but but but...

[3seas]

there is so many to chose from...

Being right 40% of the time...

[tangent3]

...is the same as being wrong 60% of the time.

Doesn't look so impressive when you look at it this way.

Re:Being right 40% of the time...

[Chrisq]

I was going to say the same thing. Still, it didn't do George Bush any harm.

Re:Being right 40% of the time...

[Icarium]

Without knowing the baseline they're working on, this could range from extremely impressive to completely useless.

Ok. So 4 out of the 9 bugs they expected to see exploits codes for actually had exploits meterialise. How many bugs had exploits coded that were not in thier 9 candidates? What is the total number of bugs taken into consideration?

If you were playing "battleship" on a 3x3 board with 4 "ships", taking 9 guesses to hit all 4 would be pretty dismal. Change that into a 30x30 board and suddenly 9 guesses to hit all 4 looks pretty damn impressive.

Re:Being right 40% of the time...

[dubl-u]

Doesn't look so impressive when you look at it this way.

Depends on the payoff.

It's not good if you're betting even money on coin tosses. But if you're a venture capitalist, it's great. The general rule for tech VCs is that 7 bets out of 10 will fail, 2 will do ok, and 1 will be a big success. If that 1 success is buying 10% of Google in the very early days, your 70% failure rate is still pretty awesome, because you're still up billions of dollars.

Re:Being right 40% of the time...

[stupidllama]

If you bat .333 in baseball you might win a batting title, if you do that over your career and your in the hall of fame. All of this and you were only successful 33% of the time.

It depends on which exploits they call correctly

[Gazzonyx]

Granted, they're doing better than guessing... but in reality, I only care that they get it right on the risks that count. They could be 1 for 10, if the harm that the single exploit would cause was more than the sum of the other 9, and be doing decent.

For instance, if they patched the priv. escalation to SYSTEM that has a broad surface area (think, say, remote IIS exploit) over 9 exploits that require physical access and can only get guest access. If someone else has physical access to your box, it's no longer yours, anyways. Risk assessment has to account both for the opportunity and consequences of a given security hole.

Re:Attention U.S.citizens

[Barny]

Ahh, here we go.

http://www.snopes.com/politics/satire/revocation.asp

More exciting than reading about how badly microsoft can classify security bugs eh? :)

ps. NO FIREFOX, I WILL NOT CAPITALISE THE "M" IN mICROSOFT!

Well....

[morgan_greywolf]

Well, I for, one, welcome our new stiff-upper-lipped, bland food eating, emotionless British overlords!

This is what it's come to?

[joedoc]

Microsoft is now bragging about the fact that they predicted 40% of their bugs would be turned into exploits?
I realize that Windows is a complex hunk of crap...errr...operating system, but wouldn't they be better served trying to find and correct these issues rather then just releasing them into the wild and keeping their fingers crossed?
Their attitude is sort of like pointing the gun at your foot and firing five times, and bragging that you only hit two of your toes.
This is why, every day when I arrive at work, I log into this XP box and ask myself why my organization continues to put up with this garbage.

Re:This is what it's come to?

[joedoc]

No, I'm not criticizing Microsoft for patching anything, or not patching anything (hard to say without a double negative). I don't care what Microsoft does.

I use Windows at work. I'm not a sysadmin or network admin (not anymore; I was for 10+ years). I don't work in security (I did, as part of the above). I'm a developer on a very large, distributed and secure network. I don't concern myself with bugs, security, malware, Trojans, etc. etc. because I don't need to. My work is secure and replicated, and if something evil gets on the network and screws up my machine, I can go home early. Tomorrow is another day.

At home, I don't use Windows. At all. Server, desktop, laptop. Not a byte of anything controlled by Microsoft. So, again, their inability to correct their problems before releasing doesn't concern me.

And while I know there are bugs in Linux, fifteen years of experience with it has demonstrated to me that it's a shorter problem, a faster fix, it's out in the open, and there's no one keeping a tally of how long it takes for a bug to get exploited. And none of my boxes have ever been compromised.

And that came out of the middle of my mouth.

Dear Queen

[tjstork]

We would be delighted to become subjects of the crown again, but we doubt that her majesty could afford all the cameras that her subjects are so accustomed to.

PS. The Irish make better beer than you do, and soccer still sucks.

Re:Dear Queen

[wisty]

Well, it sounds like Slashdot has it's quota of citizens who are, as Peter Cook once wrote, "neutral, i.e. from a foreign country, and probably bearing a deep seated resentment towards a nation that once ruled three quarters of the world, and ruled it well".

Re:Attention U.S.citizens

[Exitar]

With the exception of points 7. and 9. it all seems quite reasonable.
Maybe one day you'll learn to drive on the right side.
And vinegar is acceptable on salad only, not potatoes.

By the way, I live in Continental Europe and my ancestors, at the time you were wearing animal furs and piling rocks in bizarre patterns, were building aqueducts.

So, in the end, Her Majesty, please
1. learn to drive
2. learn to cook
3. understand that fox hunting isn't a sport
4. stop using that absurd currency that is the pound sterling

Toss

[ezwip]

Kudos to Microsoft for choosing to toss the coin rather than to spin it. My research team argued for months with the board that spinning a penny instead of tossing it results in heads only about 30% of the time.

Re:It is TERRIBLE

[91degrees]

What REALLY happened is this: Every security hole that MS discovered on its own, was exploited BUT we are supposed to be happy because in 40% of the cases MS correctly predicted that it would be exploited.

No. What happened was this - MS spotted 18 potential security holes. 9 of them were regarded as more serious. A company that focussed on protecting against those 9 would not have been affected at all and would have had less disruption than a company that protected against all 18.

They are offering this as a means to tell their bug fixing department and other companies which areas to prioritize.

Are they serious?

[abcjared]

Predicting something will change the outcome in this case.
Not only will hackers know the most exploitable bug but they will also know the least likely to be updated.
Yet another bloat to innocent users.
FFS can someone make them a feature usefulness vs crap predictor..

Worse than random

[QuietLagoon]

So let me get this correct. Microsoft's determination whether or not there would be an exploit was correct less frequently than if they had just randomly chosen yes or no, and Microsoft calls that good performance?

With such low standards of good performance, it is no wonder that the software coming out of Redmond lately has been so horribly poor.

News at 11

[Henry V .009]

Hackers are missing 60% of opportunities to exploit Microsoft code.

Here's why only 40% of the time

[KWTm]

"This month, we're going to predict whether evil hackers will exploit bugs in our code. What do you predict?"

Steve Ballmer: "No."
James Allchin: "Yes."
Mike Reavey: "Yes."
Jim Gries: "No, I fixed all the bugs."
Sarah Ford: "I dunno. I'd say no; I'm confident in Microsoft."
Val Mazur: "No."
Rui Chen: "Well, the possibility is there, but they'll never prove that they did, so it's the same as no."
Kathleen Dollard: "Of course I will! er --I mean, THEY will. Yes."
Michel Fournier: "How am I supposed to know? How many people said Yes so far? Oh, okay, then I'll say yes."
Bill Gates: "No. Of course I count as part of Microsoft! Write my vote down. No."

Mike Reavey: "Okay, so, what was the right answer? Oh, umm... we were 40% correct. That's not too bad --there's been improvement."

Re:It is TERRIBLE

[Nick Ives]

What REALLY happened is this: Every security hole that MS discovered on its own, was exploited BUT we are supposed to be happy because in 40% of the cases MS correctly predicted that it would be exploited.

I know we don't RTFA but please at least RTFS.

'Four of the [nine] issues that we said where consistent exploit code was likely did have exploit code appear over the first two weeks. And another key was that in no case did we rate something too low.'

So no, at least according to the summary not every security hole was exploited. If you're going to claim otherwise at least provide some links to an article; hopefully one supporting your claims although that's not always necessary for the +5 informative.

In fact I just actually bothered to RTFA, just to make sure, and it said that no exploit code appeared for the low ranked vulnerabilities.

More fail from MS

[foldingstock]

They can predict exploits in their own software. Well paint me yellow and call me a phone directory!

How can a PR team for one of the largest corporations in the US seriously release a statement like this? What kind of company fails so badly that they can only predict 40% of exploits in their own [proprietary] software?

If a major car (or car part) manufacturer "accurately" predicted that 40% of their automobiles would explode and burn their owners alive due to a fuel system defect....would people still buy their cars? Oh right...firestone.

Re:More fail from MS

[Actually, I do RTFA]

What kind of company fails so badly that they can only predict 40% of exploits in their own [proprietary] software?

They had zero false positives. So, put it this way, looking at the source, they came up with some number of exploits that had to be fixed. Other people came up with only 40% of that number.

It's a good thing, both that MS fixed them all, and that outside people seeking exploits are only 40% efficent.

Flip a coin

[KiwiCanuck]

Then their probability will increase to 50%! j/k.

Re:Attention U.S.citizens

[PearsSoap]

And vinegar is acceptable on salad only, not potatoes.

What about potato salad?

/British

the new bar

[mevets]

Microsoft Security Research Centre is a success as a disaster agency? A bit harsh, but I suppose so...

40% with only two possible outcomes

[Britz]

They tried to predict if a hole will be exploited or not. Those are two outcomes. If you were to guess you would end up with a 50% chance of guessing right.

And they were only 40% right and 60% wrong?

Re:Attention U.S.citizens

[camperdave]

Well, roundabouts (aka traffic circles) would make a lot more sense than four way stops or traffic lights in a lot of locations. However, "yeild" doesn't seem to be in the American psyche anymore.

I think it's about time to go fully metric, though. It's a much simpler system. Practically all of the rest of the world uses metric. The sciences and military use metric. Alas, "yeild" doesn't seem to be in the American psyche anymore.

Re:Attention U.S.citizens

[Psiren]

Alas, "yeild" doesn't seem to be in the American psyche anymore.

It's also not in any dictionary that I'm aware of either. Yield is though. Sorry, couldn't resist :)

What about old bugs with old exploits?

[flyingfsck]

The ones I worry about are the 12 year old bugs that have had exploit code for 8 years already and only now gets fixed - maybe.

That's odd...

[catdevnull]

Does anybody else think it's really funny that Microsoft's predicting abilities are better than their patching abilities?

You mean unlike ...

[DrYak]

And a third party finds a bug and reports it to them. Now would be good to predict the severity of the bug, so that the more exploitable ones can be fixed first?

You mean, unlike the press which tends to overblow every report of a vulnerability on Linux and/or FireFox, although in reality the "vulnerability" only work in a few very rare cases where a complex mix of condition. Plus a very gullible and cooperative user who will go through a long process in order to reach the point ...

Thats exactly what they are doing, and they are able to get the severity 40% of the time right, with no false negatives (that not a single severe one has been classified as a low priority one).
So, now, do you think this is bad or wrong or something?

It wouldn't be bad, if it weren't for microsoft's software quality being so bad, that simply calling every bug as critical would be a 100% sensitive 99% specific test.
Today's news almost sounds as if sponsored by Captain Obvious.

Re:Attention U.S.citizens

[camperdave]

It's also not in any dictionary that I'm aware of either. Yield is though. Sorry, couldn't resist :)

[Sigh]

/Me hangs head in shame, and yields to your greater spelling prowess, but is humbly constrained to point out that the duplication of the error was due to copy/paste, and not necessarily a faulty knowledge of "yield"'s proper spelling.

Huh? It's completely backwards!

[Joce640k]

Call us when they rated something TOO HIGH, or OVERESTIMATED the number of exploits, not the other way around.

(boggle)

Re:Attention U.S.citizens

[drsmithy]

Well, roundabouts (aka traffic circles) would make a lot more sense than four way stops [...]

I ran into a few 4-way stop signs on a recent holiday to the US. Exactly how *is* the stalemate of 4 cars arriving at the same time supposed to be resolved ?

Re:Attention U.S.citizens

[warsql]

The election was won by the party that made the rules. Just like 2000 and 2004.

http://dispatch.com/live/content/local_news/stories/2008/10/28/ajudgerule.html?sid=101

In other words...

[DrYak]

Four of the [nine] issues that we said where consistent exploit code was likely did have exploit code appear over the first two weeks. And another key was that in no case did we rate something too low.'

Yup. Trust me, we got it. But given Microsoft's track of security and to get back to the metaphor of grand parent's post :

MS predicted that 40% of the time a coin flipped into the air, would come back down. That ain't a good guess, it shows that MS has a fundemental lack of understanding software security.

Yes. Eventualy from time to time the coin will magically manage to get stuck somewhere and not land down on one side. By saying it will always land on its said, surely they will have a good ratio.

In other words : Microsoft's security is so bad that simply calling all bugs critical is a 100% sensitive 99% specific test. You'll rarely have a false negative that way.
They simply auto-congratulate themselves because sometimes they might predict when the coin won't hit the ground.

Re:Attention U.S.citizens

[St. Alfonzo]

Why by yielding to the right of course! So while everyone is yielding to the right the people in the biggest hurry can slam on the gas simultaneously and meet in the middle. Hopefully thus increasing the average IQ by a small fraction.

Seriously though, I've never had a problem with that. Where it's unclear who was the right of way I just coast into the intersection a bit more slowly than usual so that it's clear who came to a stop first.

Re:Attention U.S.citizens

[flappinbooger]

I watch Top Gear. Isn't that enough?

Re:Attention U.S.citizens

[diqmay]

When four drivers arrive simultaneously at a 4-way stop, the appropriate order shall be determined as follows:

(1) Largest Truck/SUV
(2) Most guns in gun rack
(3) Most teeth

If the appropriate order cannot be determined by the above methods, then the Yuppie Exception is brought into play:

(1) Whomever holds the most complicated Starbuck's drink
(2) Largest age difference between parent and first born
(3) best gas mileage.

Re:Attention U.S.citizens

[flappinbooger]

First in, First out. If 2 cars arrive at the same time, the person "on the right" goes first. If all 4 arrive at the same time? On the odd chance that it happens, the person with the largest gun, tallest 4x4 or shiniest belt buckle goes first. (just kidding, a little brit humor for you)

To be honest, I don't know, and I don't think I've ever experienced it. I've been at 4 way stops with cars in all directions, but there was always a clear order of who went next, it's basically clockwise or in order of arrival.

Re:Attention U.S.citizens

[Penguinoflight]

The car to the left actually has the right of way, seeing as how it is more difficult to see a vehicle on your right, than your left. In the UK, I assume the opposite is true since the drivers side of the vehicle is on the right.

Re:Attention U.S.citizens

[HappySmileMan]

Yeah by allowing homeless people to register to vote they basically rigged the election.
I mean who the fuck wants homeless people to be allowed vote, society has made it clear that they aren't wanted

Thanks, Microsoft!

[scribblej]

No one seems to be looking at this from the opposite angle.

If I'm writing malware that's going to need to exploit Windows, this gives me an easy chart of which exploit I should pick -- the ones with the lowest patch priority, of course.

deep into the 4's?

[sgt scrub]

Warning. If you love Microsoft, don't read. Your delicate sensibilities might be hurt 40% of the time.

Engineering sector 99.999 Microsoft 40.000 Sounds about right.

Closed source software + Security = Oxymoron

[jawahar]

Selling source closed software will be illegal in future.

Re:Attention U.S.citizens

[warsql]

If it was only the homeless, then fine. But you have Acorn all over the place paying people to vote with the only requirement to vote being your name and last 4 digits of your social.
Add to that the top Ohio elections official who doesn't want to verify the identity of new voters. http://www.ohio.com/news/break_news/31101144.html
You end up with dead people voting and all kinds of fraud.
http://www.newsnet5.com/news/17859950/detail.html?rss=nn5&psp=news

Re:Attention U.S.citizens

[camperdave]

No sir. In North America, when two or more vehicles arrive at a four way stop at the same time, the car to the right has the right of way.