Slashdot Mirror
NSS Labs Browser Report Says IE Is the Best, Google Disagrees
adeelarshad82 writes "Independent testing company NSS Labs recently published a report on the ability of popular browsers to block socially engineered malware attack URLs. The test, funded by Microsoft, reported a 99 percent detection rate by Internet Explorer 9 beta, 90 percent by Internet Explorer 8, and 3 percent by Google Chrome. However, Google doesn't entirely approve of this report's focus and conclusions. According to Google not only didn't the report use Chrome 6 for the tests, the current version is Chrome 8; it also focused just on socially engineered malware, while excluding vulnerabilities in plug-ins or browsers themselves. Google defended its browser by claiming that it was built with security in mind and emphasized protection of users from drive-by downloads and plug-in vulnerabilities."
It's not clear why Microsoft and NSS Labs waited until December to release the results.
Maybe it's like the last time this happened?
Furthermore, Moy said, the study started as a private test for Microsoft's engineering team, which was seeking to make internal improvements. "They decided to release it based on the positive results. Many of the test reports we write do not get released by vendors, but they do get used to improve products. So what does 'sponsored' mean in this case?"
So you (internally) strike a deal to test your browser (but also your competitors') with an "independent company" that you pay to perform this service. You get to define the "success parameters" of the test. Then you get the results back and you fix everything. After that time spent fixing has passed, you release the report and add that you have fixed all the problems with your product. Unsurprisingly, you look really really good when this news hits. Since your competitor is not also paying NSS Labs, NSS has no reason to update the report to meet the latest and greatest version of browsers. Meanwhile you can decide if your competitor's browser performed inadequately enough or not for the report -- maybe you even select the success parameters afterward? Heck, you already waited to see if you could release the report.
Independent? HA!
My work here is dung.
Google is complaining that a report on socially engineered attacks is only focused on socially engineered attacks? And they're whining that a study done back when Chrome 6 was the most recent release doesn't mention Chrome 8, which is currently the most recent release? Seriously?
According to Google not only didn't the report use Chrome 6 for the tests where as the current version is Chrome 8...
Should it be:
According to Google not only did the report use Chrome 6 for the tests, whereas the current version is Chrome 8...
I'm well aware of what social engineering is, but what are "socially engineered malware attack URLs"? Those things that pop up in my inbox say "check out this picture of us!" with a link that looks like someone smashed their head on the keyboard?
The test, funded by Microsoft
That says it all.
Free Martian Whores!
An "independent" test that was "funded by Microsoft". WTF? How is that independent?
This: "The test, funded by Microsoft"
The real warning flag is that it doesn't say that on NSS Lab's site nor does it say it anywhere in the report. So if I was being paid to do this, I would have that in big bold letters as a disclaimer on the front page of the report if I wanted to maintain credibility. So either the Google response article is wrong (which the same IE8 report from last year was funded) or you're just being flat out disingenuous when you say "independent." We just happen to receive funding from one of the participants and they decide when and if the report is released.
One more thing, if you dig into this report, the parts where they reference Microsoft read like an advertisement:
It became obvious from this test and comparisons to the earlier test that Microsoft continues to improve their IE malware protection in Internet Explorer 8 (through its SmartScreen® Filter technology) and in Internet Explorer 9 (with the addition of SmartScreen application reputation technology). With a unique URL blocking score of 94% and over-time protection rating of 99%, Internet Explorer 9 was by far the best at protecting against socially-engineered malware. The 89% zero-hour block rate suggests a far superior malware identification, collection, and classification method.
"What kind of registered application reputation technology did you say they used? Simply revolutionary progress!" Compare that section to that same section on Chrome:
With a protection rating of just 3%, Chrome 6 dropped more than 14% from our last test. And, Chrome’s unique URL score of 4% was also a major decline. Chrome’s overall poor protection makes it difficult to compare it to other Safe Browsing API-related products.
"Boo, Chrome sucks!" Hahaha oh my this is too funny. Google shouldn't have to explain themselves. Just take what you can to improve from this report, become aware of your opponent's tactics and move forward.
My work here is dung.
So its results are unquestionably incorrect and/or irrelevant?
Looks like the test was a perfect example of social engineering.
They certainly cannot be considered "independent" or "unbiased" at a minimum. So they aren't of much value until real 3rd party tests are performed.
Seriously. What were they even testing? I was under the impression that social engineering was a security flaw in the user, not in the application. Reading the report, it sounds like they were just testing the browsers' databases of known malware/phishing sites. Which, really, has little to do with the security of the browser itself.
...researchers discovered that hot supermodels would be most fulfilled in a relationship with Slashdot user GodfatherofSoul*.
* This study funded by GodfatherofSoul
I swear to God...I swear to God! That is NOT how you treat your human!
So its results are unquestionably incorrect and/or irrelevant?
They may be technically true in some sense or other. However, in past such situations, Microsoft has been seen commissioning several similar reports; possibly even iterating the instructions for running the reports; then throwing away (under NDA) all the ones which don't match with their marketing wishes. You can basically assume that whatever it says is the opposite of the truth in some way or another because if it was true they would be able to just say directly it instead of commissioning someone else to say it to they can avoid claims of false advertising (for example, their old "Get the Facts" campaign was one of the few things of this type the ASA has clearly stated was misleading). And yes; most companies do this to some extent, but few other companies could come near to sustaining the level of deception Microsoft does because eventually some employee would become disenchanted and start leaking results. For example, have a look at the Comes documents, which only came out because of a lawsuit, to get some idea of the kind of things they can keep secret. Nowadays Microsoft's data destruction policies are much stricter and they ensure that all deals are finalised by lawyers and so are legally privilaged. This kind of secrecy and professional deception means that almost any marketing claim from them should be disregarded completely until there is some level of independent confirmation.
=~ s,(.*),<sarcasm>$1</sarcasm>,g if any_point_you_wish();
UL is to test your products for saftey, this is a *comparative* test against several competing products for quality.
Apples, meet Oranges, meet troll.
a handful of selfish greedy people are no match for millions of selfish, greedy people -u4ya
...Or posts on a site that promotes open source and LAMP stacks and images Bill Gates as a Borg. What I find interesting is how no one questions the monthly posts here about IE losing market share from a site (Net Applications) that only polls their own clients, but no one ever points that out.
The test, funded by Microsoft
That says it all.
And the response from google criticizing it was by someone right on google's payroll representing google's interests. I guess we can ignore their criticism then too?
Or perhaps we should let the work stand for itself, evaluate the methodology, strip away the marketing spin, and come away with some nugget of truth, regardless of who funded it. Of course that's "work".
Tests like this are done for marketing purposes. The professionalism of the tester will make sure the test is rigged to give Microsoft the result they want. Get the facts.
I think you missed the other important part: "Also, the version of Chrome that NSS says all this about is two major versions behind the current stable release, while the version of IE they say is better is the current beta release."
A more relevant comparison would be IE 8 to Chrome 8 (current generally release version of both version), or IE 9 to Chrome 9 (current publicly available pre-release version of each browser.)
Perhaps someone should do a similar comparison, but using Chrome 9 and IE 6, instead...
The report is of greater value to Microsoft, the paying customer, the less obvious it is the Microsoft is the paying customer.
I don't know about you but I rarely receive tarballs, rpms or debs from friends to compile or install on IM or facebook. That's the good thing about the repository system, where there is a (hopefully) trusted source where you install the majority of your applications.
I can't really see socially engineered malware taking off under Linux, really.
:. Ultimate Control Dedicated/VM Servers
I know this isn't in the spirit of the other posts on this topic today, but I applaud MS for concentrating on security and the best interests of their end users. It's good to see they are taking these matters seriously as part of the product development process.
Don't get me wrong, I'm always happy when security is improved -- even in the most hated of products by the most hated of companies. The problem I have is when marketing gets a hold of this and spins it to attack competitors, thereby improving the public perception of their own product. This could have all been avoided had Microsoft just kept the report internal like most of NSS Labs' customers. And doing so while comparing the latest IE9 to Chrome 6 and releasing that to the public as a 'current' report now ... well, that's what I have a problem with. If a Chrome user read that report as today's news they're going to think that it's been done with today's Chrome.
My work here is dung.
What the Faceless Google rep said was that this test cannot be peer reviewed because they did not release all the data (specifically the URLs visited). Now releasing a report that does not allow for independent review does not make for good science.
The tests may be valid. But until there is enough information to confirm this, I can only be skeptical of the faceless Microsoft rep.
A sig is placed here
To display how futile
English Haiku is
Woah.
I haven’t seen style this terrible in a long, long while. Even the GNAA trolls are more legible.
Ignore this signature. By order.
This is totally different.
In this case, the tester tested two products and rated one "99%" and one "3%" against some standard.
The key difference is that UL tests against a pre-existing standard. Not a standard that they made after looking at the product. UL can't customize their test to make one product look better or worse.
The methodology might have been totally bogus (no idea), but the act of paying for the test isn't automatically so.
The act of paying for a test to be designed for you, or a test you designed ahead of time to make your product look good, is bogus. Paying to have a test executed for you is not bogus. One is independent, the other is not.
The report is almost useless because it has compared the latest stable and dev releases of IE with versions of Firefox and Chrome that are years old.
What. No, wait, what?
Read on to the end, because later I'm going to tell you what's really wrong with the test and why it's bullshit, but I have to first burn down the obvious straw man you've introduced.
The report was released in October 2010. http://www.nsslabs.com/assets/noreg-reports/NSS%20Labs_Q32010_Browser-SEM.pdf
It used Google Chrome 6, which was the current stable Chrome at the time (6 came out in September 2010). Google Chrome has gone from 6 to 8 in two months. It used Firefox 3.6, which is the current stable Firefox RIGHT NOW, two months after the report was released. 3.6 was released in January 2010, but Mozilla has only done "dot" releases since October. It also included Internet Explorer 8, which was released in March 2009.
In other words, if you want to say "older is worse", then IE8 should have been absolutely fucking pasted by this test. Ummm, right? It's the oldest browser in the test by almost a year.
Now we get to the point that won't upset you, because THIS is what is wrong with the test.
According to their test, what they were really testing was vendor responsiveness to known threats (on-time maintenance of the blacklist), not some response internal to the browser. They took a bunch of really recent entries of bad sites from someone and plugged them into the browsers, getting a new batch of URLs every few hours. The time was measured in hours, so what this is really saying is that Microsoft seems to be the best vendor at maintaining the server-based "bad URLs" list, though it took them 4 hours on average to block sites as opposed to Firefox's 6 hours.
If they got these sites from their paid sponsor, then the list could easily have been biased. But there's more actual provable bias to the test than just that.
The real bias is in the percentages. They do not actually represent "Microsoft browsers blocked 90% of sites while Firefox only blocked 20%". they are a grade-type score, where 100% means all sites were blocked immediately, while a 0% means no sites were blocked, ever. Early detection (measured in hours) seems to play a much larger role than actual number of sites detected. The scores appear to have been done on some form of normalization curve, with the sweet spot being somewhere around "One Half Hour Longer than Internet Explorer".
Otherwise, how does an increase in response time from 4 hours (IE, both versions to within a few minutes plus or minus) to 6 hours (Firefox) make your score go from 90% to 20%?
The net conclusion is, if you're going to use a web browser and you depend on vendor-maintained "baddie" lists as your primary line of defense (rather than script protections like NoScript, which don't depend on a vendor to maintain stuff for you), you're better off with Internet Explorer than any other mainstream browser in the market.
It doesn't make you "70% safer" or protect you from "70% more threats", it means that it has, on average, 2 hours of lead time on the next-best browser in terms of the list of sites it protects you from. It's like saying that McAfee is better than Norton because McAfee generally releases specific virus signatures, on average, 2 hours before Norton does.
So, the test is correct, it's just expressing the results in a very misleading way, showing a very low number for "everyone but Microsoft" because the test results were designed to score what IE did best in the highest way possible. They even spelled that out in their results:
The value of this table is in providing context for the overall block rate, so that if a browser blocked 100% of the malware, but it took 264 hours (11 days) to do so, it is actually providing less protection than a browser with a 70% overall bloc
"This post contains words, known to the State of California to cause thought. Wash brain thoroughly after reading."
Lame troll is lame. How many boxes do YOU repair in a week? I average about 6, sometimes more when I've not got so many builds in progress. And I can tell you that without a shadow of a doubt that socially engineered attacks account for a HUGE amount of infections and is in fact growing rapidly. try looking up "Security Tool 2010" or "Rogue AV 2010" and looking at the numbers these things are racking up. As home users slowly move away from XP to Windows 7, which has file and registry virtualization, ASLR and DEP, and which you can even easily add Structured Exception Handling Overwrite Protection , the low hanging fruit is increasingly becoming PEBKAC. I can tell you I see socially engineered bugs spreading a hell of a lot faster on newer OSes than I do anything else, whereas with XP it is still drive bys thanks to running as admin. As XP dies out this problem will only be getting worse.
Now I don't recommend IE OR Chrome to my customers, as I don't like the data mining in Chrome and have had bad luck in the past with IE, if MSFT can get 99% of the social engineered bugs blocked, along with someone cooking up something like ABP for IE 9? Then I'll be happy to recommend my customers use IE over other browsers. I'm already starting to get pissed at Mozilla for refusing to support low rights mode in Windows 7 even though this tech has been out since 07 simply because Linux doesn't have it. Chrome mines waaay too much data for my taste, so that leaves Comodo Dragon and IE. Does anyone know of a good ABP for Chromium based browsers? Or an ABP for IE 9? Because in the end ABP will be the deciding factor for me and my customers. If IE 9 can block 99% of the social engineered attacks while I can block ad based attacks with an ABP clone then it is a no brainer to switch. I just wish the Mozilla team wouldn't act like asses and refuse to support a technology that would help protect so many simply because it isn't supported on a platform that doesn't need the damned thing anyway.
ACs don't waste your time replying, your posts are never seen by me.
I can only be skeptical of the faceless Microsoft rep.
Agreed. Skepticism of every studies conclusion is healthy and necessary. However outright disregard for a study based on a single data point: "who paid for it" is not.
I believe that most people who will be influenced by this kind of report are NOT in a position to methodically evaluate the test methodology. They are people who watch Survivor, Big Brother, YourCountryHere Idol and idolize Oprah. They do not have the experience or skills for critical analysis of marketing spin. So when Microsoft (or McDonalds or the US Govt or Buy n Large) claim research that shows their product is superior to others, the reader gets one claim stuck in their head and it is repeated as fact*.
Of course that's a sweeping generalisation; there are many who do think critically (it's possible that some critical thinkers watch big brother but I expect the number is small) but it makes my point.
*Which incidentally, is why I think we should teach critical thinking at all levels of school, not just leave it until university.
Why can't we let people believe whatever they like? It's not like a little religion has ever hurt anyone.
Additionally Google's statements about the study should set off the VERY SAME red flags about googles statements.
No, Google's complaints don't set off the same red flags at all. Microsoft citing a third party study is an appeal to an external authority. The claim is that Microsoft is trying to get their opinion on their own browser credibility by having it come from a mouthpiece that isn't first party. There is no analogous complaint to be made about Google, because they aren't trying to complain about the study by hiring an external firm to make the complaint.
However, it is completely unrelated to who paid for it, now isn't it?
Exactly the opposite. The study arguably has that particular problem because Microsoft paid for it, for a couple different reasons. Either there is a flaw in their methodology that they are hiding because they are being paid for a specific result, or it was an unintentionally flawed study that was alone among dozens of other studies that Microsoft ordered at the time in arriving at a pro-Microsoft conclusion, or Microsoft got to pick the study targets to their benefit, or any number of other things that I can't think of off-hand. Either way, it is a conflict of interest and we should throw out the result regardless of how valid it might appear because there is no other foolproof way of making sure we avoid accepting corrupted study results.
It doesn't mean that much when you consider that Chrome can't be trusted not to pass information about you to Google.