Slashdot Mirror

← Back to Stories (view on slashdot.org)

Microsoft Explains Why Windows Defender Isn't Ranked Higher in New Antivirus Tests (zdnet.com)

Posted by msmash on from the closer-look dept.

In its most recent reports, AV-Test had very few flattering things to say about Windows Defender. Microsoft's security suite was rated as the seventh best antivirus product in the independent test. In total, 15 AV products were tested. Microsoft, however, has now disputed AV-Test's methodology and conclusion. For some context, the top AV products rated by AV-Test on Windows 10 were Trend Micro, Vipre, AhnLab, Avira, Bitdefender, Kaspersky, and McAfee.

Windows Defender was able to detect 100 percent of new and old malware, but it lost few points for performance (which, AV-Test measures on the basis of how a security suite slows applications and websites on the test computer); and usability (which counts false-positives or instances where AV wrongly identifies a file as malicious.) From a report: Windows Defender's performance rating was dragged down because it slowed the installation of frequently used applications more than the industry average, and wrongly detected 16 pieces of legitimate software compared with the industry average of four. But Microsoft wants enterprise customers to know that Windows Defender is only half the picture, given the option for customers to also deploy Windows Defender Advanced Threat Protection's (ATP) "stack components" including Smartscreen, Application Guard, and Application Control.

In the January and February test Windows Defender also scored 100 percent on protection. However it did miss two samples. Since then it's retrained its machine-learning classifiers to detect them. But Microsoft notes in a new paper that Defender ATP did catch them, which isn't reflected in AV-Test's or other testing firms' result. Microsoft hopes to change this so that testers include so-called stack components available in ATP. "As threats become more sophisticated, Microsoft and other security platform vendors continue evolving their product capabilities to detect threats across different attack stages," Microsoft's Windows Defender Research team writes. "We hope to see independent testers evolve their methodologies as well. Our customers need greater transparency and optics into what an end-to-end solution can accomplish in terms of total preventive protection, including the quality of individual components like antivirus."

9 of 85 comments (clear)

  1. Attack surface by sinij · · Score: 4, Insightful

    MS Defender has one very clear advantage over competition - it doesn't create an additional attack surface and installs yet another vendor's application with deep kernel hooks, network connectivity, and an equivalent of root privileges.

    1. Re:Attack surface by Anonymous Coward · · Score: 3, Insightful

      but it lost few points for performance (which, AV-Test measures on the basis of how a security suite slows applications and websites on the test computer);

      I would like to know which non-Microsoft AV is this polite. Long, long ago, McAffee was a minimal AV option, but then it joined Norton and all the other "security suites" as a bloated and unwieldy mass of advertising other McAffee products and panicing over 1st party software patches.

    2. Re:Attack surface by phantomfive · · Score: 5, Informative

      it doesn't create an additional attack surface

      Unfortunately, yes it does.

      --
      "First they came for the slanderers and i said nothing."
    3. Re:Attack surface by danbert8 · · Score: 4, Interesting

      I use Windows Defender because it's the only AV that isn't worse than the viruses it is supposed to be protecting against...

      --
      Yes it's an anecdote! Were you expecting original research in a Slashdot comment?
    4. Re:Attack surface by butzwonker · · Score: 3, Informative

      An additional attack surface is one that exists if you install and run the software but doesn't exist when you don't install or run the software. Microsoft Defender adds an additional attack surface like any other antivirus software.

    5. Re:Attack surface by Riceballsan · · Score: 3, Insightful

      so by surface, you mean company? Windows defender is an attack surface, in the sense that it is a piece of software with admin access that rests in addition to the OS as a whole and can in some situations be tricked into doing bad things. If you install bitdefender or something else they generally disable windows defender, which closes down those possible attack vectors, and replace them with whatever the other protection's vectors are. No matter what protection you are using, you've got the same number of attack surfaces, it's just that all attack surfaces are owned by the same company, instead of by 2 companies.

  2. Relative rankings mostly worthless. by Anonymous Coward · · Score: 4, Insightful

    Anyone should understand that Relative rankings are mostly worthless. If all the products in the top 10 are excellent, but one product has slightly less points than the top 9, does it really matter than it ranked 10th?

    The main advantage of Windows Defender is it's free. For most people that trumps all the other rankings. It's free, it protected against everything the competition did, it's nearly as usable, and slightly slower. That's good enough to not buy something else.

    The AV vendors should be quaking in their boots. Why would you buy another product when what MS puts out is generally fine? My guess is they'll improve the usability a bit, and they'll rank in the top 3. Then start saying goodbye to several of the other AV vendors.

    1. Re:Relative rankings mostly worthless. by ranton · · Score: 3, Interesting

      The AV vendors should be quaking in their boots. Why would you buy another product when what MS puts out is generally fine?

      One reason is because many users have learned they should pick an anti-virus software suite every time they go to Dell and order a new computer. Retailers have an incentive to only offer paid versions because they will get their cut. So many users will keep on choosing either McAfee or Norton just because those are options they are given.

      I'm not sure how many users this describes, but my guess is a lot of them. Then again any significant loss is sales should have them quaking in their boots.

      --
      -- All that is necessary for the triumph of evil is that good men do nothing. -- Edmund Burke
  3. defending defender... by Anonymous Coward · · Score: 5, Interesting

    Ok, direct experience here, and I am absolutely no fanboy of ms software. But, as part of a offensive security cert a few months back, I got heavily into writing and compiling windows exploit code, and one of the course exercises walk through testing a piece of malware by the virus total site.
    So as part of my studies and self learning I wrote a non self propagating malicious exploit, but it did elevate privileges from the user to admin and get access to things and start calc as a admin user to prove it was exploiting. I took a common windows POC exploit and modified it heavily in ways I will not discuss to a wider audience (because teaching people av evasion techniques is best left to offsec and their ilk, to the right people) and compiled it.
    Out of sheer curiosity I submitted the original POC code, one encoded by a old common packer & my heavily modified "malware" to virus total, and the original and encoded packed version was picked up by about 45/47 av's straight off. The *ONLY* av that managed to detect my custom payload was.... Windows Defender. It must have opened the executable and saw where it hooked when it shouldn't, and the competition seem to rely on pattern matching instead.
    So yeah, sign me up for free windows defender. When the subject comes up with lay people who ask me what to use, its what I would recommend them. From first hand testing.

    Anon, because even with all the above, I'm basically admitting to authoring a custom exploit, and while I'm employed in this field, I could do without the extra attention.