Slashdot Mirror
'Leak' Test of 21 Personal Firewalls
mork writes "Matousec.com, as part of a larger analysis of personal firewalls on Windows, has conducted a thorough leak test of 21 pieces of firewall software. Leak tests imitate common methods used by trojans or spyware to send your information from your computer. Windows Firewall XP SP2 fails every test, so the fears that the days of third party firewall software was over seem groundless. Surprisingly the two top programs are both freeware." From the article: "Some firewalls totally failed tests made against their default settings but their results on the highest security settings were much better. Kaspersky Internet Security 6.0.0.303 is the product with the biggest difference between the default settings score and the highest security settings score. Another such product is Safety.Net. Some products like BitDefender, F-Secure, McAfee, Panda, etc. include antivirus engines. The sad and funny thing in once is that lots of them mark leak-testing software as viruses or malware."
I think I'll stick with PF
Windows Firewall XP SP2 fails every test, so the fears that the days of third party firewall software was over seem groundless.
The fears aren't because MS figured out how to build a good firewall; the fears are based on supposed "features" in Vista that would make it very hard/impossible for third party vendors to access parts of the OS needed to build good security software without first going through MS for some kind of certification. Not only that, but as MS integrates other security into Windows, like anti-virus, it may become very difficult to install third party AV and firewalls because the built-in AV wouldn't allow it.
Now, I'm not sure how much of these fears were grounded in reality, but I'm pretty sure they had nothing to do with some perceived accomplishment of the built-in Windows Firewall.
This may seem obvious to me.... but the leak-testing software's imitating how a virus or trojan sends messages to the net, right? Wouldn't that of course mean that anti-virus software is going to mark it as malware?
I mean, the anti-viruses must be matching either the behavior of the program itself, or the signature of that data-sending bit. Of course they'll think it's a virus.
+++OUT OF CHEESE ERROR+++ REDO FROM START +++
Just to say I've been running comodo for ages, and find it great to use. Slows down the computer allot less then Norten and is far easier to customise and make rules for. Not to mention it has a very helpfull message board and its free. Comodo Site.
Yes, but how many of these firewalls run on Linux?
I've really only seen Linux firewalls based on iptables/ipchains. I use one, called TuxGuardian (try Google/SourceForge if you want a link) that seems to work well.
What is "sad and funny" about catching a program that uses the same techniques as malware, techniques which are outside the range of normal software, and flagging it as potential malware?
It's also annoying to see a firewall listed as a failure because it's a firewall and not a host-based IDS.
I'd also argue that the host-based IDS programs are being sold for a purpose that is not their best use. Once a system has malicious software on it, expecting a process on the same machine to protect you and itself is, um, optimistic. Sure they try to defend themselves but that puts them on the wrong side of an arms race.
What they're best for is monitoring and control of "legitimate" software. I have Zone Alarm set to prompt me every time a program tries to run IE6, and to block media players from phoning home to whisper about what I'm watching.
Leak tests are not enough. ;)
Another test to perform: just browse some adult oriented sites. That's the so-called "lick test". If your firewall licks, then it sucks !
-- Rastignac was here.
I've used the free version of ZoneAlarm for years and I've always been happy with it. Does anyone know how the free version compares to Pro? There probably isn't much difference.
PS- I use AntiVir for virus protection and have been happy with it as well.
There are many tongues to talk, and but few heads to think. -Victor Hugo
The "personal firewall" in Windows XP SP2 was never advertised to block outgoing connections. In fact, this PDF states: "Windows Firewall blocks unsolicited incoming traffic. However, you cannot configure Windows Firewall to block outgoing traffic."
So of course it failed every test.
They tested Zone Alarm PRO, and it tested very favorably. Can we assume that the free version would fare as well?
My mom always said, "Jim, you're 1 in a million." Given the current population, there are 7000 of me. God help us all!
Yes, the pro has more stuff to control your environment plus comes with antispyware and anitvirus which the free version doesnt, all in all it is my favorite all round tool to have although i keep spybot running as backup plan2
What's important is a firewall stops incoming traffic, to prevent worm attacks.
Stopping outgoing traffic is for the obsessively insane.
[sarcasm] Ok, so let me get this straight. I am stupid enough to allow something to be installed on my system like a trojan or malware, but I'm supposed to be smart enough to secure my system to prevent them from getting back out? [/sarcasm]
I have used firewalls that let me control my outbound. I've found them to be a pain in the ass because I have lots of things that need to get out. And of course every time I update one of them I have to update my list. Try using a Firefox nightly and changing it at least once a week and you'll soon be tired of that. I protect my system by scanning things I download, running A/V, and occasionally verifying my system with an automated spybot check.
The product I used for a long time, Outpost, is there. It's good but it has too many issues. However where's Core Force? It's not a decent roundup if they didn't test that.
Leak tests imitate common methods used by trojans or spyware to send your information from your computer.
This is the least important piece of security I care about on my PC.
If there is a trojan already running on my PC, then I have already lost the war. It is irrelevant if it can communicate directly with an outside server or not. It could send data in a PLETHORA of undetectable ways aside from this (could send stealth emails from my default email program, could post data stealthily in a hidden frame it sets as my browser start page, etc etc).
The goal is to not get the spyware and virii on your PC in the first place. Once it's there, you're already screwed.
I notice that there was no column in there about how aggravating the installed firewall rendered your system. How many of those firewalls are going to try to pop up a dialog box on a game that just went full screen and freeze the game (so you can't even alt-tab out) until you click on a box you can't even see? I mean I could have designed a firewall that would easily pass their tests with 100% reliability, it's called "unplug the network firewall", and it's very simple to install, just reach behind your computer, find the ethernet cable, and pull it out. Viola! Perfect Score!
One thing that struck me about Windows Firewalls as compared to Unix firewalls is that Unix firewalls are focused on keeping malicious traffic out of your machine. Windows firewalls are designed to keep malicious traffic from getting out to the internet. In the end, it's no surprise that the results are a mixed bag, once your system is compromised you really can't expect these firewalls to save you. It's a lot like the antivirus market, where you have a constant arms race between the virus writers (do people write honest to goodness viruses anymore?) and the antivirus companies.
My final complaint is that programs like ZoneAlarm Pro are exceedingly resource hungry for what they do. ZoneAlarm takes over a minute to start on my fairly modern laptop, whereas everything else in the system takes about 30 seconds or so total. Why does a firewall need 24 MB of resident memory?
I read the internet for the articles.
Our experience is that ZoneAlarm is fine. We've checked using the SysInternals Process Explorer, installed as Task Manager, hundreds of times and never found that ZA was using too much CPU power. This checking was done on perhaps 25 computers over a period of many months.
ZoneAlarm sometimes gives false positives, but that is a small problem compared to worrying about networks being infected.
If that's the case, they shouldn't have included any outbound protection at all. It's just lazy to half-ass outbound protection then claim "well it was never meant for that use". Don't include it at all if it was never meant for that use. The outbound protection it has now is basically useless and is a hindrance more than anything.
If an officer ever threatens to taze you, say you have a pacemaker.
Blocking incoming problems with a program running on the machine and being started way after the system already accepts connections is asking for trouble. Yes, the time frame in which you're vulnerable is (unless you're starting a bunch of processes) maybe very small, but seconds become millenia in a computer that can process a few million lines of assembly code per second.
Outgoing is, given the amount of problem programs that come piggybacked on other software today, at least as problematic. All it takes for an infection with a malicious trojan is a malformed page on the web. And I'd love to know when that thing starts phoning home, so I know that something running wrong in my system.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
It might have been nice if they'd compared all of those to a handful of routers. Firewalls are great, but I think a lot of home networks now consist of at least 2 hosts. I'd be curious to see how they fair.
Quote: The "personal firewall" in Windows XP SP2 was never advertised to block outgoing connections.
Why did people moderate that comment up? Microsoft never claimed it made good software, so the quality of its software should ignored?
George W. Bush never advertised himself as a moral person, so he shouldn't be impeached? The U.S. government never advertised itself as non-violent, so the fact that it has killed 650,000 Iraqis should be ignored?
I had used the free ZoneAlarm on my W98 boxen for years, always happy with it and (AFAIK) never got hacked (plenty of attempts though!). When I upgraded the machines to W2K I found that ZoneAlarm screwed up the user profiles. The, apparently was a "known problem" and I chose to uninstall it. Shorly after I got broadband and am using a router/hub which has its own built in firewall.
But Zonealarm was good, easy to use. I imagine the 'Pro' version would be just as good.
*--BigMan--- Time flies like an arrow.. but personally I prefer a nice glass of wine!
ZoneAlarm was tested by the company that did the leak testing.
Quote: ZoneLabs "programmers lack important knowledge needed for writing security products for Windows NT operating systems."
This fits with our experience. ZoneLabs was sold to CheckPoint Software. After that, ZoneAlarm seemed to have many, many problems.
Comodo Personal Firewall 2.3.6.81 diapers for their outstanding level of anti-leak protection.
...Is that the test fails to simulate the overall protective capability of the firewall. In order for the outbound traffic filter to be relevant, the offending software has to first get past the inbound protection as well as antivirus/antimalware protection that is resident on the machine. I think ZoneAlarm, AVG and others who provide a complete suite have a better solution than the best leak protection firewall out there, because in effect these programs never are allowed to execute in the first place.
Lets run these tests again, but first specify that you must accomplish the tests without physical access to the machine. Then lets compare how these products fare...
They have an "interesting" business model. Basically, they do voluntary security checking on software, then SELL the information for a set price. It comes with a not-so-veiled threat of releasing the information, although they do offer to sell the bugs to the vendor first.
On the surface, it looks like blackmail. "Nice firewall you got here, sure would hate to expose a hole in it..." But when you consider how much work is involved, it's more like being forced to hire these people for their results. Kind of like paparazzi getting a picture of a celebrity in a compromised situation.
Frankly, I disagree with the business model, simply on the basis that they are positioning themselves as contractors to a company who were never asked to be hired nor interviewed. The claim is that they force the security companies to make more secure products. I guess this is true, although their motives are not so pure. Certainly the work is worth something, but they want money for the work they performed, even though they were never actually asked to do the work... that's the aspect that bothers me.
Matousec, which did the testing, found that the Comodo free firewall is the best. Are Matousec and Comodo completely separate organizations? Matousec is Japanese, and English is clearly not the native language of whomever runs Comodo.
Matousec's review covered "personal firewalls", an artificial category which may eliminate products of interest. For example, Comodo doesn't recommend its own firewall, it recommends the Trustix Enterprise Firewall, which is free, also.
At minimum, this is VERY weird. I'm not saying there is anything wrong, but anyone should wonder when all the traditional companies are shown to be producing products of poor quality, and three new companies are shown to be the most trustworthy. Especially when two of those companies give their products away free.
I've thought for years that Symantec and ZoneLabs were not hiring enough people with technical knowledge; their products show that. I discovered that Sunbelt Software was doing something fishy. Certainly the major suppliers have shown many examples of bad behavior.
But, what about these 3 new companies? How can it be true that they are immediately better than all the others?
"Stopping outgoing traffic is for the obsessively insane."
No it's not. Your software is chattier than you assume. A lot of software "contacts" home every time you start it and for reasons that aren't document as to why or what's in the payload.
Is that obsessive to know what 3rd party programs are doing and block it if you don't think it helps you? Not at all. That's what used to be called "common sense" before some people decided they would let their machines be under some stranger's control so they can use a "free" screen saver.
Are these likely? Well, some have probably already occurred, others might never occur, and the rest might happen to your computer as you are reading this text. The whole point is that you have zero idea what form the next attack will be in because it hasn't happened yet and you have zero idea of what is trustable because the chain of trust is very very long and almost totally outside the scope of us mere mortals to validate. ANY break ANYWHERE in that chain can invalidate the assumptions made on a one-sided firewall. If components outside of the firewall were safe, you wouldn't need the firewall. Since the firewall is of value, you are implying that total trust on outside components is unsafe and by inference that a total trust that the outside will make no attempt to appear to be on the inside is also unsafe.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
The test programs imitate malware. If we don't detect them we are accused of being ineffective. If we do detect them we are accused of being overzealous. There's no way to win that battle. We detect them and note in any extended information exactly what their purpose is (if known).
--
http://www.moosoft.com/
Once infected, what's to stop the program from changing the firewall to allow outgoing without notifying the user?
Software firewalls are to keep you from being attacked in the first place, or possibly for privacy. They won't protect you once you're infected.
I went for the Comodo firewall today, did the install and as part of it the installer suggested I disable the XP firewall which I agreed to. Then it wanted to reboot but I was (am) in the middle of something that will take another hour or so. The problem is that the installer has already turned off the XP firewall and the average time to penetrate an XP machine without firewall is under an hour. So I turned the XP firewall back on until I reboot, but how many would know to do this?
Why can't the installer turn off the XP firewall _after_ a successful reboot/config/test to make sure it is installed and running.
I come here for the love
Matousec is a Czech group (at least, the whois data gives contact address in Prague).
The personal firewall is not an artificial class (I would say) - they're firewalls meant for installation on a single host that is not a gateway between networks, and whose administration functions are meant to be accessible to a non-technical user. You can argue over whether one product or another should fit in that category,
The list of compared firewalls on Comodo's site that you link to, is a comparison of a few firewalls designed for the a completely different deployment scenario - a gateway device that serves as a firewall and only as a firewall, separating two or more networks. Trustix is a hardened Linux distribution. The Trustix Enterprise Firewall is a gateway firewall for an organization's network - a significantly different beast from the Comodo firewall, or any other endpoint firewalls, personal or otherwise.
There is a connection between Comodo and Trustix - Comodo are the chief maintainers of the Trustix distribution, and offer commercial support for it.
Thanks for the information. It's very useful.
Now, we still need to address why we have never heard of these companies before today, and now they are the best?
Sure, let's put a NAT router in there -- and how does that help with outbound connections? If, by default, it were not transparent, it would generally be returned as defective.
Ok, let's put a non-NAT router in there. If THAT isn't transparent by default, it would definitely be returned as defective.
So how DOES a router compare at all?
Now, if you obtained your router from your broadband supplier, port 25 outbound may be blocked (I've never seen this, but it IS possible). That may be acceptable. But try blocking bittorrent...
Having done the mental experiment, I will tell you the answer to your question -- the router would fair the same as the built-in XP firewall. It would "fail" all the tests.
Thanks for reading,
Ratboy
Just another "Cubible(sic) Joe" 2 17 3061
Actually many routers have the ability to create firewall rules and filter various things, both inbound and outbound. Hence why it would have been useful to compare their abilities to the various firewalls.
Hahahaha... Mcafee reports the leaktests.zip on that site as being infected with Exploit-ghost... Quality.
turn off the radio. duh.
To have a right to do a thing is not at all the same as to be right in doing it