Slashdot Mirror
Network Intrusion Detection Systems Fail to Impress
TheBongPipe writes "I'm reading a nice test here about 7 commercial IDSs. Who won the prize? Nobody..." They also looked at Snort, but found that all the products generated way too many false alarms.
They also looked at Snort, but found that all the products generated way too many false alarms.
Too many false alarms isn't necessarily a bad thing. In intrusion detection you'd rather take the false positives, than the alternative.
The rate of false fire alarms, and false burgular alarms is VERY high compared to the actual number of real emergencies.
I had a nice experience using snort.
:/ Point and click is not always the best solution...
Come on, reading the article I saw the guy said a Snort disadvantage is not having a GUI. What kind of technical user this guy is?
Fabio - Sumare/Sao Paulo/Brazil/South America/Earth/Solar System/Milky Way/Universe
http://www.morroida.com.br
It'd be nice to have some more detail on their results. The chart on the page shows Snort detected all the attacks listed in the chart except the SYN flood. And the footnote on that entry says Snort was down because of "configuration error."
Gee, whose fault is that?
They also go on to mention all ask too much of their users in terms of time and expertise to be described as security must-haves. IDSs are not screen-savers. Those who are setting up an IDS better have a good understanding of how they work and how to configure these applications. Point-and-click doesn't really apply to something this involved.
Like Car Alarms, if it goes off all the time, people will just ignore it -- At some point, the noise drowns out the signal.
You would hope that the increase in false positives decreases the number of false negatives but that isn't necessarily true either.
I am not a number! I am a man! And don't you
I would also like to remind everyone having pride in their own IDS that NIDS will never catch every single attack. (At least for the next little while)
Signature based detection is only good if the attack utilize abnormal or unique traffic to exploit the vulnerability. It will not pick out attacks that uses normal common traffic (for obvious reasons).
IDS evasion techniques are also heavily worked on, plus all application level evasive techniques (eg. sidestep). We can just never be totally dependent on the NIDS for telling us intrusion has occured. It works for most attacks but will fail for some.
If you look at the table, snort looks like it was doing great, except that it somehow missed the SYN attack. So, based off that chart, none of the IDS corrected detected all of the attacks... however, you read on a bit further, and..
Snort was off the air at the time of the attack because of misconfiguration on our part.
I don't have a lot of confidence in their results.
Well, that's true, unless
ALERT: Intrusion attempt detected! User Liora mistyped his password!
the rate of false-positives is high enough that
ALERT: Account Liora has recieved login attempts from three different IPs in the past 12 hours!
you stop paying attention; unlike with AIDS testing (which has a very high false positive rate) the user is simply likely to ignore the system even if real threats occur
ALERT: 1,262 attempts to login as user "root" in past 7 seconds!
So it becomes desirable to lower the false positive rate to a manageable level, WHATEVER the rate of false negatives is, because otherwise you won't actually catch anything.
The purpose of the AIDS test is to assure you that you don't have AIDS. False negatives are unacceptable, false positives can be dealt with.
The purpose of the IDS is not to prevent intrusions - that would be nice, but it's not going to happen. The purpose of the IDS is to identify the (coloquially) hackers, so that you can retaliate against them / deter them, before they get you too many times, or get too many different people once. To do this, you need a deterence-level set of positives which is small enough (and true positive rich enough) for you to actually act on them.
Oh, and because I get off on it when people with agree with me: this is no substitute for real, human-level, security measures. Someone who expects any system of this kind to protect them from lousy sysadmin decisions deserves the rusty metal sodomy they will no doubt recieve.
The good and new comes from no quarter where it is looked for, and is always something different from what is expected.
Too many false alarms isn't necessarily a bad thing. In intrusion detection you'd rather take the false positives, than the alternative.
Spoken like someone who does not carry the IDS support pager at nights and on week-ends!
The problem with too many false IDS alarms is that the staff tend to treat it like the boy who cried wolf. After awhile, you disregard the pages or treat them with less consideration because the last n pages have all been false alarms.
I think that IDS is important, but if there are too many false IDS alerts, it becomes difficult to put up with. Because they are strictly reactive systems, it is improbable that there will ever be a perfect IDS that never raises false alarms, but clearly there is a lot of work to do. I am surprised that Snort did so poorly, since it really is a nice system, but it takes a long time to build up a good set of heuristics...
The rate of false fire alarms, and false burgular alarms is VERY high compared to the actual number of real emergencies.
That's right. And in my area, if the police department are called out to the same location for three false burglar alarms in one year, they will not respond to any subsequent alarms automatically. And the fire department charges a fine of $300.00 per incident if they receive more than three false fire alarm calls to the same location in one year. Why? Because, as you said, the number of false alarms is much higher than the number of actual emergencies. The false alarms cost time and money and if all the resources are busy dealing with false alarms, there is nobody left to help when a genuine emergency occurs.
*** Where are we going? And what's with this handbasket?
To wit, we've used all of the products they complain endlessly about, and all I can say is RTFM. All of the problems they encounter are either configuration problems or worse, PEBKAC.
If you want to really learn about IDS, and you don't have the budget to buy a commercial IDS, download a copy of snort and learn for yourself. This report strikes as the type of complaing you get from an IT customer that wants to buy a product, turn it on, never configure it and expect it to magically work.
Wow! What a revelation! You mean you have to know what you're doing and it actually takes time to configure these powerful tools?! In a word, DUH. IDS'es must be tuned. IT products must be configured properly. These things take time, sometimes a lot of time. The core of their complaints revolve around their inability to do either of these things well. Given that lots of people manage to do this effectively everyday and have been for years and years, we're left to conclude that these reporters were not up to the task. And here it is:
These folks actually expected NIDS to be plug-and-play, and thats what they seem upset about. NIDS are powerful sniffers, they need to be tuned, they need to be configured and yes, this IS an ongoing process - but they are not plug-and-play devices.Futhermore, all of IT is an ongoing process. A big, circular, ongoing process that requires competent personnel to manage, maintain, tune, test, patch, configure , deploy and yes, spend TIME on. Anyone that expects to be able to deploy close to a dozen different IDS products as plug and play devices into a production network in 90 days with questionable expertise is fooling themselves.
And then they say as much. Again, this report is total waste of time. Its overly sensationalized and stems from a lack of expertise on the products in question. Skip it, download snort or buy one of the commerical products, take a class, read a book and learn for yourself. You won't learn much from this report that common sense wouldn't have told you already.Python
Was this test run outside of a firewall/filter? Because it would seem like the IDS is best used behind a full strength firewall in the first place; that is, most of the attacks should never reach the IDS anyway. We use Snort to monitor our networks, and under proper configuration the number of alerts is manageable / predictable under normal circumstances.
The author of the article complained that the majority of the systems reported results by IP and not domain name.
This statement alone gives me reason to doubt their abilities. Combine this large amounts of traffic they have with a reverse-DNS lookup for each and you would have crippled your DNS servers.
This is configurable in Snort, and mentioned in detail in the Snort docs.
A good solution is to either dedicate a DNS server to the IDS box, or use a script/utility to do reverse-lookups on the items you are interested in. Not live, but when a human is looking at things.
They are right about one regard -- IDS configuration, monitoring and maintenance isn't for non-professionals. You *need* to know what you are doing.
Learning HOW to think is more important than learning WHAT to think.
This article would have benefited greatly from a diagram or two. IDS behind the firewall? IDS in front of the firewall? Possibly they mentioned it, but I failed to find reference in the article. Myself, I prefer to keep the IDS behind the firewall as I only care about packets that get through. How do /.'ers deploy?
This is so true, I wish the PHBs would get this! Detection and prevention are two different things, it is like comparing a pregnancy test to a condom.
You should have at least used the later, but to be sure, use the former as well.
Exaclty. For a good example check out this past IDS round up:
l
http://www.networkcomputing.com/1217/1217f2.htm
These guys obviously know how IDSs work, and beautfully documented their testing methodology.