Network Intrusion Detection Systems Fail to Impress

TheBongPipe writes "I'm reading a nice test here about 7 commercial IDSs. Who won the prize? Nobody..." They also looked at Snort, but found that all the products generated way too many false alarms.

Comments

A False Alarm is still an Alarm

[pheph]

I worked in a NOC for some time and found that while false alarms generally take away from the impact of real alarms, they still alarm you that something not quite right is going on in your network.

They also go on to mention all ask too much of their users in terms of time and expertise to be described as security must-haves. IDSs are not screen-savers. Those who are setting up an IDS better have a good understanding of how they work and how to configure these applications. Point-and-click doesn't really apply to something this involved.

... As for stability, I think this report is correct, the only IDS I've used that didn't crash consistanty was snort (with ACID)

Car Alarms

[Codex The Sloth]

Like Car Alarms, if it goes off all the time, people will just ignore it -- At some point, the noise drowns out the signal.

You would hope that the increase in false positives decreases the number of false negatives but that isn't necessarily true either.

Re:False Positives...

[sam_handelman]

Well, that's true, unless

ALERT: Intrusion attempt detected! User Liora mistyped his password!

the rate of false-positives is high enough that

ALERT: Account Liora has recieved login attempts from three different IPs in the past 12 hours!

you stop paying attention; unlike with AIDS testing (which has a very high false positive rate) the user is simply likely to ignore the system even if real threats occur

ALERT: 1,262 attempts to login as user "root" in past 7 seconds!

So it becomes desirable to lower the false positive rate to a manageable level, WHATEVER the rate of false negatives is, because otherwise you won't actually catch anything.

The purpose of the AIDS test is to assure you that you don't have AIDS. False negatives are unacceptable, false positives can be dealt with.

The purpose of the IDS is not to prevent intrusions - that would be nice, but it's not going to happen. The purpose of the IDS is to identify the (coloquially) hackers, so that you can retaliate against them / deter them, before they get you too many times, or get too many different people once. To do this, you need a deterence-level set of positives which is small enough (and true positive rich enough) for you to actually act on them.

Oh, and because I get off on it when people with agree with me: this is no substitute for real, human-level, security measures. Someone who expects any system of this kind to protect them from lousy sysadmin decisions deserves the rusty metal sodomy they will no doubt recieve.

The problem with false alarms

[why-is-it]

Too many false alarms isn't necessarily a bad thing. In intrusion detection you'd rather take the false positives, than the alternative.

Spoken like someone who does not carry the IDS support pager at nights and on week-ends!

The problem with too many false IDS alarms is that the staff tend to treat it like the boy who cried wolf. After awhile, you disregard the pages or treat them with less consideration because the last n pages have all been false alarms.

I think that IDS is important, but if there are too many false IDS alerts, it becomes difficult to put up with. Because they are strictly reactive systems, it is improbable that there will ever be a perfect IDS that never raises false alarms, but clearly there is a lot of work to do. I am surprised that Snort did so poorly, since it really is a nice system, but it takes a long time to build up a good set of heuristics...

The rate of false fire alarms, and false burgular alarms is VERY high compared to the actual number of real emergencies.

That's right. And in my area, if the police department are called out to the same location for three false burglar alarms in one year, they will not respond to any subsequent alarms automatically. And the fire department charges a fine of $300.00 per incident if they receive more than three false fire alarm calls to the same location in one year. Why? Because, as you said, the number of false alarms is much higher than the number of actual emergencies. The false alarms cost time and money and if all the resources are busy dealing with false alarms, there is nobody left to help when a genuine emergency occurs.