Slashdot Mirror
Network Intrusion Detection Systems Fail to Impress
TheBongPipe writes "I'm reading a nice test here about 7 commercial IDSs. Who won the prize? Nobody..." They also looked at Snort, but found that all the products generated way too many false alarms.
Compare with my program that suddenly displays "!!! RED ALERT !!!" at random.
They also go on to mention all ask too much of their users in terms of time and expertise to be described as security must-haves. IDSs are not screen-savers. Those who are setting up an IDS better have a good understanding of how they work and how to configure these applications. Point-and-click doesn't really apply to something this involved.
Like Car Alarms, if it goes off all the time, people will just ignore it -- At some point, the noise drowns out the signal.
You would hope that the increase in false positives decreases the number of false negatives but that isn't necessarily true either.
I am not a number! I am a man! And don't you
Yeah, me too. All that special lab equipment to refine it, and the look out always saying the cops are coming when half the time it's just a meter-maid....
This review wasnt done very well. There was a lot of discussion on the Security Focus Focus-IDS list. Robert graham, main craeted of the BlackICE engine (and the guy who wrote altivore) summed it up nicely in this posting (text below): http://online.securityfocus.com/archive/96/279595. Also, the entire thread can be found at: http://online.securityfocus.com/archive/96/280125/ 2002-07-08/2002-07-14/1
u rity1.html
Actually, most of his posts tend to have interesting (and qualified) views on IDS> sure he is biased (a vendor) but his commentary is usually thought out and not vendor-ish.
> From: Andrew Plato [mailto:aplato@anitian.com]
> In-Reply-To:
> >http://www.nwfusion.com/techinsider/2002/0624sec
> Next time they should do RealSecure on one of my Win2k
> appliances.
No.
While it is true that the reviewer found a bug with the Nokia platform that
doesn't exist on Windows or Solaris, there wasn't anything especially wrong
with the platform.
The issue is that the reviewer was hostile towards IDSs. A customer wants
his product to work, so when they don't, they will keep calling tech support
until it does. Reviewers want the products not to work, so they will
construct the nature of the test in order to make sure this happens. The
reviewer, in this case, never called ISS; the first we heard about him was
at the end of this review, not at the first crash of the Nokia box.
RealSecure has a unique feature called "audit" events. These are supposed to
trigger on normal traffic, such as every HTTP GET request. These are useful
either to create audit trails, or as "anomaly detection": turn on all
audits, then turn off those that trigger normally on your network.
This reviewer turned on audit events, which flooded the console. The setup
that Nokia provided them (256-megs of RAM and a database limited to
2-gigabytes) is perfectly reasonable for the network they had, but not if
all audits were turned on. (The Nokia bug we fixed was related to the fact
that it didn't have enough memory to handle the event load). The reviewer
complained about an overload of false-positives and the box crashing, but
this was because the reviewer drove the product to the point where this
happened.
In truth, it isn't always obvious which of our events are "Audits" and which
ones are "Attacks"; this is an issue fixed in 7.0 of our product. I doubt
this would have made a difference in the review: 7.0 has a lot more audits,
allowing reviewers to overload the product even more if they desire.
Imagine a review of automobiles, where a reviewer grabs a Ford Explorer and
starts complaining that it still crashes, even with the Firestone tires
fixed. One might ask if the there is a problem with the Ford, but one might
also ask if the reviewer intentionally drove the car until it crashed. Next
time you are driving down the freeway, violently jerk the steering wheel all
the way to the right. If you survive, you'll understand what I mean.
I'm not saying the review is wrong. As the reviewer said, he learned a lot
about IDS during the process of reviewing these products. If you, too, don't
know much about IDS but are planning to install one, you will likely get the
same experience: being overwhelmed with alerts that are "false-positives",
and a general sense that the product isn't working. The first few months of
running the IDS are likely to be particularly frustrating. I suggest (a)
working with a consultant to tune the system, (b) working with the vendor's
support in order to get suggestions from them, (c) learning more about the
system. You are going to do (c) anyway: after a few months, you are going to
have learned a heck of a lot more about hacking and defense then you ever
dreamed possible. Read the review: take it with a grain of salt knowing the
reviewer wanted all the products to fail, but realize that this likely to be
your experience the first few months after installing the product, you are
likely to be overwhelmed with events and unlikely to be impressed during the
first few months of ownership.
Robert Graham
Chief Architect
Internet Security Systems
Not having a GUI?!?
/etc/snort/snort.conf
I've been running Snort for some time now, and love it! I'm using MySQL logging with ACID and ADODB under Apache for a front end. You just can't get any easier than fill-in-the-blanks SQL querys and intuitive packet layouts. Obviously, they want a strictly out-of-the-box product, and aren't willing to invest any time to make a solid IDS.
As to the false positives, I can concur that in the beginning it was daunting seeing the flood of alerts, but in time, you figure out what is normal and what is not. A little restructure, or a few rule overrides, or rewritten rules, and it's seamless. All it takes is time. This is akin to bitching that your fresh *nix install doesn't have everything just the way you want it, with all your custom apps and modules. You can easily reduce the number of snort alerts by passing the command option as:
snort -D -o -i eth2 -c
This (the -o) changes the rules order to Pass:Alert:Log killing home network normal activity before alert processing. It helps immensely!
Well, that's true, unless
ALERT: Intrusion attempt detected! User Liora mistyped his password!
the rate of false-positives is high enough that
ALERT: Account Liora has recieved login attempts from three different IPs in the past 12 hours!
you stop paying attention; unlike with AIDS testing (which has a very high false positive rate) the user is simply likely to ignore the system even if real threats occur
ALERT: 1,262 attempts to login as user "root" in past 7 seconds!
So it becomes desirable to lower the false positive rate to a manageable level, WHATEVER the rate of false negatives is, because otherwise you won't actually catch anything.
The purpose of the AIDS test is to assure you that you don't have AIDS. False negatives are unacceptable, false positives can be dealt with.
The purpose of the IDS is not to prevent intrusions - that would be nice, but it's not going to happen. The purpose of the IDS is to identify the (coloquially) hackers, so that you can retaliate against them / deter them, before they get you too many times, or get too many different people once. To do this, you need a deterence-level set of positives which is small enough (and true positive rich enough) for you to actually act on them.
Oh, and because I get off on it when people with agree with me: this is no substitute for real, human-level, security measures. Someone who expects any system of this kind to protect them from lousy sysadmin decisions deserves the rusty metal sodomy they will no doubt recieve.
The good and new comes from no quarter where it is looked for, and is always something different from what is expected.
Too many false alarms isn't necessarily a bad thing. In intrusion detection you'd rather take the false positives, than the alternative.
Spoken like someone who does not carry the IDS support pager at nights and on week-ends!
The problem with too many false IDS alarms is that the staff tend to treat it like the boy who cried wolf. After awhile, you disregard the pages or treat them with less consideration because the last n pages have all been false alarms.
I think that IDS is important, but if there are too many false IDS alerts, it becomes difficult to put up with. Because they are strictly reactive systems, it is improbable that there will ever be a perfect IDS that never raises false alarms, but clearly there is a lot of work to do. I am surprised that Snort did so poorly, since it really is a nice system, but it takes a long time to build up a good set of heuristics...
The rate of false fire alarms, and false burgular alarms is VERY high compared to the actual number of real emergencies.
That's right. And in my area, if the police department are called out to the same location for three false burglar alarms in one year, they will not respond to any subsequent alarms automatically. And the fire department charges a fine of $300.00 per incident if they receive more than three false fire alarm calls to the same location in one year. Why? Because, as you said, the number of false alarms is much higher than the number of actual emergencies. The false alarms cost time and money and if all the resources are busy dealing with false alarms, there is nobody left to help when a genuine emergency occurs.
*** Where are we going? And what's with this handbasket?
Trollem mirabilem hanc subnotationis exigiutas non caperet
I recall a user we had on our network who thought it'd be cute to install BlackIce on his box, to better secure it. Nevermind the fact that I, and the rest of the admins at my company, had firewalls in place and had never had an intrusion on our network.
;-)
I hate to tell you this but, at this day and age when everything is being outsourced, some users feel they need to protect their machines against the "IT support".
"I have opinions of my own, strong opinions, but I don't always agree with them." -- George H. W. Bush