Slashdot Mirror
Network Intrusion Detection Systems Fail to Impress
TheBongPipe writes "I'm reading a nice test here about 7 commercial IDSs. Who won the prize? Nobody..." They also looked at Snort, but found that all the products generated way too many false alarms.
"As for stability, I think this report is correct, the only IDS I've used that didn't crash consistanty was snort (with ACID)"
I've run NAI's IDS (the one that came bundled with PGP 7.1) for a year on Win2k, and it hasn't crashed yet. It does come up with false positives, especially if you configure it to be "sensitive", but once they occur, you can determine whether or not you want to continue to listen for them. It consistently tagged something the Mac's on our LAN were doing as a "fraggle" attack, so I turned off "fraggle" detection.
Not a perfect solution, but soooo much better than nothing.
This article came from the point of view of a normal administrator trying to also manage security. It is mostly based on the assumption that you use the default ruleset (there's no mention of what ruleset is put to use).
Nowadays you really have to be selective about what ruleset you use, logging too much isn't a good thing. This is part of the reason you need a qualified Intrusion analyst who have the expertise to determine which ruleset is useful and which isn't.
The worst thing that can happen (which does happen quite often) is after paying for the expensive distributed sensor IDS system, the logs are never processed or read by anyone.
As stated by the article, an IDS is suppose to log anomalies, that is any abnormal behaviour. But anomalies is only useful if you have a technical guy capable of analysing the traffic. In fact, I would rather have a faulty IDS system that misses packets than to have a good IDS system and all logs go down the drain at the end of the day.
But Opus One's servers run OpenVMS, not Windows. Even though it is trivially easy to figure out what operating system a Web server uses, not one of the IDSs did so. Instead, they collectively generated literally millions of alarms about attacks that never happened.
That's an unrealistic expectation to place on an IDS, from the start. You get an IDS to log attack attempts first, not the attacks themselves - if the attacks are known (have signatures) your machines should be protected against them in the first place.
Hands in my pocket
This review wasnt done very well. There was a lot of discussion on the Security Focus Focus-IDS list. Robert graham, main craeted of the BlackICE engine (and the guy who wrote altivore) summed it up nicely in this posting (text below): http://online.securityfocus.com/archive/96/279595. Also, the entire thread can be found at: http://online.securityfocus.com/archive/96/280125/ 2002-07-08/2002-07-14/1
u rity1.html
Actually, most of his posts tend to have interesting (and qualified) views on IDS> sure he is biased (a vendor) but his commentary is usually thought out and not vendor-ish.
> From: Andrew Plato [mailto:aplato@anitian.com]
> In-Reply-To:
> >http://www.nwfusion.com/techinsider/2002/0624sec
> Next time they should do RealSecure on one of my Win2k
> appliances.
No.
While it is true that the reviewer found a bug with the Nokia platform that
doesn't exist on Windows or Solaris, there wasn't anything especially wrong
with the platform.
The issue is that the reviewer was hostile towards IDSs. A customer wants
his product to work, so when they don't, they will keep calling tech support
until it does. Reviewers want the products not to work, so they will
construct the nature of the test in order to make sure this happens. The
reviewer, in this case, never called ISS; the first we heard about him was
at the end of this review, not at the first crash of the Nokia box.
RealSecure has a unique feature called "audit" events. These are supposed to
trigger on normal traffic, such as every HTTP GET request. These are useful
either to create audit trails, or as "anomaly detection": turn on all
audits, then turn off those that trigger normally on your network.
This reviewer turned on audit events, which flooded the console. The setup
that Nokia provided them (256-megs of RAM and a database limited to
2-gigabytes) is perfectly reasonable for the network they had, but not if
all audits were turned on. (The Nokia bug we fixed was related to the fact
that it didn't have enough memory to handle the event load). The reviewer
complained about an overload of false-positives and the box crashing, but
this was because the reviewer drove the product to the point where this
happened.
In truth, it isn't always obvious which of our events are "Audits" and which
ones are "Attacks"; this is an issue fixed in 7.0 of our product. I doubt
this would have made a difference in the review: 7.0 has a lot more audits,
allowing reviewers to overload the product even more if they desire.
Imagine a review of automobiles, where a reviewer grabs a Ford Explorer and
starts complaining that it still crashes, even with the Firestone tires
fixed. One might ask if the there is a problem with the Ford, but one might
also ask if the reviewer intentionally drove the car until it crashed. Next
time you are driving down the freeway, violently jerk the steering wheel all
the way to the right. If you survive, you'll understand what I mean.
I'm not saying the review is wrong. As the reviewer said, he learned a lot
about IDS during the process of reviewing these products. If you, too, don't
know much about IDS but are planning to install one, you will likely get the
same experience: being overwhelmed with alerts that are "false-positives",
and a general sense that the product isn't working. The first few months of
running the IDS are likely to be particularly frustrating. I suggest (a)
working with a consultant to tune the system, (b) working with the vendor's
support in order to get suggestions from them, (c) learning more about the
system. You are going to do (c) anyway: after a few months, you are going to
have learned a heck of a lot more about hacking and defense then you ever
dreamed possible. Read the review: take it with a grain of salt knowing the
reviewer wanted all the products to fail, but realize that this likely to be
your experience the first few months after installing the product, you are
likely to be overwhelmed with events and unlikely to be impressed during the
first few months of ownership.
Robert Graham
Chief Architect
Internet Security Systems
Yes, the box was working.. it is an entirely different type of IDS. apples to oranges type of thing.. IMHO, anyoner who tried to deploy it to replace snort/iss/dragon is an idiot. It does work well to COMPLIMENT a tradional IDS, but not to replace.
It functions by analysis of traffic flow.. system x communicating to system y on ports a,b,c. if the pattern changes beyond a threshold, an alarm.
I can vouch for this also. About six months ago I setup a snort logging to mysql box at work to monitor our class-C and have had zero problems. It does take a while to tweak and prune things to eliminate all the portscan misdetects (any/any is certainly not advised) and such, but overall it's performed sweet. The price is right certainly, comparing it to those listed in the article. As far as console analysis, you simply cannot do without ACID as the parent said. Head to CERT and download a copy. My only complaint is that the db lookups perform a little sluggishly on the p2-233.
Not having a GUI?!?
/etc/snort/snort.conf
I've been running Snort for some time now, and love it! I'm using MySQL logging with ACID and ADODB under Apache for a front end. You just can't get any easier than fill-in-the-blanks SQL querys and intuitive packet layouts. Obviously, they want a strictly out-of-the-box product, and aren't willing to invest any time to make a solid IDS.
As to the false positives, I can concur that in the beginning it was daunting seeing the flood of alerts, but in time, you figure out what is normal and what is not. A little restructure, or a few rule overrides, or rewritten rules, and it's seamless. All it takes is time. This is akin to bitching that your fresh *nix install doesn't have everything just the way you want it, with all your custom apps and modules. You can easily reduce the number of snort alerts by passing the command option as:
snort -D -o -i eth2 -c
This (the -o) changes the rules order to Pass:Alert:Log killing home network normal activity before alert processing. It helps immensely!
RE:
had firewalls in place and had never had an intrusion on our network.
HAHAHAHAHA. seriously. HAHAHA. This is called the hard-candy security princple.. Hard and crunchy on the outside, soft and chewy inside. Perimiter firewalls are insiufficient for medium-large businesses, and generally small ones as well. Got VPN connections? Allow ANY traffic in to a DMZ or internal network?
As for BlackICE.. 'NetBIOS intrusion' isnt even an attack type. What happened was probably that the user blocked the netbios PORT via the firewalling and it showed as a 'NetBIOS port probe' with the reason=firewalled. Also, BlackICE does not disable ALL network activity. By IP address or port, yes. Not by network or everything. That would be stupid. See http://www.networkice.com/advice for BlackICE attack details.
Trollem mirabilem hanc subnotationis exigiutas non caperet
IDS's are just plain silly. Do the following:
1. Turn off unused services, and disable suid software. If this is not possible, then don't use the application.
2. Take steps to minimize compromise of at-risk services. Use chroot, aplication specific uid/gid.
3. Leran how to apply packet filters to ingress/egress points.
4. Apply patches for applications and O/S
5. Stop going to expensive, lame 'security' seminars.
fini!
IDS will someday approach NMS systems in uselessness
IDS systems need to be tuned! Don't have any NT machines on that subnet? Turn off all of the NT related signatures! Get tons of false alarms on a particular alert which isn't applicable? Turn it off! It's a matter of risk assessment. Are you more likely to miss something important because of this alert which goes off all the time and has a low probability of being legitimately triggered? Turn it off! You won't catch everything this way but the goal is to at least catch SOMETHING that you would not have if you didn't have the IDS!
IDSes are NOT meant to work out of the box. Snort's FAQ specifically states that you should disable rules for things you don't need! By default, it includes a lot of stuff. Luckily, the rules are neatly organized into files, so you can comment them out, and stop getting warnings you don't want! Likewise, using Snort without Acid is well... not very common. Yet, there is no mention of Acid in this article. I can only imagine that the rest of this article is flawed due to the reviewer's lack of knowledge.
I am the Director of Managed Security for a company on Hawaii, and we rovide managed Security Services to various companies around the state based on Snort. Snort truly is a very good IDS, and if configured properly it will generate few if any false positive alerts. Most of the reason that people say bad things about a product is due to their own lack of experience in setting it up.
Wherever you go, there I am...
If you're a Network or System Administrator, you should KNOW you're not safe.
You SHOULD be testing your systems constantly.
You SHOULD be installing new patches.
You SHOULD be subscribed to CERT style mailings.
You SHOULD NOT think you are safe because you're small. Security though obscurity is the biggest false sense if security I've seen. Former employees, especially the guy you replaced are a pretty large threat.
For beginners out there, here are some places to start... (Some of these are OLD links, but still contain some useful information and yes, they're Linux oriented.):
Beginners Guide to Armoring Linux
Linux Security Guide
Nessus
Traditional HOWTO
I don't do this for karma, I do it for cash. It's much better.
It is really too bad the Enterasys product wasn't available. I've implemented that on a _very_ large government department network with dual T-3 pipes and collecting >1Gig of data per day. Yes, still many false alarms to sift through but the uptime was measured in months not days or hours. This same gov't department has had other IDS vendors try to bring in their products to no avail because none of them can stay up >24 hours.
Do really dense people warp space more than others?
The company that makes the opensource security tools portsentry, logcheck, and hostsentry released something a few weeks back that they claim helps with this problem. The tool is called ClearResponse and it (supposedly) will investigate each attack seen by your IDS to see if the alarm is real or false and will then downgrade or upgrade the alarm and report to the admin. It looks pretty cool and we're getting an eval to check out. Here is their website:
ClearResponse
Things I like about it:
installation script is truly a marvel (installs snort, mysql, apache, perl modules)
Login screen/authentication
Big Brother like monitoring
File integrity checking
IDS using Snort sensors
free to use for non-commercial use
no I don't work for them I just like the software.
Fact is any IDS, now or in the forseable future, assuming we do not crack that artifical intelligence thing, is going to require a qualified and skilled security professional to monitor and tweek any IDS.
You are not gonna find security out of a box. There is no software out there that alone will outsmart even the most lame brained...you gotta have people there to protect your system or sooner or later you will be owned.
I don't trust any "real world" shootout that doesn't show how the IDS were plugged into the network, how they determined an attack, and other such key points. You can't just say "we plugged it in and nothing worked." IDS are much more complicated than that. How and where were they plugged into the network fabric? Were they using switch port mirroring or passive ethernet taps at the uplinks? How do they know these attacks happened without initiating them themselves? That last one is the biggest single problem with "real world" testing. Unless you're launching the attacks yourself you do not know, and unless I missed it, they were relying on attacks to just happen out of the blue.
Now, they do raise some important issues with the backend storage of events and the need for clarity with the false positives and false negatives, but many of these can be dealt with by implementation of a real-time security console that does some form of event correlation from multiple security devices that says "The IDS sees this as a problem, the firewall sees it as a problem, and the target sees it as a problem. It's probably a problem. RED ALERT!" It's a much more intelligent way of dealing with events than just forwarding each one to a pager.
We've always said security is a process which must be maintained and firewalls/IDS systems are not a panacea to network security. As someone who's been responsible for a large scale IDS roll-out at Enron Broadband Services, where we were ISS' single largest customer for RealSecure before everything went to hell, I feel confident saying that Network IDS is a very useful tool, provided you keep it out of the hands of people who have absolutely no clue what they're doing with it, like the three gentlemen who are responsible for this article.
JosephThere is a misconception among IDS noobs that alerts are like alarms: alerts generate alarms. An IDS generating 1000 alerts should not mean that an admin would receive 1000 alarms/pages.
For example as an IDS admin I want to see alerts for failed telnet attempts internally. If it's 5 within one minute directed at one host, it's not a problem, probably just human error. If it's 1000 per minute, then I want to see an alarm, and get paged. The "reviewers" of the IDS products would have understood this, and taken it into consideration had they ever deployed an IDS in a production environment.
Alerts are not a bad thing even if they are false positives. False positive alarms are a bad thing, especially when they wake you up at 3AM. Again it comes back to tuning.
I can however verify what they experienced with ISS. At my last company we had ISS come out and install, configure and tune Real Secure. I can tell you from first hand experience, that the ISS products suck. I ended up installing Snort in order to keep the ISS products honest. My experience was that ISS had a nasty habit of dropping packets. Snort had no problem keeping up with our frational DS3 (30 Mb).
More recently I also had the priveledge of seeing Real Secure crash repeatedly on a Nokia 530 (installed and configured by ISS engineers) during an IDS pilot. We kicked ISS to the curb and are now in the process of installing Snort with support for ACID, and MySQL.
IMO the reviewers were idiots. They obviously didn't spend much time with any of the products. Which is unfortunate, because some of the good products got lumped in with some of the bad ones, which were all failed together because some reviewers obviously didn't RTFMs.
FYI there is also a really pretty GUI for Snort:
www.demarc.org