Slashdot Mirror
Network Intrusion Detection Systems Fail to Impress
TheBongPipe writes "I'm reading a nice test here about 7 commercial IDSs. Who won the prize? Nobody..." They also looked at Snort, but found that all the products generated way too many false alarms.
I had a nice experience using snort.
:/ Point and click is not always the best solution...
Come on, reading the article I saw the guy said a Snort disadvantage is not having a GUI. What kind of technical user this guy is?
Fabio - Sumare/Sao Paulo/Brazil/South America/Earth/Solar System/Milky Way/Universe
http://www.morroida.com.br
Like a pregnancy test, I think the false positives are preferable to sitting around thinking you're safe.
Liora
Compare with my program that suddenly displays "!!! RED ALERT !!!" at random.
It'd be nice to have some more detail on their results. The chart on the page shows Snort detected all the attacks listed in the chart except the SYN flood. And the footnote on that entry says Snort was down because of "configuration error."
Gee, whose fault is that?
They also go on to mention all ask too much of their users in terms of time and expertise to be described as security must-haves. IDSs are not screen-savers. Those who are setting up an IDS better have a good understanding of how they work and how to configure these applications. Point-and-click doesn't really apply to something this involved.
Like Car Alarms, if it goes off all the time, people will just ignore it -- At some point, the noise drowns out the signal.
You would hope that the increase in false positives decreases the number of false negatives but that isn't necessarily true either.
I am not a number! I am a man! And don't you
Yeah, me too. All that special lab equipment to refine it, and the look out always saying the cops are coming when half the time it's just a meter-maid....
It amazes me that people will pay $20,000 for a product that regularly crashes, doesn't detect all intrusions, and can only be kept up by constant, expensive intervention from the vendor, when for $20,000 less you can have a similar product that doesn't crash, detects just as many intrusions (though not all of them) and can be maintained either by the vendor, or by anyone else with the wit to understand it.
IDS are complex systems. Anyone pretending they have a packaged solution should rot in jail.
--
E_NOSIG
I recall a user we had on our network who thought it'd be cute to install BlackIce on his box, to better secure it. Nevermind the fact that I, and the rest of the admins at my company, had firewalls in place and had never had an intrusion on our network.
Imagine the fun the first time we try to deploy an antivirus package to his desktop just to be blocked for -- are you sitting down? -- an attempted NetBIOS intrusion.
After the second time we tried to deploy (and failed) BlackIce locked down the system so that it couldn't be accessed across the network by any other workstation, despite our having adminsitrative rights. That was cute.
Just throwing up a little real world example of how annoying these false alarms can be.
This article came from the point of view of a normal administrator trying to also manage security. It is mostly based on the assumption that you use the default ruleset (there's no mention of what ruleset is put to use).
Nowadays you really have to be selective about what ruleset you use, logging too much isn't a good thing. This is part of the reason you need a qualified Intrusion analyst who have the expertise to determine which ruleset is useful and which isn't.
The worst thing that can happen (which does happen quite often) is after paying for the expensive distributed sensor IDS system, the logs are never processed or read by anyone.
As stated by the article, an IDS is suppose to log anomalies, that is any abnormal behaviour. But anomalies is only useful if you have a technical guy capable of analysing the traffic. In fact, I would rather have a faulty IDS system that misses packets than to have a good IDS system and all logs go down the drain at the end of the day.
This review wasnt done very well. There was a lot of discussion on the Security Focus Focus-IDS list. Robert graham, main craeted of the BlackICE engine (and the guy who wrote altivore) summed it up nicely in this posting (text below): http://online.securityfocus.com/archive/96/279595. Also, the entire thread can be found at: http://online.securityfocus.com/archive/96/280125/ 2002-07-08/2002-07-14/1
u rity1.html
Actually, most of his posts tend to have interesting (and qualified) views on IDS> sure he is biased (a vendor) but his commentary is usually thought out and not vendor-ish.
> From: Andrew Plato [mailto:aplato@anitian.com]
> In-Reply-To:
> >http://www.nwfusion.com/techinsider/2002/0624sec
> Next time they should do RealSecure on one of my Win2k
> appliances.
No.
While it is true that the reviewer found a bug with the Nokia platform that
doesn't exist on Windows or Solaris, there wasn't anything especially wrong
with the platform.
The issue is that the reviewer was hostile towards IDSs. A customer wants
his product to work, so when they don't, they will keep calling tech support
until it does. Reviewers want the products not to work, so they will
construct the nature of the test in order to make sure this happens. The
reviewer, in this case, never called ISS; the first we heard about him was
at the end of this review, not at the first crash of the Nokia box.
RealSecure has a unique feature called "audit" events. These are supposed to
trigger on normal traffic, such as every HTTP GET request. These are useful
either to create audit trails, or as "anomaly detection": turn on all
audits, then turn off those that trigger normally on your network.
This reviewer turned on audit events, which flooded the console. The setup
that Nokia provided them (256-megs of RAM and a database limited to
2-gigabytes) is perfectly reasonable for the network they had, but not if
all audits were turned on. (The Nokia bug we fixed was related to the fact
that it didn't have enough memory to handle the event load). The reviewer
complained about an overload of false-positives and the box crashing, but
this was because the reviewer drove the product to the point where this
happened.
In truth, it isn't always obvious which of our events are "Audits" and which
ones are "Attacks"; this is an issue fixed in 7.0 of our product. I doubt
this would have made a difference in the review: 7.0 has a lot more audits,
allowing reviewers to overload the product even more if they desire.
Imagine a review of automobiles, where a reviewer grabs a Ford Explorer and
starts complaining that it still crashes, even with the Firestone tires
fixed. One might ask if the there is a problem with the Ford, but one might
also ask if the reviewer intentionally drove the car until it crashed. Next
time you are driving down the freeway, violently jerk the steering wheel all
the way to the right. If you survive, you'll understand what I mean.
I'm not saying the review is wrong. As the reviewer said, he learned a lot
about IDS during the process of reviewing these products. If you, too, don't
know much about IDS but are planning to install one, you will likely get the
same experience: being overwhelmed with alerts that are "false-positives",
and a general sense that the product isn't working. The first few months of
running the IDS are likely to be particularly frustrating. I suggest (a)
working with a consultant to tune the system, (b) working with the vendor's
support in order to get suggestions from them, (c) learning more about the
system. You are going to do (c) anyway: after a few months, you are going to
have learned a heck of a lot more about hacking and defense then you ever
dreamed possible. Read the review: take it with a grain of salt knowing the
reviewer wanted all the products to fail, but realize that this likely to be
your experience the first few months after installing the product, you are
likely to be overwhelmed with events and unlikely to be impressed during the
first few months of ownership.
Robert Graham
Chief Architect
Internet Security Systems
Just read the article. A bit poorly written. What were the IDS run on? Why no analysis of Snort? I'll say that I find Snort way over my head, but that's because I haven't RTFM enough. Why would one want a GUI on a server? (one of the points they marked it down for). Why did it crash? I've NEVER had a linux box crash. NEVER. I've also very, very rarely had a program freeze up enough to require a kill -9 (other than Netscape Navigator and some other buggy stuff. Not stuff like exim, apache, etc.) As a matter of fact, scroll down, and it seems that the downtime was due to their problem, not Snort (footnote at bottom of uptime table).
There are complaints about false-positives. I've played with Snort and there are ways to decrease the alarms put up. For example, a certain number of bum packets in a certain length of time. Not each and every packet.
Looking at the info at the bottom of the article, the authors should know what they are doing. But given the misrepresentations and inaccuracies releative to Snort, why should I believe their testing of non-Free software was any better?
Maybe it was eWeek or some similar publication about six or nine months ago did a similar check. The article was much longer and more in depth. They were also more appreciative of the programs out there. Now, some will say "just to appease their advertisers". Well... Maybe. But if that is the case, why did Snort get their nod as the best?
Jesus was all right but his disciples were thick and ordinary. -John Lennon
I would also like to remind everyone having pride in their own IDS that NIDS will never catch every single attack. (At least for the next little while)
Signature based detection is only good if the attack utilize abnormal or unique traffic to exploit the vulnerability. It will not pick out attacks that uses normal common traffic (for obvious reasons).
IDS evasion techniques are also heavily worked on, plus all application level evasive techniques (eg. sidestep). We can just never be totally dependent on the NIDS for telling us intrusion has occured. It works for most attacks but will fail for some.
Not having a GUI?!?
/etc/snort/snort.conf
I've been running Snort for some time now, and love it! I'm using MySQL logging with ACID and ADODB under Apache for a front end. You just can't get any easier than fill-in-the-blanks SQL querys and intuitive packet layouts. Obviously, they want a strictly out-of-the-box product, and aren't willing to invest any time to make a solid IDS.
As to the false positives, I can concur that in the beginning it was daunting seeing the flood of alerts, but in time, you figure out what is normal and what is not. A little restructure, or a few rule overrides, or rewritten rules, and it's seamless. All it takes is time. This is akin to bitching that your fresh *nix install doesn't have everything just the way you want it, with all your custom apps and modules. You can easily reduce the number of snort alerts by passing the command option as:
snort -D -o -i eth2 -c
This (the -o) changes the rules order to Pass:Alert:Log killing home network normal activity before alert processing. It helps immensely!
I have used Snort and Qualys (the high priced commercial outsourced IDS) and both give false positives quite frequently. However, proving they are false positives is part of the skill of a good human sysadmin. This is why IDSes will never replace a good sysadmin. He or she should be able to see the report and say without any shadow of doubt in his speech that any particular exploit shown by the IDS is a false postive or not.
This still means that each IDS has its good points; but why anyone would pay a lot for a system that cannot, by definition, be any better than an up to date Snort and human reading of the report, and knowing your network inside out. Those who buy into big commercial IDSes clearly are investing in software when they should be investing in people, training those people, and understanding those people. Too many middle managers think their sysadmin speaks a language they will never learn, and therefore need these things to understand. But a good sysadmin should try hard to find ways to communicate with them, and can if need be annotate a nice little Snort report and be done with it.
Conversion Rate Optimisation French / English consultant
If you look at the table, snort looks like it was doing great, except that it somehow missed the SYN attack. So, based off that chart, none of the IDS corrected detected all of the attacks... however, you read on a bit further, and..
Snort was off the air at the time of the attack because of misconfiguration on our part.
I don't have a lot of confidence in their results.
Too many false alarms isn't necessarily a bad thing. In intrusion detection you'd rather take the false positives, than the alternative.
Spoken like someone who does not carry the IDS support pager at nights and on week-ends!
The problem with too many false IDS alarms is that the staff tend to treat it like the boy who cried wolf. After awhile, you disregard the pages or treat them with less consideration because the last n pages have all been false alarms.
I think that IDS is important, but if there are too many false IDS alerts, it becomes difficult to put up with. Because they are strictly reactive systems, it is improbable that there will ever be a perfect IDS that never raises false alarms, but clearly there is a lot of work to do. I am surprised that Snort did so poorly, since it really is a nice system, but it takes a long time to build up a good set of heuristics...
The rate of false fire alarms, and false burgular alarms is VERY high compared to the actual number of real emergencies.
That's right. And in my area, if the police department are called out to the same location for three false burglar alarms in one year, they will not respond to any subsequent alarms automatically. And the fire department charges a fine of $300.00 per incident if they receive more than three false fire alarm calls to the same location in one year. Why? Because, as you said, the number of false alarms is much higher than the number of actual emergencies. The false alarms cost time and money and if all the resources are busy dealing with false alarms, there is nobody left to help when a genuine emergency occurs.
*** Where are we going? And what's with this handbasket?
Homer: Now, here's my "Everything's O.K."alarm!
[Homer flips a switch on the device, and it begins to emit a high pitched, incredibly loud beep. The rest of the Simpsons cover their ears as Homer speaks up]
Homer: This will sound every three seconds, unless something isn't okay!
Marge: Turn it off, Homer!
Homer: It can't be turned off! [alarm fizzles out] But it, uh, does break easily.
-- "The Wizard of Evergreen Terrace"
This sounds about as useful.
Carthago delenda est!
Funny part is, you can take your pick of UI's for snort, on just about any platform (I run snort on WinNT on one network, and snort on Linux on another. And I've got a GUI for both of 'em ;-)
Davis Ray Sickmon, Jr - looking for something to read? Check out my three free novels at MidnightRyder.org
Trollem mirabilem hanc subnotationis exigiutas non caperet
I recall a user we had on our network who thought it'd be cute to install BlackIce on his box, to better secure it. Nevermind the fact that I, and the rest of the admins at my company, had firewalls in place and had never had an intrusion on our network.
;-)
I hate to tell you this but, at this day and age when everything is being outsourced, some users feel they need to protect their machines against the "IT support".
"I have opinions of my own, strong opinions, but I don't always agree with them." -- George H. W. Bush
IDS systems need to be tuned! Don't have any NT machines on that subnet? Turn off all of the NT related signatures! Get tons of false alarms on a particular alert which isn't applicable? Turn it off! It's a matter of risk assessment. Are you more likely to miss something important because of this alert which goes off all the time and has a low probability of being legitimately triggered? Turn it off! You won't catch everything this way but the goal is to at least catch SOMETHING that you would not have if you didn't have the IDS!
IDSes are NOT meant to work out of the box. Snort's FAQ specifically states that you should disable rules for things you don't need! By default, it includes a lot of stuff. Luckily, the rules are neatly organized into files, so you can comment them out, and stop getting warnings you don't want! Likewise, using Snort without Acid is well... not very common. Yet, there is no mention of Acid in this article. I can only imagine that the rest of this article is flawed due to the reviewer's lack of knowledge.
To wit, we've used all of the products they complain endlessly about, and all I can say is RTFM. All of the problems they encounter are either configuration problems or worse, PEBKAC.
If you want to really learn about IDS, and you don't have the budget to buy a commercial IDS, download a copy of snort and learn for yourself. This report strikes as the type of complaing you get from an IT customer that wants to buy a product, turn it on, never configure it and expect it to magically work.
Wow! What a revelation! You mean you have to know what you're doing and it actually takes time to configure these powerful tools?! In a word, DUH. IDS'es must be tuned. IT products must be configured properly. These things take time, sometimes a lot of time. The core of their complaints revolve around their inability to do either of these things well. Given that lots of people manage to do this effectively everyday and have been for years and years, we're left to conclude that these reporters were not up to the task. And here it is:
These folks actually expected NIDS to be plug-and-play, and thats what they seem upset about. NIDS are powerful sniffers, they need to be tuned, they need to be configured and yes, this IS an ongoing process - but they are not plug-and-play devices.Futhermore, all of IT is an ongoing process. A big, circular, ongoing process that requires competent personnel to manage, maintain, tune, test, patch, configure , deploy and yes, spend TIME on. Anyone that expects to be able to deploy close to a dozen different IDS products as plug and play devices into a production network in 90 days with questionable expertise is fooling themselves.
And then they say as much. Again, this report is total waste of time. Its overly sensationalized and stems from a lack of expertise on the products in question. Skip it, download snort or buy one of the commerical products, take a class, read a book and learn for yourself. You won't learn much from this report that common sense wouldn't have told you already.Python
The author of the article complained that the majority of the systems reported results by IP and not domain name.
This statement alone gives me reason to doubt their abilities. Combine this large amounts of traffic they have with a reverse-DNS lookup for each and you would have crippled your DNS servers.
This is configurable in Snort, and mentioned in detail in the Snort docs.
A good solution is to either dedicate a DNS server to the IDS box, or use a script/utility to do reverse-lookups on the items you are interested in. Not live, but when a human is looking at things.
They are right about one regard -- IDS configuration, monitoring and maintenance isn't for non-professionals. You *need* to know what you are doing.
Learning HOW to think is more important than learning WHAT to think.
It is really too bad the Enterasys product wasn't available. I've implemented that on a _very_ large government department network with dual T-3 pipes and collecting >1Gig of data per day. Yes, still many false alarms to sift through but the uptime was measured in months not days or hours. This same gov't department has had other IDS vendors try to bring in their products to no avail because none of them can stay up >24 hours.
Do really dense people warp space more than others?
I don't trust any "real world" shootout that doesn't show how the IDS were plugged into the network, how they determined an attack, and other such key points. You can't just say "we plugged it in and nothing worked." IDS are much more complicated than that. How and where were they plugged into the network fabric? Were they using switch port mirroring or passive ethernet taps at the uplinks? How do they know these attacks happened without initiating them themselves? That last one is the biggest single problem with "real world" testing. Unless you're launching the attacks yourself you do not know, and unless I missed it, they were relying on attacks to just happen out of the blue.
Now, they do raise some important issues with the backend storage of events and the need for clarity with the false positives and false negatives, but many of these can be dealt with by implementation of a real-time security console that does some form of event correlation from multiple security devices that says "The IDS sees this as a problem, the firewall sees it as a problem, and the target sees it as a problem. It's probably a problem. RED ALERT!" It's a much more intelligent way of dealing with events than just forwarding each one to a pager.
We've always said security is a process which must be maintained and firewalls/IDS systems are not a panacea to network security. As someone who's been responsible for a large scale IDS roll-out at Enron Broadband Services, where we were ISS' single largest customer for RealSecure before everything went to hell, I feel confident saying that Network IDS is a very useful tool, provided you keep it out of the hands of people who have absolutely no clue what they're doing with it, like the three gentlemen who are responsible for this article.
Joseph