Free Open-Source vs. Commercial Security Tools?

sahirh asks: "I work as a penetration tester and recently started writing a whitepaper on the benefits of free, open-source security tools over commercial tools. Through my own experiences, I've found that many free tools such as Nessus and Kismet are more reliable and have better features than expensive commercial alternatives like ISS Internet Scanner or Airopeek. I've also noticed that tools like Ettercap have no commercial alternative. Further, the flexibility offered by the open-source nature of such tools is a great benefit. I'd like to ask for Slashdot's experiences and opinions on why you don't need to spend thousands of dollars on an expensive tool to perform a professional security assessment." Update: 02/07 11:15pm EDT by C : Thanks to all who wrote in to let us know the proper URL to the Kismet site.

Comments

I want his job

[YankeeInExile]

I have no joke here, I just like saying, I work as a penetration tester ...

Re:I want his job

[Aliencow]

As long as you're not a "backdoor AnalYzer" ..

Snort

[ikewillis]

One of the best NIS tools available, the only thing you can get better are... commercial Snort derivatives. Not mentioned, WTF?

Re:Snort

[SquadBoy]

Sourcefire. Martin Roesch's company. It gives you, the admin, the goodness of Snort and OSS tools and gives your bosses a contract to feel all warm and fuzzy about. Pretty much a win-win. I love my Sourcefire boxen and they cost less than the other commercial IDSes.

Re:Snort

[checkitout]

It gives you, the admin, the goodness of Snort and OSS tools and gives your bosses a contract to feel all warm and fuzzy about.

Actually we found that Sentarus is a much better snort-based product. We kicked Sourcefire out after 2 weeks, they just don't get the concept of a GUI. Talk about butt ugly and unmanagable.

Re:Snort

[gclef]

Snort's not really a pen-test tool, though.

For pen-testing, check out the Metasploit framework. It's truly cool.

Also, have a look for scanrand, part of paketto keiretsu (doxpara.com appears to be having trouble right now, so don't go looking right now).

There's always the old standbys, as well, like dsniff.

Valuable Open Source Security Assement Tools?

[kiwidefunkt]

Ethereal, nmap, and snort always get the job done for me.

Re:Valuable Open Source Security Assement Tools?

[Mr. Sketch]

Agreed. I usually throw in tripwire too from the start, it makes things easier later on.

Re:Valuable Open Source Security Assement Tools?

[Gyorg_Lavode]

How do you use Snort and Tripwire (from the child's response) for penetration testing and risk assessment? I understand using them as part of an IDS, but not for the initial risk assessment.

Re:Valuable Open Source Security Assement Tools?

[niekko]

Same here. And about the open vs. commercial, I've been using both Ethereal and Network General's Sniffer and in my opinion Ethereal is way much better starting from the simple GUI.

Re:Valuable Open Source Security Assement Tools?

[Homology]

Ethereal, nmap, and snort always get the job done for me.

Heh, recommending a security tool that OpenBSD removed because the Ethereal team does not care about security

Mark it as BROKEN:

Right during 3.5, it had more than
a dozen remote holes being fixed, that we shipped with. Weeks later
things have not improved, and there continue to be problems reported
to bugtraq, and respective band-aids - but it is clear the ethereal
team does not care about security, as new protocols get added, and
nothing gets done about the many more holes that exist.

Just because something is open source does not imply that it's secure.

Re:Valuable Open Source Security Assement Tools?

[Stephen Samuel]

Heh, recommending a security tool that OpenBSD removed because the Ethereal team does not care about security

I was just thinking about structural ways to work around this in ethereal (like priv sep) -- in the meantime, I would point out that the biggest difference between ethereal and it's commercial equivalents is is that, with ethereal, you find out about the security problems quickly -- whereas with commercial equivalents, you might not find out for a while (if ever), and you'll probably end up paying for the upgrade to make it secure.

Another point is that it's most often the newer disectors that contain the holes. If you're worried about security and working in a 'hostile' environment, you're probably best to disable any disector that you're not intending to use. -- in fact, that might be a good idea to do in Ethereal, generally: Disable all but the most common dissectors and wait for the user to enable them explicitly.

Re:Valuable Open Source Security Assement Tools?

[Stephen Samuel]

Right during 3.5, it had more than a dozen remote holes being fixed

Part of the nature of ethereal is that just about any hole is going to be a remot hole, since it is pretty much only dealing with remote (network) data. This is made worse by the fact that it's usually run as root and has no privelege separation (that I know of). OBSD, on the other hand has the luxury of separating remote holes from local holes when they carp about OpenBSD's security.

This, however, does not excuse the ethereal community's somewhat lackadasical attitude towards security. Quite to the contrary, you could argue that it makes security in the design all the more important.

Re:Valuable Open Source Security Assement Tools?

[scottv67]

The other cool thing you could do with Snort (if you are a consultant conducting a network security assessment) is to deploy Snort on the inside network and then show the customer all of the IIS-based attacks that are making it through their Layer 3 firewall because they have their firewall configured to allow inbound TCP port 80 to their webserver.

"But I thought my firewall blocked that stuff!!!"

-Scott

Accountability

[JaxWeb]

If I were to choose software protecting my company, I would use commercial software. Why? Because if something does go wrong, it is the vendors fault and not yours. With free software, it is your fault.

However, for protecting yourself, I think there are ethical reasons to use Free Software - Stallman argues that you should choose software for those reasons alone, and not technical reasons. If you listen to Linus, however, he tells us that technical reasons are valid reasons to choose to software. Your decision on this issue is the first step to your overall decision.

Re:Accountability

[Nothinman]

Right, because pointing a finger at someone you can't really hold accountable or make a lawsuit against is worthwhile. Telling your CEO "but the tool didn't see that problem" potentially makes you look just as dumb as the tool you paid for.

I'm on our network security team and when doing audits we do have a few commercial tools, but we also use OSS tools like Nessus because IME they're better overall.

Re:Accountability

[yamla]

So, you believe that EULAs are completely unenforceable?

Re:Accountability

[Keamos]

Yeah, and if it were up to Stallman, we'd be using HURD.

Re:Accountability

[fm6]

Telling your CEO "but the tool didn't see that problem" potentially makes you look just as dumb as the tool you paid for.

Why? It's not your job to see the problem. By hiding the implentation of the security software, its designers assumed responsibility for making it reliable.

Passing the buck is standard corporate politics. It's true that this leads to a lot of dysfunctional organizations and bad decisions. But if you choose to fight this trend, you better be very good at what you do. And at covering your ass.

Re:Accountability

[ifwm]

"you can't really hold accountable or make a lawsuit against is worthwhile"

Why can't you? The law on this is untested in many areas. What makes you so sure you couldn't make a case against them?

Re:Accountability

[Daedala]

"Because if something does go wrong, it is the vendors fault and not yours. With free software, it is your fault."

Um, check the EULA. Unless you've written a change into your contract, it's unlikely that the vendor actually is responsible.

Free software relieves you of the burden of believing the vendor's got your back. For the most part, they don't.

I have a similar job.

[bigtallmofo]

My job duties sound similar to the story poster... My job description is "Penetration Preventer". My business card title just says, "Cockblocker".

That's your day job...

[AtariAmarok]

"Penetration tester" is your day job, but tell me, do you solve crimes in the evening as a "private dick" ?

Hmmm

[spiffy_dude]

It seems like there is an implicit bias in the question. I would like to see a fair assesment of commercial vs open source tools over a biased statement about how open source tools are better. I'm sure there are worthwhile products in both categories.

Go to SANS training.

[Matey-O]

$3200 spent in a snort bootcamp made the need to buy a $120,000 IDS box moot.

We were reviewing everal six-figure pieces of equipment and found the same thing - we knew they saw traffic they didn't like, but we didn't know WHY.

Now that everybody uses snort rules, the training is still helpful to show you WHAT you're seeing and IF it's truly bad or just another false positive.

FWIW, why get the snort stuff one vendor removed? Just go straight to the source.

Re:Go to SANS training.

[JimmytheGeek]

Amen! Go if you can.

I dig Snort, been using it a while. The SANS training made it USEFUL. The course filled in gaps in my knowledge at a rapid rate, and I usually HATE computer training courses precisely because the bandwidth is too low.

Richard Beijtlich wrote "Tao of Network Security Monitoring" which is a really, really good next step.(http://www.bookpool.com/.x/kzaxqc7ob1/sm/032 1246772)
It covers the use of a variety of different types of intrusion indicators to quickly get to the meat of the matter. He's critical of the SANS course as too bit-addled. I can see what he means - you do spend 2 days (of 6) on tcpdump, vs. just one on Snort per se, but that gives you a great background to use tons of other tools. Once you have that, the other tools are easy.

SANS also has security auditing, incident handling, firewall + VPN, and some PHB type classes.

I'm a fanboy.

Re:Go to SANS training.

[arnie_apesacrappin]

He's critical of the SANS course as too bit-addled. I can see what he means - you do spend 2 days (of 6) on tcpdump, vs. just one on Snort per se, but that gives you a great background to use tons of other tools. Once you have that, the other tools are easy.

I took the IDIC course a while back (i.e. my analyst number is less than 100) and noticed the same thing. The layout was a bit different then, but I caught myself thinking "why are we spending a day and a half reviewing TCP/IP?" After listening to the questions that some people asked, I realized that no matter how much you warn them, people just aren't prepared for the class. Having realized this themselves, the instructors adjusted the curriculum so they can drag as many people through to some level of competency.

I did enjoy my SANS training and I wish I could find another employer that would pay for it. But if you are the kind of student that already knows the first three days of material, you may want to stay away from the track-based courses and do a mix-and-match if possible. Finally, taking a SANS course should be the beginning of your studies in an area, not the end. I see the training as a good broad base in a particular area. You must become an expert on your own.

Re:Go to SANS training.

[fm6]

In other words, you guys rely on the intelligence of well-trained employees, rather than expensive security (no pun intended) blankets.

Wild idea. It'll never catch on.

Penetration Tester

[RasendeRutje]

Penetration Tester?? Not only looking for the obvious (security) holes, but also the tricky ones? Those you don't normally see? Damn where do you learn that

OSSTMM

[randori82]

Even a great methodology is open source [osstmm]

VIsa / MC Compliance

[jfroot]

One reason that many companies need to use a commercial security tool is because of Visa and Mastercard CISP and SDP compliance.

In order to comply you must have various levels of security testing done and certified by an approved vendor.

Don't Forget

[iammrjvo]

There is security implied simply by the fact that the product is open source. That is to say that its failings and potential security weaknesses have been evaluated by a community beyond the original developers and is always open to scrutiny.

What a pile of shit?

[Foofoobar]

So if something goes wrong with your setup, a commercial company will quickly take credit? Riiiiight.

I know Microsoft readily accepts monetary responsibility for their products being crap and causing crashes, viruses and trojans in my system.

In fact, Bill and Steve cut me a check weekly.

besides the obvious

[JeanBaptiste]

snort, ethereal, nmap, etc

one commercial one that I _really_ like is Languard Network Scanner from GFI.

While it is closed source, it has 30-day full functionality, and has limited functionality after that. Still even with the 'limited' functionality, it provides the full scanning capabilities, it just doesn't let you use some of the features that I never use anyways (scheduling, etc).

I'd really recommend giving it a try, its pretty slick.

Anything, as long as...

[tod_miller]

a) it does the job
b) see a.

I do not see the need to stick to ideals in a world of security, use the best tool for the job, and stay vigilant (if OS is the best tool, then only merit it on this, not the fact that it is OS)

Assumed a thief

[rtkluttz]

I work for a company that has an EtherpeekNX license. When they started with the NX line, they now have activation. One time per license. I had to call and threaten a move to open source alternatives with a forced refund due to their policy.

They provide a remote collection agent that can be monitored with the licensed full version. That was not good enough in our instance due to the layout of our network and needing to install our licensed copy, at the work site, fix the problem and then uninstall the software. After much desk pounding they finally gave in and let us have unlimited installs of the same number. But only after threatening a move to open source.

Our take on the issue is, we need to install the product how we see fit. We payed for it. It doesn't matter to us if we aren't using the software how they "envision" it should be used. We were due a refund if they refused to let us use a product we payed for.

Accountability vs Responsibility

[A nonymous Coward]

How do you know you can get any resolution from the people who sold you the software, or developed it? Have you checked the contracts or EULAs? Most EULAs I've seen explicitly disclaim any responsibilty.

Your responsibility is to protect your company AND get it back on its feet after a breakin. You can't rely on a lawsuit to do that in any timely fashion, only after the company has gone out of business and everyone has long since gotten new jobs. Even then, you'd be lucky to get pennies on teh dollar in restitution. So what good does it to sue the developer or seller?

You have to get the company going again as quickly as possible. It just might be helpful to have sources to what failed to see how it failed and how the breakin occurred. Proprietary software is useless there.

Re:Huh?

[OblongPlatypus]

You don't use programs? What, you put the cat-5 in your mouth and try to *taste* the intruders?

The advantage of creating your own security

[hellfire]

IANASS (...Security Specialist) but to me, logic seems to state that having an open source system has an advantage in that the code is there for everyone to see, and that you can add your own code.

Take physical security as a metaphor. You want to secure your physical plant, so you hire a security specialist. You hire his services and he peruses your building. He suggests locks here, cameras there, and a whole plan on making your business less prone to break-ins and the like.

However, what's so great about this? Two things. One, everything is transparent. It's not like joe security officer is selling you a security package and not telling you where he's going to put that $50,000 you just paid for. He has to give you a full plan (the code!) that you approve of. Plus, the plan is customized for you. It's your plan, not someone elses. It's based on your requirements and your specifications. If a security company comes to you and says they'll put a camera in every room and be done with it, is that really enough for you?

Tie that back to open source. The code for open source security solutions are that plan you need. You can provide input on it and change it as much as you want to match your individual needs. And the code will be more unique than a commercial security program, which is the same from site to site.

I can't say that open source is necessarily for everyone. Maybe a camera in every room is all you need. Maybe you just need a security guard out front. The advantages I see here are businesses where security is an important part of business, and where companies don't want control of their own data in the hands of anyone but themselves.

Agreed

[paranode]

Those are great tools to use and the fact that they are free is even better. The only thing I might recommend replacing for a commercial alternative is Nessus. If you can afford it, something like eEye's Retina scanner is a very nice product. It doesn't come cheap, but if you work in a big corporate environment you can probably justify the cost. Not to mention, Nessus is a bit flaky so if you start crashing machines during your testing you will have some angry people to answer to. Don't get me wrong, Nessus is great for a free tool, but it lacks professionalism and is a bit overintrusive at times, even with the safe settings activated.

Counter-point instead

[RyoShin]

I don't have a lot of experience with free software, but I can tell you why people prefer to pay for it: Security in spending.

Basically, most people (including CEOs and the like) think that the more something costs, the better it must be. After all, if Product A costs you $100 and Product B costs you $5, then there must be a lot more features and hard work put into Product A to make it cost more than Product B.

Plus, when people hear 'open source', they think of crackers/evil people getting their hands on the source code and exploiting all sorts of 'holes'. Since they can find out how it works, it must be really easy for them to exploit it.

I wouldn't be surprised if many people, on first look, would rather pay $10 for a Linux distro rather than get it for free because 'free' has all sorts of bad connotations locked in with it this day and age. They assume it's the difference between going to a 12-year old's lemonade stand and going to starbucks for a smoothie. "You get what you pay for."

Deploying Software

[markmcb]

I work for DoD. We tend to go with commercial software for several reasons:

1. Personnel changeover. DoD loves to move people around between departments and installations. It's hard to find people savvy enough to run open-source software and keep them in one spot. It's much easier to give whoever is holding the position a phone number and tell them to call tech support with problems.
2. Personnel skills. DoD is huge. Because of this, the chances of getting skilled and motivated people at all of your sites is slim. Again, the phone call seems to make everything better.
3. Contracts. Things are usually purchased in bundles and as part of a big plan. It's much easier to brief to a non-tech boss that you have the support of another company and not that "I'm sure we can figure it out."
4. Uncle Sam's pockets are deep.

I agree that open source software is often better. But it doesn't give the non-tech group that warm fuzzy it needs to. In the end, the boss doesn't want to up a creek without a paddle. Having that phone number to call adds a much wanted security blanket, even if it's only a facade.

Re:Deploying Software

[Stinking Pig]

Bingo -- same attitude exists in most of the American corporate market, in spades. Maybe rightly so, maybe not, but take note of Red Hat and IBM's successes... this is not about source code or product licensing, it's about that tech support phone number.

Linuxcare and the like flamed out for poor core business practices and poor market targeting (do not ever, I repeat do not EVER, try to make money directly supporting end users). MySQL AB, Best Practical, Trolltech, &c seem to be doing pretty well though....

Re:Deploying Software

[Locke2005]

4. Uncle Sam's pockets are deep.

Thank you very much for wasting my tax dollars, cretin! Seriously, I think this attitude that the "government has lots of money!" is going to be the downfall of the US... here's a subtle reminder: all the money is taken from hardworking citizens, at gunpoint if need be. Or borrowed against future taking from citizens...

Read your contract with your vendor. Fact is, most commercial software contracts don't protect against anything more than refunding the purchase price, even if completely unfit for the purpose for which it was sold!

Re:Deploying Software

[99BottlesOfBeerInMyF]

Having that phone number to call adds a much wanted security blanket, even if it's only a facade.

My first though on reading this was, "well their must be dozens of security firms that will offer support for such popular tools." I mean surely IBM if no on else will be happy to take your money and answer calls about nmap or snort or ethereal? So I did what anyone would do, and googled a bit. I could not find anyone in 2 minutes of searching. Perhaps my google-fu is weak. Is this really an untapped market? I find it hard to believe that no one has jumped on this. I know most enterprise and government shops like to have their own security team, which may preclude the need for this, but it really seems like their should be a huge mid-sized market just looking for a safety net. Heck if you can sell a 5K or 10K a year support contract for open source tools, and all you need is personnel a phone line and a pretty web/download site it seems like you could make out like a fiend. Hmm, forget I said anything.

how can you be sure of quality of closed source ?

[Eternally optimistic]

For security applications, how can you say with any confidence that a closed source product does an adequate job? You are not allowed to examine what it does, instead you have to rely on what the vendor says. Maybe some tool is certified by some "trusted" entity in your industry, but you don't have any control yourself. With open source, you can look, or hire someone to look who works for you.

Re:Accountability -- Reminde me not to hire you

[Stephen Samuel]

I would use commercial software. Why? Because if something does go wrong, it is the vendors fault and not yours.

grunt: Admiral! There's a missile comming our way, and the defence systems have just blue screened!
admiral: Thank god I can blame Microsoft for this!
missile: BOOM!

So you'd use inferior software just because you can point the finger at someone else when the software fails??? Wouldn't you rather use the best software for the job (even if it's cheaper)??

I mean, it's not like most commercial vendors take any responsibility for their software, anyways -- have you read your EULA's recently?

At least with open source software, you have the option of fixing any bugs yourself if the vendor refuses to. With Proprietary code, your only choice is to grin, bend over and wait for your bill.

Right Question?

[Comatose51]

Is that the right question to ask?

"I'd like to ask for Slashdot's experiences and opinions on why you don't need to spend thousands of dollars on an expensive tool to perform a professional security assessment."

It sounds like you're already set in your opinion and just asking for justifications. That doesn't usually develop any new insights or make good comparisons. If you really want to sell people on Open Source, do a fair and un-biased comparison. An obviously biased comparison is easily detectable and loses credibility. I really don't think Open Source needs biased comparisons to look good.

Docmentation

[CKnight]

I'm thinking of writing a how-to for "penetration testers". It'll be titled "Locating Unprotected Backdoor Entrances" or more aptly, "Lube"

Re:Docmentation

[FrankDrebin]

I'm thinking of writing a how-to for "penetration testers". It'll be titled "Locating Unprotected Backdoor Entrances" or more aptly, "Lube"

Don't forget a section on avoiding Trojans. Although they sometimes help with L.U.B.E., they can often get in the way of a successful test.

Different markets

[ectoraige]

The market for commercial security tools is quite different. To begin with, it's smaller than the market for OSS tools. While security professionals may use either, any crackers worth their (or somebody elses) salt are won't be caught using commercial products. Thus, there're probably more 'feature requests' and feedback for the OSS developers to respond to.

Also, a number of commercial products are not written with just the user in mind - the larger ones also involve things like generating pretty reports for use in the CTO's bonus negotiations and suchlike.

Finally, lots of the commercial products try to be competitive by doing everything at once, whereas the OSS tools tend to be more focused on specific functionality, following the traditional unix approach.

Of course, all these points are generalisations and there are exceptions to them all, but that's what you get for asking such a general question.

"I work as a penetration tester..."

[BigZaphod]

If I would have been drinking something when I read that, my screen would be soaked right now...

There is commercial free/open source software

[latroM]

I work as a penetration tester and recently started writing a whitepaper on the benefits of free, open-source security tools over commercial tools.

What if some of the developers of those F/OSS packages are paid money to code free software? MySQL comes to mind when I think commercial free software, although it isn't related to the software you search. There has been always money to be made in free software business. Your question should be about free vs. non-free.

Quoting RMS:``Free software'' does not mean ``non-commercial''. A free program must be available for commercial use, commercial development, and commercial distribution. Commercial development of free software is no longer unusual; such free commercial software is very important.

Re:Accountability -- Reminde me not to hire you

[ifwm]

"So you'd use inferior software"

Commercial is not the same as inferior. MANY MANY commercial products are better than the open source version. Your bias is showing.

Check the license first

[gelfling]

If I recally the openSSH license had some really weird language in it that amounted to "There is a lot of code in this tool. I'm not sure of everything and there may very well be something in here that belongs to someone else. So if they come after you Mr. MegaCorp, don't ask me. It's not my problem."

And that is a bigger problem for our lawyers then the efficacy of the tool itself.

Otherwise, why must it be an either/or decision? Why can't you have a mix of open and commercial and achieve a balance of cost and effectiveness?

Also consider the total lifecycle costs. A $30,000 appliance out of the box may be cheaper than an open source tool running on an 'extra' server you have laying around plus 250 hours/year of your time fucking with it. Sometime the best security is the security that makes the most rational sense for you to afford.

Interesting Business Card

[catdevnull]

I was just wondering about that title "Penetration Tester." Somehow, it seems to garner immediate respect.

Bad link

[parkrrrr]

Kismet can be found at http://www.kismetwireless.net/ ; the link above redirects to the no doubt appropriately-named wirelesscon.com.

Re:Bad link

[KingBahamut]

http://airfart.sourceforge.net/ Is also a really good project.

Nessus new (weird) plugin licensing terms

[andrew71]

I just received e-mail from Fyodor and had this bad bad news.

Nobody mentioned that here.

(and probably nobody will read that since I'm stuck at 0 :)

Nessus is not quite free anymore

[neoThoth]

While the tool itself *is* still free Lightning has made a change in their pricing model regarding the plugins.
Check it out for yourselves, there are three feeds now. The main feed which used to be free is now on a seven day delay. While this doesn't affect a lot of the scanning efforts it is nice to know about the vulnerability that just came out.
Often when a new serious vulnerabilty makes news a company would like to know how they are affected right away. Now they will have to wait 7 days!
I don't think that there is anything wrong with this, I mean the developers at nessus (tenable lightning) have to eat too. But calling it free just seems sort of inaccurate now. Scanners without updated signatures work about as well as razors without the blades.
A 'Direct Feed' is commercially available which entitles subscribers to the latest vulnerability checks. Customers who purchase a Lightning Console or NeWT Pro scanner receive access to this feed with their annual product maintenance.
A 'Registered Feed' is available for free to the general public, but new plugins are added seven days after they are added to the 'Direct Feed'. To obtain access to the 'Registered Feed', users are required to enter contact information for tracking and also agree to Tenable's license agreement for the plugins.
The 'GPL Feed' does not require registration, and includes plugins written by the user community. As manager of the Nessus project, Tenable continues to accept plugins written from the Nessus and NeWT user communities. Plugins accepted with a copyright under the GNU Public License will be distributed to the Direct, Registered and Public feeds at the same time.
Pricing
The access to the GPL feed and to the Registered Feed is free. Pricing for the 'Direct Feed' is based upon the number of Nessus or complimentary copies of NeWT in use within your organization, consultancy or service. The cost is $1200 per scanner per year. For more information, please contact Tenable's sales staff.

vuln scanners

[neoThoth]

With vulnerability scanning there are a few different aspects to consider. the most important feature of a scanner (aside from speed and accuracy) is the level of updates. An out of date scanner is only mildly better then no scanner at all. In this regard commercial software has some advantage for the consumers (IT organizations). It's not that they can blame anyone (as was mentioned in several posts) but there is someone to yell "hey! where the hell is my signature for Vuln XYZ?" With open source there isn't a guarentee that the signature will be made quickly enough. Even nessus (as I pointed out in another post here somewhere) has moved to a pay model for plugins because of the cost of keeping those signatures up to date.
Now one can also take the Open Source approach here and write their OWN signatures but many companies just don't have the staff for that type of thing. The vulnerabilty details are so sparse these days (not so open disclosure rules) that recreating the actual exploit never mind finding a way to detect it remotely is beyond the skill of most teams in the limited timeframe that it's of vital importance. A team will have around 24-48 hours after a patch is released until some evil doer[s] have reverse engineered the patch and created an exploit out of it, slipped in a pre packaged payload and owned 3 out of your 7 class B segments. Sometimes less. I think the ISS worm last year was the record, something like > 20 hours from patch to worm [witty worm i think].
Some intersting article on scanning here and here

Just one other side note about the articles, Foundstone was purchased by McAfee last year so disregard those.

Good penetration testing resource

[PerryResearch]

Mostly open source security tools for pentesting, although there are some commercial products listed as well.

http://www.liveammo.com/LiveAmmo_Security_Tools_Di rectory.htm

One problem in SANS ids methodology

[JimmytheGeek]

Northcutt et. al. have a seriousness assessment that is completely broken. Their model rates an incident by a formula that does not make sense:

S = (C + L) - (HCM + NCM)

Where:
S = severity
C = Criticality (how important the target host is)
L = Lethality of attack
HCM = Host-based countermeasures
NCM = network-based countermeasures

They use different variable names, I think.

Assign a value from 1-5 for C,L,HCM, and NCM

Remember ordinal numbers? You can't multiply them (or do other operations on them) and get any sensible result. For example, last year the Mariners finished 4th (last) in the AL West. You can't multiply their rank of 4 by anything. They aren't 4 times as sucky as Oakland or 4/3rds as sucky as Texas. They are ranked 4th and that's all you can say. More sucky than Texas. If they finish 1 game behind #3, they are ranked 4th same as if they finish 150 games behind.

Similarly, you can't say a Criticality=5 host is 25% more important than a C=4 host. Adding Lethality to Criticality is like adding Favorite Ice Cream to AL West Standings.

Further, Lethality probably has no sensible 5 step progression. I count 4 max steps. No lethality, recon, user-level, 0wn3d. If a step is not at all lethal, why does that increase the severity? (Should be 0-5)

Beyond the mathematics, I have some other conceptual problems: subtracting the assessment of network-based countermeasures. Well, let me see. Give the assessment for network-based countermeasures a high value if it stopped the attack and a low value if it didn't. This tautology advances our interests how? If the exercise doesn't provide the severity, but instead takes it as an input, then the exercise is just busywork. Or take an independent assessment of the network countermeasures- we're proud of our kick-ass firewall, score it 5. It didn't stop the attack, as the vector was entirely within permitted traffic. How does a cool firewall that didn't stop the attack reduce the severity of the event?

The same argument holds for host-based countermeasures (host firewall, av, tripwire, current patching, etc)

I grant that the folks proposing this model have a lot more experience than I do, but they should probably admit that people pull these numbers out of their asses to fit a predetermined conclusion. The severity rating should inform decisions about response. Most of the steps should give binary results: respond | not respond

Is this an attack/hostile? yes/no
Is the target something we care about? yes/no
Did the attack succeed? yes/no
Does it represent a threat even if it failed? yes/no
and so on

The prioratization of responses is probably inevitably a second calculation.

It bugged me that I had to use this methodology to get my certification.

I am otherwise impressed: do not hold SANS/GIAC certs in the same contempt that the CNE and MCSE deserve. The GCIA was a massive amount of work that actually exercised the skills being evaluated. The papers of those who pass it are publically available at the SANS website so you can see someone's chops and writing style, if you are checking someone out for a job or contract.