Slashdot Mirror
Recommendations for Third Party Security Audits?
palehorse asks: "I am a developer/DBA/etc for a very large State Govt. Agency on the East Coast. We have been subjected to an increasing number of break-ins and website defacements over the past few months. My boss has recently been tasked by our CIO to find a reputable third party (not us or our ISP) to come in and do a complete and independent security assessment/vulnerability analysis for us. Since I'm the guy who usually bugs folks about security, she tasked me to come up w/ a list of firms who could do this for us. and a plan on what to test for and how. I've done the whole Google search/ZD-Net search/etc, which has given me way to many folks who do this kind of stuff, from ISS and IBM on down. Consequently I wanted to get some feedback/suggestions from the Slashdot community on where to go from here."
"Please keep in mind that while we're a large government agency, we have a small and overworked IT staff who have no real experience in internet/web security, and who are just now getting into a serious web presence.
Here are the main questions that I have:
- Who have you used, and were they any good?
- What should we look for in evaluating who to contact and their proposals?
- What would you have done differently?
- What services should we ask for?
- How do we manage the contract to make sure we're not getting a snow-job?
- How do we use the result to get buy-in from management (who will probably need to lay out money for changes) and from the developers (who may be adamantly opposed to changing their systems, either for ego reasons or it's just a lot of work)?
- How often should we re-do these audits?
Check out Foundstone. They'll do it and do it right.
The truth about Scientology, Xenu, and you: Operation Clambake
Walk down to your local highschool. Walk over to the kid with the purple hair and the /. tshirt.
Tell him you'll give him or her a free laptop, and 5 cases of Code Red if they can break in and tell you how they did it.
\Drew National Data Director, John Edwards for President
We had an audit done by ISS about a year ago. They did a good job. They came in, did some interviews, and proceded to test the specified systems. We got back some very good documentation showing any problems as well as things that were not problems.
I don't remember the cost, but I'd use them again.
The 1st rule is never, ever ask anyone who sells security products to do an audit, they will just try to sell you something.
IMHO opinion an audit is not what you need, spend the money employing someone who does know about security to get (and keep) things ship shape. Security is an ongoing issue and can't be solved by a one of check, the audit could be perfect but your still wide open the next time some kiddie finds a hole in your preferred webserver software.
These guys did an audit of one of my website networks once for a bank, not too bad. Guy mostly knew his stuff and was easy to work with. Cute name too:
http://www.wealsowalkdogs.com/
I don't know if counterpane.com does audits, but you should definitely consider their managed security service if you don't have a dedicated on-staff security person.
Finally beware these types of audits, they often don't look at your procedures and policies, which are the root cause of most problems. It's always good to have external cross checks from a different point of view, but be very careful about assigning too much importantace to them.
What about Mitnick...
Oh but he can't access computers...
[My english is better than most other people's Turkish, so please point out mistakes politely. Thank you.]
I've worked both for a big 5 accounting firm and a defense contractor doing these things.
You should look for:
- resumes of staff performing this activity, for the folks who will actually be conducting the work. How experienced are they? Beware of firms that send their people to a one week training class then turn them loose as experts.
- Breadth of experience in OS, server and middleware products. Don't hire a bunch of UNIX bigots if you have WIn 2K servers. Not only will these folks not be familiar with the technology, they will also have a bias towards bad-mouthing it.
- Do they understand how to rank and prioritize the risks based on the needs of *your* environment? Anyone can generate a cookie-cutter report from a packaged tool. To what extent do they apply some human intelligence to this?
- Following from this, what does the report look like? Do you get a cookie-cutter intro with a zillion pages of ISS output, or do you get something meant for a human being to read?
- Breadth of assessment - do they look at routers and switches? Servers? Applications (is that Oracle financial application wide open)? Desktop machines?
- Are results based solely on a network scan, or do they actually look at host configs that may not be visible from an outside scan? Do they interview staff to get some idea of practices?
Perhaps other agencies within your state might already have someone doing this. This someone could come up with recommendations that could be used across the board. Plus it might make writing the contract easier.
Wait.. What am I saying? This is government; agencies don't work together. Nevermind...
There are a couple things you want from an audit (I've seen a couple from the recieving end, both really good and absolutely terrible):
1) you want a complete report, not just a management summary. Make sure there's guidance in the report on how to fix the problems they find, or at least a pointer to where to find the information to fix them.
2) black-box "we can hack anything" audits are sexy, but won't show you the whole picture. Make sure they're looking at both the external settings and any local policy security settings on the machine.
3) Ask to have some of your staff sit in on the audits...you want to learn from this audit as much as possible. If they say "no", ask why. If they're just trying to protect their "script-fu", run...they're probably fake.
4)Get a contract in place that makes it very clear what they are supposed to audit, what they are not supposed to audit, and how they are allowed to do it...get that in place *before* the audit starts. (a "terms of engagement"). This includes what IPs to audit, and what techniques (DoS, social engineering, etc) are allowed.
5) as others have mentioned above, ask for references. If they can't provide them, worry.
I'll stop now. I'm sure there's more, but that's what occurred off the top of my head.
Depending on the scope, Systems Experts did a very good job for my company, and we're about 30,000 people. These guys are just what their name states- experts in the field. I've worked with two of them, and they take their job very seriously. Their job is to find vulnerabilities. They will, if you ask them, recommend a fix. See www.systemexperts.com.
Another company that you might find useful is Lumeta. This is Bill Cheswick's company, and they take an innovative approach, in particular relating to networking audits. They map your network and create visualizations. See www.lumeta.com. One of their senior folk is Tom Limoncelli, whose book "The Practice of System and Network Administration" was recently reviewed on SlashDot.
Security is a mindset and process at least as much as an implimentation. Therefore you don't just need a good aduit, but you need continuing aduits.
Counterpane and Bruce Schneir are the best known names in cyrptography consulting today, but I don't expect them to know much about much about virus attacks.
You probably need several different audits (or maybe an extensive IBM audit) just to get started. However never allow the same auditors in more than two years in a row. (The first year to find problems, then second to find problems in the fixes) People who know what is going on in detail should be working for you, you want an outside, untainted by prior knowledge and and hard work.
Make it a policy that you hire auditors on a two year contract, and make it clear that it is NOT renewable, and they cannot get further buisness in this audit for two years.
Try everyone. Once all the big guys have been through and given you a stamp of approveal you should allow the common theif to see your entire procedures, and get his recomendataions. (Don't nessicarly follow them of course). Try small companies and big ones. Small companies tend to cover one area very well, big ones broad areas not as deep. You need both.
This isn't an overnight fix. It took openBSD several years to become secure. Today they have a well earned reputation as least breakable system. If I remember right they had to go over the same code 3-6 times before they got most of the secuirty problems out. They were not even looking at security, they were looking for things that were wrong.
If you buy closed source code (nothing wrong with it), make sure you vender works for security. You can't fix the holes in a sieve with confidence that the fix will hold. Open source is a little better, but you might have to pay someone to fix those.
Remember that external audits are an assurance. Most of the work is internal. So make sure management is giving everyone enough time to fix the bugs in their own code/implimentation.
When I was working with the RCMP (via a System Integrator), they were undergoing a complete evaluation of the security of the various public wireless providers that they planned to deploy their police mobile products upon. This required extensive reviews of communications protocols, physical and procedural aspects of security, who was getting access to what/when/how was it controlled, auditing, and physical security of the various locales.
The guys the RCMP had do it were experienced, knowledgeable, and had ties/backgrounds that included work with the Canadian Security Establishment (Canadian NSA) and the Canadian Military. One of the guys I worked with had just finished some serious security work for CSE. I know enough about crypto and comms protocols myself to know when (as far as security)I meet people who are "the real deal". These guys were it. And they opened the eyes of some of the public wireless providers in a big way.
They can be found via the info at the bottom of this link here.
-- Mal: "Well they tell you: never hit a man with a closed fist. But it is, on occasion, hilarious."
Surely you've already contacted Gibson Research to help protect you against script kiddies, armed with the raw sockets in Windows XP, from taking over not only your servers, but the entire internet!
www.grc.com
J
You are going to have to define the scope of the audit. Is it just web servers, desktops, your security policies, legacy or the whole ball of wax? Are you talking a mixed environment (multiple-Unix, Windows, Mac, other?)
How wide is your network area? Multiple locations? Same cities?
How about your network infrastructure itself? Routers, switches, etc.
A complete audit can take a while and cost a lot of $$, especially if you have a wide range of system types and network spread. It also can depend on how deep you want the audit to go.
I work for Lucent doing large scale audits, so can only comment on what I've experienced. Security is as much policy, training and implementation as it is software/hardware.
E-mail me if you want some detailed information.
Charles Hill
Learning HOW to think is more important than learning WHAT to think.
I dunno, at least you can be sure that Arthur Anderson won't be leaving your passwords around on paper.
Up front I want to point out that I don't want to make a completely shameless plug for my company and what I do. I did leave some contact info available in case the person in question wanted to contact me. The comments here are my own and not that of my employer, etc. If the person who submitted this Ask Slashdot is happy with another firm, that is fine with me, I'm an engineer _not_ a salesman.
:)
Here are the main questions that I have:
Who have you used, and were they any good?
I work for a company that does full service security penetration testing, secure network architecture design and implementation, remote monitoring of IDS and other logs. You can email me through my slashdot user name link if you wish or hit our website www.caci-nsg.com. Therefore I use my own knowledge and that of my co-workers (some of whom work for Attrition.org btw) and yes, we are very good.
What should we look for in evaluating who to contact and their proposals?
You should make sure they have experience with the various OSes you run. People who know how to knock over a UNIX system may not do well against Microsoft and vice versa. Make sure they tell you what needs fixing AND how to fix it.
Some companies I've had to compete with only showed up with one system to run the ISS scanner, generated a _very_ thick report of what was wrong, and left.
No single scanner is perfect and if you don't have human intelligence to interpret the results the test may be meaningless. I've seen the ISS scanner tell people they had a Windows NT system that needed to be fixed. When we checked out the system in question it turned out to be HP-UX!
What would you have done differently?
There are things our team learns at every pen-test we do. Some things I want to do differently would be to standardize our methodology more. One problem is that every network has something about it that makes it unique. This is where you can either go cheap for an "off the rack" solution to your testing or pay for a "tailored suit". Be sure that the team has some real experience behind them though. You don't want the tailor fresh out of tailoring school.
What services should we ask for?
You should ask for a complete report of what the team is able to access on your network. You want to know what can they break into from the internet and what can they break into if they were sitting internally. You need to understand the difference between a theoretical exploit based on how your network is configured and a real vulnerability based on a missing service pack. This tells you about what external attackers can do as well as what disgruntled employees can do. It may also tell you how bad a Sys Admin you have running things. I've gotten one Sys Admin fired because of what I found and his poor reaction to my findings. You'll want a report that explains in detail why you are vulnerable, what to do to fix it, and if possible the impact this may have on your day to day operations.
How do we manage the contract to make sure we're not getting a snow-job?
You can have the team demonstrate for you how they got in. Have them leave a file behind, pull down a password file and crack it, etc. Any team should be willing to discuss things very honestly with you. You may wish to start small. An external test only for a small amount of $$ and time. This lets you evaluate them without being burned too badly.
How do we use the result to get buy-in from management (who will probably need to lay out money for changes) and from the developers (who may be adamantly opposed to changing their systems, either for ego reasons or it's just a lot of work)?
When I broke into a customer that was a credit union and got customer account data, it got their attention. If the test team steals emails or other things from the CEO or other big-wigs, etc. and it doesn't get proper attention with management, I'd look for a new place to work.
How often should we re-do these audits?
Generally twice a year. The main thing is that after the first one you may have a ton of work to do to fix things. You don't want another test until you have had reasonable time to complete your changes. I've had some customers take a year to get fixed up for another test.
Again, I know there's a lot out there by the security firms themselves about this, but I'm really looking for a 'keyboarder-in-the-trenches' view as well. Horror stories are appreciated, since they may give us an idea what to watch out for."
I just hope I was helpful with what I mentioned here. Keep in mind that if you are a government agency you probably have to put the contract out for a bidding process. Write up your expectations as clearly as possible and leave time for a question/response period from the bidding companies. The intelligence of their questioning will tell you a lot. If they don't ask many questions it probably means they don't know what to ask and won't be very good.
Do really dense people warp space more than others?
The first step (really!) is to get a security policy in place. This really doesn't have to be anything special-- but it does need the buy-in of ALL groups affected (sysadmins, developers, marketing, sales, executives, etc.) That's really the only hard part.
Probably the quickest way to get started is to head to the SANS security policy project and adapt their sample policies to your company. This is one of those rare cases where it's more important to get something in than it is to get it right the first time. Policies can be changed fairly easily-- but you don't want to go to all the trouble to implement a secure environment only to have someone on the inside fighting you every step of the way.
Now the fun part-- actually securing your systems. Here are some pointers on places to start:
1) Review the SANS "top 10" security vulnerabilities and make sure they're covered.
2) Review Lance Spitz's excellent collection of host security information and make sure to follow his recommendations.
3) Make sure your firewall rules are set up with the security best practice of "minimum access to get the job done". Far too many firewalls allow traffic they shouldn't.
4) Get NMAP, a network mapper, port scanner, and OS identifier and run it from the Internet to your exposed (i.e. DMZ) hosts. Also run it from your exposed hosts to your internal network to validate that only the traffic that should get in can get in. (The traffic allowed back in from your DMZ should be very little, preferably none.) If you find anything that is inconsistent with what you think should be happening, check your firewall rules again.
5) Grab a copy of the Nessus security scanner and run it against your newly secured systems. If it finds anything, read the description of the problem and see if it's something you can fix. You can bet that everything you find here will also show up on your "security audit" since most "audits" are just someone running a tool like this and then feeding the output to the consultants to make it all pretty for management.
6) You should have most of the obvious, widespread holes plugged by now. This would be a good time to get some sysadmins out to some classes. Verisign has a number of excellent general Internet security classes. I'm sure there are lots of other good places, too. I was pleased with Verisign because of their Internet focus. Too many security classes only concentrate on host security and neglect network security.
7) Get the application developers at your site to read and follow Dave Wheeler's writing secure programs guidelines. This is a lower priority than OS/network security since these holes are likely to be specific to your site only. Only a determined hacker is likely to find and exploit them-- however exploiting application bugs/holes can severely disrupt your business. What happens when an electronic data interchange transaction gets bogus data inserted? How far will that bogus information make it in before it's detected? In the worst case these bugs could result in people getting free products/subscriptions, stealing credit card info, or destroying data inside your systems.
8) Now it's time to get that audit. They will be able to tell you what you missed in the previous 7 steps. Why wait so long? Most places will keep looking until they find something to report. If you do this too soon, the subtle security problems will be lost in the noise of all the obvious problems the previous 7 steps would have fixed. If you do this last, only the "hard" problems are left for them to find.
Remember above all that this is an ongoing process. Keep current on your patches, and repeat all the above steps regularly to keep all the bad guys away.