Slashdot Mirror
Recommendations for Third Party Security Audits?
palehorse asks: "I am a developer/DBA/etc for a very large State Govt. Agency on the East Coast. We have been subjected to an increasing number of break-ins and website defacements over the past few months. My boss has recently been tasked by our CIO to find a reputable third party (not us or our ISP) to come in and do a complete and independent security assessment/vulnerability analysis for us. Since I'm the guy who usually bugs folks about security, she tasked me to come up w/ a list of firms who could do this for us. and a plan on what to test for and how. I've done the whole Google search/ZD-Net search/etc, which has given me way to many folks who do this kind of stuff, from ISS and IBM on down. Consequently I wanted to get some feedback/suggestions from the Slashdot community on where to go from here."
"Please keep in mind that while we're a large government agency, we have a small and overworked IT staff who have no real experience in internet/web security, and who are just now getting into a serious web presence.
Here are the main questions that I have:
- Who have you used, and were they any good?
- What should we look for in evaluating who to contact and their proposals?
- What would you have done differently?
- What services should we ask for?
- How do we manage the contract to make sure we're not getting a snow-job?
- How do we use the result to get buy-in from management (who will probably need to lay out money for changes) and from the developers (who may be adamantly opposed to changing their systems, either for ego reasons or it's just a lot of work)?
- How often should we re-do these audits?
We had an audit done by ISS about a year ago. They did a good job. They came in, did some interviews, and proceded to test the specified systems. We got back some very good documentation showing any problems as well as things that were not problems.
I don't remember the cost, but I'd use them again.
These guys did an audit of one of my website networks once for a bank, not too bad. Guy mostly knew his stuff and was easy to work with. Cute name too:
http://www.wealsowalkdogs.com/
I don't know if counterpane.com does audits, but you should definitely consider their managed security service if you don't have a dedicated on-staff security person.
Finally beware these types of audits, they often don't look at your procedures and policies, which are the root cause of most problems. It's always good to have external cross checks from a different point of view, but be very careful about assigning too much importantace to them.
I've worked both for a big 5 accounting firm and a defense contractor doing these things.
You should look for:
- resumes of staff performing this activity, for the folks who will actually be conducting the work. How experienced are they? Beware of firms that send their people to a one week training class then turn them loose as experts.
- Breadth of experience in OS, server and middleware products. Don't hire a bunch of UNIX bigots if you have WIn 2K servers. Not only will these folks not be familiar with the technology, they will also have a bias towards bad-mouthing it.
- Do they understand how to rank and prioritize the risks based on the needs of *your* environment? Anyone can generate a cookie-cutter report from a packaged tool. To what extent do they apply some human intelligence to this?
- Following from this, what does the report look like? Do you get a cookie-cutter intro with a zillion pages of ISS output, or do you get something meant for a human being to read?
- Breadth of assessment - do they look at routers and switches? Servers? Applications (is that Oracle financial application wide open)? Desktop machines?
- Are results based solely on a network scan, or do they actually look at host configs that may not be visible from an outside scan? Do they interview staff to get some idea of practices?
There are a couple things you want from an audit (I've seen a couple from the recieving end, both really good and absolutely terrible):
1) you want a complete report, not just a management summary. Make sure there's guidance in the report on how to fix the problems they find, or at least a pointer to where to find the information to fix them.
2) black-box "we can hack anything" audits are sexy, but won't show you the whole picture. Make sure they're looking at both the external settings and any local policy security settings on the machine.
3) Ask to have some of your staff sit in on the audits...you want to learn from this audit as much as possible. If they say "no", ask why. If they're just trying to protect their "script-fu", run...they're probably fake.
4)Get a contract in place that makes it very clear what they are supposed to audit, what they are not supposed to audit, and how they are allowed to do it...get that in place *before* the audit starts. (a "terms of engagement"). This includes what IPs to audit, and what techniques (DoS, social engineering, etc) are allowed.
5) as others have mentioned above, ask for references. If they can't provide them, worry.
I'll stop now. I'm sure there's more, but that's what occurred off the top of my head.
Depending on the scope, Systems Experts did a very good job for my company, and we're about 30,000 people. These guys are just what their name states- experts in the field. I've worked with two of them, and they take their job very seriously. Their job is to find vulnerabilities. They will, if you ask them, recommend a fix. See www.systemexperts.com.
Another company that you might find useful is Lumeta. This is Bill Cheswick's company, and they take an innovative approach, in particular relating to networking audits. They map your network and create visualizations. See www.lumeta.com. One of their senior folk is Tom Limoncelli, whose book "The Practice of System and Network Administration" was recently reviewed on SlashDot.
Security is a mindset and process at least as much as an implimentation. Therefore you don't just need a good aduit, but you need continuing aduits.
Counterpane and Bruce Schneir are the best known names in cyrptography consulting today, but I don't expect them to know much about much about virus attacks.
You probably need several different audits (or maybe an extensive IBM audit) just to get started. However never allow the same auditors in more than two years in a row. (The first year to find problems, then second to find problems in the fixes) People who know what is going on in detail should be working for you, you want an outside, untainted by prior knowledge and and hard work.
Make it a policy that you hire auditors on a two year contract, and make it clear that it is NOT renewable, and they cannot get further buisness in this audit for two years.
Try everyone. Once all the big guys have been through and given you a stamp of approveal you should allow the common theif to see your entire procedures, and get his recomendataions. (Don't nessicarly follow them of course). Try small companies and big ones. Small companies tend to cover one area very well, big ones broad areas not as deep. You need both.
This isn't an overnight fix. It took openBSD several years to become secure. Today they have a well earned reputation as least breakable system. If I remember right they had to go over the same code 3-6 times before they got most of the secuirty problems out. They were not even looking at security, they were looking for things that were wrong.
If you buy closed source code (nothing wrong with it), make sure you vender works for security. You can't fix the holes in a sieve with confidence that the fix will hold. Open source is a little better, but you might have to pay someone to fix those.
Remember that external audits are an assurance. Most of the work is internal. So make sure management is giving everyone enough time to fix the bugs in their own code/implimentation.