Slashdot Mirror
Recommendations for Third Party Security Audits?
palehorse asks: "I am a developer/DBA/etc for a very large State Govt. Agency on the East Coast. We have been subjected to an increasing number of break-ins and website defacements over the past few months. My boss has recently been tasked by our CIO to find a reputable third party (not us or our ISP) to come in and do a complete and independent security assessment/vulnerability analysis for us. Since I'm the guy who usually bugs folks about security, she tasked me to come up w/ a list of firms who could do this for us. and a plan on what to test for and how. I've done the whole Google search/ZD-Net search/etc, which has given me way to many folks who do this kind of stuff, from ISS and IBM on down. Consequently I wanted to get some feedback/suggestions from the Slashdot community on where to go from here."
"Please keep in mind that while we're a large government agency, we have a small and overworked IT staff who have no real experience in internet/web security, and who are just now getting into a serious web presence.
Here are the main questions that I have:
- Who have you used, and were they any good?
- What should we look for in evaluating who to contact and their proposals?
- What would you have done differently?
- What services should we ask for?
- How do we manage the contract to make sure we're not getting a snow-job?
- How do we use the result to get buy-in from management (who will probably need to lay out money for changes) and from the developers (who may be adamantly opposed to changing their systems, either for ego reasons or it's just a lot of work)?
- How often should we re-do these audits?
Check out Foundstone. They'll do it and do it right.
The truth about Scientology, Xenu, and you: Operation Clambake
First it depends on what OS you are running, and how you have them configured. Second, ISS is a good security team. I don't know much about them, but they have a very good reputation for security, and are a well advanced team of individuals. When my boss was hacked 2 months ago, he called me, and hired me within 5 minutes of the interview (After I went over his head about replacing RedHat with Slackware).
If you want to spend large bucks, hire a security firm such as ISS. If your agency doesn't want to spend a lot of money, call a bunch of geeks (like me) to come in and audit the system. IE: replacing wu-ftpd with pure-ftpd, IIS with Apache 2.0. Find the services that are full of holes, and replace them with somthing that has a reputation of security.
--------------------------
Is this a sig?
--------------------------
We had an audit done by ISS about a year ago. They did a good job. They came in, did some interviews, and proceded to test the specified systems. We got back some very good documentation showing any problems as well as things that were not problems.
I don't remember the cost, but I'd use them again.
These guys did an audit of one of my website networks once for a bank, not too bad. Guy mostly knew his stuff and was easy to work with. Cute name too:
http://www.wealsowalkdogs.com/
I don't know if counterpane.com does audits, but you should definitely consider their managed security service if you don't have a dedicated on-staff security person.
Finally beware these types of audits, they often don't look at your procedures and policies, which are the root cause of most problems. It's always good to have external cross checks from a different point of view, but be very careful about assigning too much importantace to them.
I've worked both for a big 5 accounting firm and a defense contractor doing these things.
You should look for:
- resumes of staff performing this activity, for the folks who will actually be conducting the work. How experienced are they? Beware of firms that send their people to a one week training class then turn them loose as experts.
- Breadth of experience in OS, server and middleware products. Don't hire a bunch of UNIX bigots if you have WIn 2K servers. Not only will these folks not be familiar with the technology, they will also have a bias towards bad-mouthing it.
- Do they understand how to rank and prioritize the risks based on the needs of *your* environment? Anyone can generate a cookie-cutter report from a packaged tool. To what extent do they apply some human intelligence to this?
- Following from this, what does the report look like? Do you get a cookie-cutter intro with a zillion pages of ISS output, or do you get something meant for a human being to read?
- Breadth of assessment - do they look at routers and switches? Servers? Applications (is that Oracle financial application wide open)? Desktop machines?
- Are results based solely on a network scan, or do they actually look at host configs that may not be visible from an outside scan? Do they interview staff to get some idea of practices?
http://www.lumeta.com/ We help by performing a scan of your network and show you the holes in it. If you're familiar with the Internet Mapping Project, and Bill Cheswick, then you'll have a good idea of some of the stuff we do here.
Lucky for me I always have Emergency Pants!
There are a couple things you want from an audit (I've seen a couple from the recieving end, both really good and absolutely terrible):
1) you want a complete report, not just a management summary. Make sure there's guidance in the report on how to fix the problems they find, or at least a pointer to where to find the information to fix them.
2) black-box "we can hack anything" audits are sexy, but won't show you the whole picture. Make sure they're looking at both the external settings and any local policy security settings on the machine.
3) Ask to have some of your staff sit in on the audits...you want to learn from this audit as much as possible. If they say "no", ask why. If they're just trying to protect their "script-fu", run...they're probably fake.
4)Get a contract in place that makes it very clear what they are supposed to audit, what they are not supposed to audit, and how they are allowed to do it...get that in place *before* the audit starts. (a "terms of engagement"). This includes what IPs to audit, and what techniques (DoS, social engineering, etc) are allowed.
5) as others have mentioned above, ask for references. If they can't provide them, worry.
I'll stop now. I'm sure there's more, but that's what occurred off the top of my head.
I get Bruce Schneier's CRYPTO-GRAM. He runs a security company www.counterpane.com. The dude knows his stuff and his employees probably aren't slackers either.
Depending on the scope, Systems Experts did a very good job for my company, and we're about 30,000 people. These guys are just what their name states- experts in the field. I've worked with two of them, and they take their job very seriously. Their job is to find vulnerabilities. They will, if you ask them, recommend a fix. See www.systemexperts.com.
Another company that you might find useful is Lumeta. This is Bill Cheswick's company, and they take an innovative approach, in particular relating to networking audits. They map your network and create visualizations. See www.lumeta.com. One of their senior folk is Tom Limoncelli, whose book "The Practice of System and Network Administration" was recently reviewed on SlashDot.
Better yet.. and maybe I'm going out on a limb here. Get an admin.
Why is a programmer at a large state agency handling security? A full time admin is a must. A security audit only checks for this weeks problems...
Hi,
I work in this specific industry and you need to be careful how you screen companies. There are a few caveats to watch for:
Ask for references but don't be surprised if they can't give a lot. Why? My company does a lot of work for the Federal Gov't as well as state governments and the work is usually under a NDA. You wouldn't like me to say "sure we audited so and so and found 25 holes" either.
Ask for their methodology and review it. Don't always believe the hype about "custom tools" etc.. Make sure they have some level of redundancy. I worked for one firm that used strobe and ISS and nothing more. Ask what tools they are going to use. Be nervous if they don't want to tell you. You'd be surprised at how many "big players" really are scam artists.
Make sure the resume's you see in the proposal are the people doing the work. You don't want to hire and pay for mudge, only to have Tony the pony come run the scan.
Check the reputation of the finalists. You definitely dont want a fly by night shop doing your work, or a company that might not have good ethics.
dewke
Oderint dum metuant
Security is a mindset and process at least as much as an implimentation. Therefore you don't just need a good aduit, but you need continuing aduits.
Counterpane and Bruce Schneir are the best known names in cyrptography consulting today, but I don't expect them to know much about much about virus attacks.
You probably need several different audits (or maybe an extensive IBM audit) just to get started. However never allow the same auditors in more than two years in a row. (The first year to find problems, then second to find problems in the fixes) People who know what is going on in detail should be working for you, you want an outside, untainted by prior knowledge and and hard work.
Make it a policy that you hire auditors on a two year contract, and make it clear that it is NOT renewable, and they cannot get further buisness in this audit for two years.
Try everyone. Once all the big guys have been through and given you a stamp of approveal you should allow the common theif to see your entire procedures, and get his recomendataions. (Don't nessicarly follow them of course). Try small companies and big ones. Small companies tend to cover one area very well, big ones broad areas not as deep. You need both.
This isn't an overnight fix. It took openBSD several years to become secure. Today they have a well earned reputation as least breakable system. If I remember right they had to go over the same code 3-6 times before they got most of the secuirty problems out. They were not even looking at security, they were looking for things that were wrong.
If you buy closed source code (nothing wrong with it), make sure you vender works for security. You can't fix the holes in a sieve with confidence that the fix will hold. Open source is a little better, but you might have to pay someone to fix those.
Remember that external audits are an assurance. Most of the work is internal. So make sure management is giving everyone enough time to fix the bugs in their own code/implimentation.
When I was working with the RCMP (via a System Integrator), they were undergoing a complete evaluation of the security of the various public wireless providers that they planned to deploy their police mobile products upon. This required extensive reviews of communications protocols, physical and procedural aspects of security, who was getting access to what/when/how was it controlled, auditing, and physical security of the various locales.
The guys the RCMP had do it were experienced, knowledgeable, and had ties/backgrounds that included work with the Canadian Security Establishment (Canadian NSA) and the Canadian Military. One of the guys I worked with had just finished some serious security work for CSE. I know enough about crypto and comms protocols myself to know when (as far as security)I meet people who are "the real deal". These guys were it. And they opened the eyes of some of the public wireless providers in a big way.
They can be found via the info at the bottom of this link here.
-- Mal: "Well they tell you: never hit a man with a closed fist. But it is, on occasion, hilarious."
Counterpane only provides monitoring services. Keen if he wants someone to look at his IDS or Firewall logs.
"oohhh... I didn't know Schopenhauer was a philosopher!"
Up front I want to point out that I don't want to make a completely shameless plug for my company and what I do. I did leave some contact info available in case the person in question wanted to contact me. The comments here are my own and not that of my employer, etc. If the person who submitted this Ask Slashdot is happy with another firm, that is fine with me, I'm an engineer _not_ a salesman.
:)
Here are the main questions that I have:
Who have you used, and were they any good?
I work for a company that does full service security penetration testing, secure network architecture design and implementation, remote monitoring of IDS and other logs. You can email me through my slashdot user name link if you wish or hit our website www.caci-nsg.com. Therefore I use my own knowledge and that of my co-workers (some of whom work for Attrition.org btw) and yes, we are very good.
What should we look for in evaluating who to contact and their proposals?
You should make sure they have experience with the various OSes you run. People who know how to knock over a UNIX system may not do well against Microsoft and vice versa. Make sure they tell you what needs fixing AND how to fix it.
Some companies I've had to compete with only showed up with one system to run the ISS scanner, generated a _very_ thick report of what was wrong, and left.
No single scanner is perfect and if you don't have human intelligence to interpret the results the test may be meaningless. I've seen the ISS scanner tell people they had a Windows NT system that needed to be fixed. When we checked out the system in question it turned out to be HP-UX!
What would you have done differently?
There are things our team learns at every pen-test we do. Some things I want to do differently would be to standardize our methodology more. One problem is that every network has something about it that makes it unique. This is where you can either go cheap for an "off the rack" solution to your testing or pay for a "tailored suit". Be sure that the team has some real experience behind them though. You don't want the tailor fresh out of tailoring school.
What services should we ask for?
You should ask for a complete report of what the team is able to access on your network. You want to know what can they break into from the internet and what can they break into if they were sitting internally. You need to understand the difference between a theoretical exploit based on how your network is configured and a real vulnerability based on a missing service pack. This tells you about what external attackers can do as well as what disgruntled employees can do. It may also tell you how bad a Sys Admin you have running things. I've gotten one Sys Admin fired because of what I found and his poor reaction to my findings. You'll want a report that explains in detail why you are vulnerable, what to do to fix it, and if possible the impact this may have on your day to day operations.
How do we manage the contract to make sure we're not getting a snow-job?
You can have the team demonstrate for you how they got in. Have them leave a file behind, pull down a password file and crack it, etc. Any team should be willing to discuss things very honestly with you. You may wish to start small. An external test only for a small amount of $$ and time. This lets you evaluate them without being burned too badly.
How do we use the result to get buy-in from management (who will probably need to lay out money for changes) and from the developers (who may be adamantly opposed to changing their systems, either for ego reasons or it's just a lot of work)?
When I broke into a customer that was a credit union and got customer account data, it got their attention. If the test team steals emails or other things from the CEO or other big-wigs, etc. and it doesn't get proper attention with management, I'd look for a new place to work.
How often should we re-do these audits?
Generally twice a year. The main thing is that after the first one you may have a ton of work to do to fix things. You don't want another test until you have had reasonable time to complete your changes. I've had some customers take a year to get fixed up for another test.
Again, I know there's a lot out there by the security firms themselves about this, but I'm really looking for a 'keyboarder-in-the-trenches' view as well. Horror stories are appreciated, since they may give us an idea what to watch out for."
I just hope I was helpful with what I mentioned here. Keep in mind that if you are a government agency you probably have to put the contract out for a bidding process. Write up your expectations as clearly as possible and leave time for a question/response period from the bidding companies. The intelligence of their questioning will tell you a lot. If they don't ask many questions it probably means they don't know what to ask and won't be very good.
Do really dense people warp space more than others?
The best thing you can do, if you really need to be online, is to TRAIN YOUR PEOPLE. First in IT, if necessary, then in security.
Doing anything else is a waste of resources that will lead only to a false sense of... well, security.
That is all.
Before you bring the auditors in, learn more about your systems first. Go to http://www.cisecurity.org/ They provide benchmarks and best practices for system security and administration. These are available for a free download. When executed on your servers, they provide you with the current state of your systems as well as a list of suggested tasks for improvement. While I take exception to some of their suggestions, on the whole I found the Solaris benchmark very good.
http://www.cert.org is also a good resource.