Slashdot Mirror
Best Security / Vulnerability Testing Firms for Web Apps?
An anonymous reader writes "I'm in charge of a web application that must be extremely secure. Users will be submitting highly sensitive information to each other using the site. Security must be world-class. We believe we've built site in such a way that minimizes security risks and we've implemented numerous policies and procedures company-wide to increase security. We'd like a third-party to perform exhaustive and ongoing security tests: automated tests, application testing, and more, to check for things like cross-site scripting issues, server misconfigurations, form/hidden field manipulation, command injection, cookie poisoning, known platform vulnerabilities, etc. What companies would Slashdot readers recommend for these types of services?"
is send the websites to your parents computer, then stop by in a few weeks to see if their machines are fucked up.
http://sandsecurity.com/
This is one of the things that SandSecurity does for its clients. Try them out!
Full disclosure: friend of the owner
Whoever stated that signature sizes should be limited to one hundred and twenty characters can just go ahead and kiss my
I'm the CISO of a Fortune 100, pretty much the defacto standard in assessments for penetration testing or Web Application Security is SecureState http://www.securestate.com. They are pretty much the standard in most large sized organizations.
Do some real good monitoring, real time, and post on a grey hat security board that it can't be hacked.
Somebody'll figure out how to do it just to say they did, and if you're monitoring properly, then you'll know how they did it. Then you can fix the problem.
Although this probably works better for bigger companies, that have less cred in black hat circles....
"City hall" in German is "Rathaus" Kinda explains a few things......
We'll point out any flaws for ya ;)
[FUCK BETA]
I've had the privilege of meeting Jeremiah Grossman at a security conference. I'd recommend reading several of his white papers and then decide if you want to call his company up. I doubt they are cheap, but the best rarely is.
http://www.whitehatsec.com/home/index.html
Securicon worked on my bank's, sorry - financial institution's online finance application. Good enough for my money- good enough for me.
> ... web application ... extremely secure ...
You contradict yourself.
Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
http://www.sectheory.com/ One of the most well-known web security experts, Robert Hansen, and network security expert, James Flom, founded the company a few years back. They're a small and specialized firm with more $billion/year clients than employees.
They also run http://sla.ckers.org the main forum where web security researchers congregate.
They mainly focus on blackbox testing of web app and network vulnerabilities, but also do whitebox source code audits. They do both manual testing and automated scanners. I think they're above average in pricing, but their employees are made up of experts rather than re-adapted programmers that the bulk of larger firms hire.
Full Disclosure: Friend's company has used them for the past year, and i'm a member of sla.ckers like most every webappsec researcher.
Financial institutions don't necessarily have the best possible security, there are plenty of precedents to prove otherwise. They may or may not but I wouldn't use that as a standard. (I've worked in the "moving other peoples' money sector for several years so this is an insider's perspective.)
I have nothing against Securicon, they may be great, but I'd try to find out who handles testing for the credit bureaus. I've used a web interface with one of them and it was at least secure enough to take effort to use. We had to import a certificate to verify the client and register the IP with them, which seems like a good start.
B) Eliminate all the stupid users. This is frowned upon by society.
When I think about secure web applications I think Stefan Esser, he has a security company:
http://www.sektioneins.de/
Most of the information security consulting companies are relatively small shops (5-50 people is common) with a handful of customers each. There is also a number of security testing divisions attached to some of the largest all-around international consulting firms, but they are relied upon primarily for regulatory compliance needs (meaning: "let's get this over with as soon as possible"), and they usually combine lack of any identifiable infosec talent with outrageous pricing.
So, with small companies serving non-overlapping groups of customers, it is almost guaranteed that no Slashdotter (of whom only a small fraction deals with information security!) can offer a meaningful, first-hand comparison of the services of key players in the field - and even if this is incorrect, there is absolutely no guarantee that the person telling you about their experiences would in fact have a sufficiently advanced understanding of computer security to make the comparison meaningful.
Unless you have enough in-house expertise and set up some controlled experiments, it's very difficult to tell if a positive outcome of a security audit means you are in the clear, or simply that the auditors are incompetent. To make things worse, even observing that auditor A identified n bugs in the setting in which auditor B identified n+m does not really tell you much, unless you truly understand their impact in the context of your services, or the reporting granularity and thresholds used.
What else? Many of the small companies may rely on PR alone, and some might be outright dishonest, for example releasing inflated security research, or simply astroturfing on Slashdot or elsewhere. And some might be run by people with actual credibility in the industry, but running subpar businesses because of poor project or team management skills. Just because they present at Black Hat, post to BUGTRAQ, or have a book published, does not mean a lot (but is a positive factor, of course).
So there's no easy solution. What you need to do is not to rely on Slashdot to give you answers, and instead, collect all the names you can easily find on the web (and in responses to this thread), then spend several days going through all the freely available primers on web application security... and come up with a decent RFQ that inquiries all the companies about their credentials, methodologies, the tools they use, sample reports they provide, and so forth. Ask technical questions, and expect them to be answered by technical people. You then need to set your bullsh*t detector to overdrive, and be wary of vague, dismissive, or nonsensical responses that look as if written by a marketing drone.
Based on this information, you then need to make the call which one would suit your business best. Good luck. It's not easy.
The best people I know that perform these services aren't working for any security testing company.
phpBB 3.0 was audited by SektionEins, a german company that focuses on PHP security. Stefan Esser is also part of this company, and he is most likely one of the best PHP security experts.
Is a startup, but formed with employees with no less than ten years of experience in pen-testing.
Be sure that, whoever does your testing, your company's "policies and procedures" are both satisfactory and being reliably followed by all employees. Social engineering is quicker, cheaper, easier, and more difficult to detect and track, generally speaking, than hacking in through some obscure loophole in the application.
Your people need to know what not to do, what not to say, and whom not to talk to, or your iron-clad web app may as well be tin foil. A top-notch security analysis company should be able to help make sure those bases are covered, too.
Coffee is my drug of choice.
Speaking as someone with much experience with third party pen testing ...
Be wary of believing that this kind of service can provide peace of mind. Consider the number of man hours (or years) that your webapp has taken to develop - these vendors will claim that they can carry out an effective audit in a matter of several days. Yes, they may well find issues in this time but the depth that they are able to go and the methods used are pretty trivial.
There is no silver bullet for this problem. A pen testing company will run all the usual tools against your site that you could do yourself with little training. Application testing your software is a different matter entirely and speaking as someone who has dealt with a number of very reputable companies in this area I will tell you: having an application pen testing service tell you your app is secure and believing it is just asking to be hacked.
Train your developers to think secure from the start. Send them on SAN's courses. Give them something to hack so they can see how easy it is.
Look at source code scanning systems such as Core Impact or Fortify. Again not a silver bullet but they will help.
Yes, absolutely, get your app pen tested. By more than one company. Good tick in the compliance box but protection? Nope.
I worked for KPMG for ten years performing penetration tests. For the last several of those years I ran the teams and worked with clients to scope the work.
The following is true for most big companies that have country or regional teams and for any team for that matter: there are good teams and bad teams. You're going to have to talk to the techies to get comfortable with them.
The bad companies will use a lot of automated methods. For example they'll tell you that they have a software product that does the pen test and then they manually review the output. There are a few of those 'pen test in a box' companies out there you should avoid. Or they'll say they know what they're doing and actually run nmap, nessus and then do some poor manual testing.
What you need is someone who will make use of some automated tools but spend a lot of time manually testing the web application. This means they are manually testings various inputs to see what they can do and they have to know what they're talking about. I don't mind companies that rely on products like WebInspect or AppScan, but that should only be a tool and not the main show. Make sure you ask to talk to the techies and not just the salesguy so you can ask them how a web app should be secured and what kind of things you should look for to get your app in shape before a pen test begins. What often distinguished us was that we could give free advice to help improve security even before our testing began.
Besides some of the teams at KPMG and the other big firms (again, you have to vet each team) I would also suggest Corsaire which is a smaller company.
In terms of scoping work you should ask for an infrastructure test and an application test. If you are really unsure of things you should ask for them also to review your architecture and things like your firewall rules. Expect to pay a minimum of 5k USD but depending on how big your app is you may get as high as 30k. After htat you can look at regular scanning but there are a lot of companies that offer that more cheaply (like Qualys)
Ask whoever you choose to first run an automated scan against the site so you can fix those things before they do their work. Give yourself a few weeks for that. You really really don't want them to test your site before it is ready. Otherwise it might be a waste of money. I now work for another global company but on the other side of the table: I use services from companies like KPMG. I'm still impressed with the service they and some of the biggies give us. They find things that I haven't even had a chance to hear about yet. And occasionally we'll have a really crappy B team that misses things I've already found in our apps but didn't tell them. That tends to happen more from some of our smaller vendors who magically got on our approved tester list.
No one is going to be able to truly validate that your site is immune to compromise, etc. The threat that gets you may as yet not even exist. But if (when) you do fumble your client's "highly sensitive" information, you can always go after the guy that you paid to tell you that you'd finally cracked the code on building a perfectly secure site.
If you have lots of money, hire two firms: one of the boutique firms that can do the job and the big company mentioned above. You need insurance...
If it really needs to be super-secure did you ever think that the Internet may not be the right network? Does anybody out there remember when there were other networks in the world?
they are really good
Cenzic will test your web security for you.
www.cenzic.com
Check out their "Click to Secure" service.
They are first class and will tell you if your if things are secure.
Regards,
Doug
Just a tip: ask multiple companies to do the first audit. You'll likely get very different results, go from there.
<shameless plug>
I do pen tests for clients (both government and banking) via my company. I wouldn't call myself the best, but there's always something that can be found.
</shameless plug>
This sig is intentionally left blank
As someone that used to lead teams that did this kind of work, too little attention was paid to the thoroughness of the testing by both client and testers. "What is supposed to be tested", "what was tested" and "what were the results" were simple questions that I would ask but testers and client were only interested in "what vulnerabilities were found". I could have 100 interesting findings but have only tested 1 out of 10 components that were supposed to be tested - and that was fine! You should make a list of what you want tested and demand that you be told what was tested and what were the results of those tests.
Netcraft do this sort of thing. Plus you get to say "It's secure - Netcraft confirms it".
Unless you've got control over the clients, you've already lost the battle to protect the application. It doesn't matter what sort of authentication and encryption you have if your user visited some shady porn site two hours ago and got RAT'd.
I can't answer the question about recommending a testing company. However, I can tell you that you will need to have your app re-tested at regular intervals, as well as after any change (no matter how small) to the code or infrastructure. You need to build that into your plan and budget, and you need to have the tests run against your staging/QA setup so that you can catch problems before they hit the production site, as well as against your production environment.
--Paul
I've found Corsaire to be good. http://www.corsaire.com/en/
I've used SecureState for the past few years, I often get multiple third parties just to make sure they are keeping up but have been extremely satisfied. We outsource our web application testing in QA to them and they have been outstanding..
I'm in charge of a web application that must be extremely secure. Users will be submitting highly sensitive information to each other using the site. Security must be world-class.
And your way of approaching this problem is to "ask slashdot"? Ye gods.
... and then they built the supercollider.
My company uses a company called Security Innovation. We have used them several times for various engagements including an embedded system, Unix applications, and our latest Web based interface into the system. Each time they have found some serious security vulnerabilities that our customers would have been extremely alarmed if they had been found by their own pentesting teams (which they use for each release of our product). Highly recommended!
http://www.securityinnovation.com/
I do not think anyone can recommend the "best" company as the criteria for "best" depend on your business needs.
That being said, I would recommend sending a request for proposal (or call for tender, I never know the correct name for this) to 5 companies with local offices so you can meet the ethical hackers if needed. This is good to avoid relying on a bunch of "not so white hackers" with little knowledge of collateral damages and potential impact of the pentest on the information system.
Make sure the intruders do not rely on automated tools. I have seen Eeye/ISS reports labelled as actual pentests reports, sold at pentest prices. A good pentest on a 3/3 application requires at least 8-10 days from my experience. These figures should be adapted to the complexity of the infrastructure of course.
I would also ask for information regarding
- system tests vs application tests. The latter cannot be automated to be effective, but both are necessary for a pentest to be meaningful
- the pentest methodology (do they have anything set or do they do it "as they feel" for each project),
- audit trails gathering (all traffic between the pentest lab and your information system should be archived)
- alert processes (what should they do if a critical vulnerability is discovered) and so on
Many companies with little knowledge of professional penetration testing sell intrusion services, from my point of view it is your job to select the best one, nobody on Slashdot can do that for you.
It is rare that I would get into a discussion like this, since it often will devolve into the equivalent of a perl vs python war, or at a minimum, vendors will try to sell their warez.
When hiring a company for an application penetration test, I like to look towards those who are actively involved in research within the security community, and hire people that contribute to the community heavily as well. For example, does the firm have people on staff that discovered and disclosed new vulnerabilities? Does the company have people that bring new ways of attacking to market, and what tools do they make available to the community.
Quite often this rules out a number of the large companies, like the big auditing firms. (Whilst in some cases they have intelligent people, I have met an awful lot of tool monkies that worked for these companies.
Some companies that I would usually consider include NGS software (David & Mark Litfield ... known for a number of Oracle vulnerabily disclosures), Immunity Security (Dave Aitel, Kostya Kortchinsky, and Nico. These guys are very well known in the community, and are the brains behind Canvas, Spike Proxy, and others...), Security-assessment.com (Paul Craig, released iKat for kiosk hacking.), and finally, insomnia security (Brett Moore, this guy knows heaps about heaps.).
Which of these are the best will depend on the particular assessment you are having performed, and what the goal of the test is. These guys are damn smart, and very professional. Go to their sites and see what they do, and then talk to references. In the end you have to be comfortable with the company.
I hope this helps..
Cy
You could try iSEC Partners. It's pretty much what they do. Training, design review, pen-testing, and so-on.
Your first stop should be OWASP, the Open Web Application Security Project. You'll find there many companies that are experts in web application security, including tools and guides to get a handle on web app sec. I'd also recommend becoming familiar with the OWASP Top 10 http://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project
I work as a penetration tester doing this kind of test. I have to say with my company it's really luck of the draw regarding which tester you get; We have a few excellent, highly skilled guys and some people who should not be doing this kind of work.
I know for certain that some people would miss critical issues that better people would find, I have seen it happen. BTW expect to pay at least £1000 or $1500 a day for a consultant's time whether they are any good or not, and a test for a reasonable size web app is going to take at least 5-7 days.
Also, discuss the format of the report in advance; I know that myself and other testers often prefer to provide a simple list of issues identified rather than fancy reports with tables and graphs that take much longer to produce but add little real value IMO. You may get more time spent on actual hacking if you go for a more barebone report format.
Hire a consultant that has passed the CREST Web Application Certification Examination. Having done it myself I'd say it's a pretty good test, I'd rate anyone who passes it as a competent webapp pentester. See www.crest-approved.org. CREST is a UK based scheme at the moment.
"Consulting Search Fee Saved, $5000. Giving people the chance to earn Informative Karma, Priceless!"
My first Journal Entry ever, in 8 years! http://slashdot.org/journal/365947/aphelion-scifi-fantasy-horror-poetry-webzine
In my experience, the less expensive boutiques normally do a poor job. Go with the firms that are large and service some of the largest companies in the world. As others have said, the testers doing the work are most important. The larger firms can normally pay the top dollar for the top testers.
How are you Gentlemen?
Let's start off with a big huge truism, and that is that pen-testing is statistically a losing game. In that sense it's pure snake oil: You buy a service, and maybe it finds something, maybe it doesn't. You still know squat about what some cracker kiddie might find, or whether there will be 0-day exploits down the road that some kiddie managed to run against your site.
And it gets worse. Your webapp viewed from a distance, including the middleware, the server software, the OS, even the firmware and the hardware, and we haven't even started about any backends and so on, is quite a huge stack, or collection of stacks, some of them wobbly, and any single component may leak and that might turn out to be big enough to give access to the crown jewels.
But you do know you consider yourself a target. One leak is all it takes.
So, what are you paying a pentest firm for, exactly? What do you want from your external consultants? To find holes in your application so you can patch them? That just tells you there used to be a hole. What then, fail to find holes?
HA HA HA HA
OmniTI:
http://omniti.com/does/web-application-security
You list many good things that you did. But if your developers were not trained in secure coding, you are going to be shocked at how much a good web app testing company is going to find.
I own and run a small information security consulting company myself and we also do web application security. But since you are looking for the "best", I have to recommend Cigital, which is easily one of the best in this space. When I was working for other people, I used to hire them, and now even though we are competitors, I have no problem recommending them when someone asks for the best.
WhiteHat security is also pretty good, so is FoundStone. I am sure there are other very good companies too.
One last recommendation: stay away from the Big-N audit firms.
www.securestate.com - They are very good and their people contributed the entire FastTrack toolset that is part of BackTrack.
Yes, what about banks? Good question. Look at credit cards. The system is quite impressively leaky. They reverse the transaction as far as they can, stiffing the merchant if they can. Solved, next!
They're not looking for actual solutions; they know their systems are leaky. All they want is the ability to demonstrate having excercised due diligence. Money is eminently replaceable, and as long as the losses can be shuffled under the carpet by accounting, all bankers need is to keep up appearances of trustability, actual trustability is entirely optional.
But that doesn't help squat against loss of irreplaceable goods. Information migth be such a good. Look at the paranoia the NSA employs around the far too many secrets it keeps. Indoctrination before and after. Compartimentalisation, need to know, keyword clearances, making sure you hardly know what you're doing most of the time. Any clandestine operation of more than a few persons does the same using the well-known cell structure. The less you know, the less likely you are to leak something important.
Now apply that to software. You'll quickly find most software was written for convenience, not integrity. And I'm not even talking about keeping it easy to use for the the users. Quite the challenge, no?
Good job you are working on it, though you have your work cut out for you. If you're the outfit I think you are, then your organisation already has my contact key.
I know it's sometimes a pain and can take time, but you might want to consider putting out an RFP for an application test. Depending on the size of your company and procurement policies, you might be required to put the job out for bid anyway. It also gives you a good idea about what's out there. Let me warn you however, that if you're only looking to satisfy an audit requirement, you're probably wasting your time, as you'll probably be force to choose the lowest bid, which will most likely provide the least value in the long run, not to mention a false sense of security. There are many things to include in the RFP, but the major points that come to mind at the moment are as follows: - Company information (size, qualifications, location (important if testing is on-site), personnel bios, insurance, etc.) - Technical Methodology (as detailed as possible) - Tools used - Reporting (make them include a sample) - References (3 professional references seem to be the norm, which should be past clients) There are many places one can place the RFP, such as magazines (SC, Infosec.), listserves (e.g., securityfocus.com) and of course you can always pick the top-10 replies to your query on slashdot and send the RFP to them. You should get at least 5-6 responses.
Security Team at Cornell University are amazing IMO. Talk to them.
Security must be world-class.
"WORLD CLASS: A phrase used by provincial cities and second-rate entertainment and sports events, as well as a wide variety of insecure individuals, to assert that they are not provincial or second-rate, thereby confirming that they are."
--unknown
Unexpect the expected!
One of the best security experts I know is Solar Designer; he once did a security audit for us and it helped a lot. Here's the link: http://www.openwall.com/services/
Core Security is the best I know about.
http://www.coresecurity.com/
Expensive, but the best.
"We believe we've built site in such a way that minimizes security risks and we've implemented numerous policies and procedures company-wide to increase security."
This is by far your biggest security threat that you should worry about before any penetration testing. The idea that implementing policies and procedures will somehow increase security. Relying on humans to adhere correctly to policies and procedures as a security measure is probably a sure fire way to end up with a security leak.
You should be working to secure your app. so you don't need these policies and procedures. This will likely end up with a system that is a little more frustrating for your users, but if security is your number 1 criteria for the application then that's the compromise.
You may think to yourself, well, anyone not following procedures risks the sack so they'll have to follow them, but this misses the fact that your CEO is as likely to ignore policies and procedures as anyone and he certainly aint going to sack himself, no, it'll be all your fault. It's also worth pointing out that even the threat of the sack isn't enough to convince some people to pay attention and realise the rules are there for a reason not to mention the fact that to be fair on users, sometimes it's just human nature to forget to do something.
This is unfortunate but after doing a dozen plus of these I haven't found one company better than the other. The big 8 consulting firms surprisingly have done a better job than other smaller security focused firms however here are some things that I have learned in practice.
1. I haven't had a good pen job done that has cost less than $75k.
2. Reserve the right of refusal and no payment for subpar work. I've had someone try and charge $30k for sending out two "experts" to run a scan of nessus and ISS on our network for 3 days.
3. If you think they are all wet and just out of college they probably are.
4. If you don't learn something new....they probably didn't do a good enough job.
5. Do it twice a year in a general sense, plus a special focus of critical applications during their initial release and then maybe a year or two later again to focus on critical apps.
Sad but true rule #1 still applies.
Static code analysis by the best. In my experience automated testing misses 1/3 of vulnerabilities. Source review is the way to go.
They will go onsite if the client needs that service as well. Request the resumes of the analysts.
www.aspectsecurity.com
IBM AppScan (AppScan or Enterprise) was developed by Watchfire and than acquired by IBM for the sole purpose of Web application security testing. When you bring in a group of security testers they usually use a tool to help them with the automated testing, but if you get this than you can have your developers do their own testing as well as have security consultants use the data to preform their own pen testing.
IBM AppScan (AppScan or Enterprise) was developed by Watchfire and than acquired by IBM for the sole purpose of Web application security testing. When you bring in a group of security testers they usually use a tool to help them with the automated testing, but if you get this than you can have your developers do their own testing as well as have security consultants use the data to preform their own pen testing. (Posted it under anonymous without thinking)
never worked with them, but seem to be an industry leader. http://www.matasano.com/services/deploysafe/web/ might be what you are looking for
I thought Foundstone was one of the Best. Didn't their employees create OWASP anyway.
I would agree with the post in for Jeremiah Grossman at WhiteHat Security. Jeremiah and his team do great work in this space, and their research is top notch.
I also wanted to offer our company's services as well. InGuardians is also well known in the industry. Our team frequently presents at major security conferences, both commercial (BlackHat, SANS, ...) and community (Defcon, Toorcon, Shmoocon, ...). In fact, I'm sure if you spoke with Jeremiah, he would give us a shining recommendation as well. And honestly, I'd say that you'd be hard pressed at finding anyone else in the industry that does better work than InGuardians and WhiteHat Security. You really can't go wrong with either choice.
Full Disclosure. I am a Senior Security Analyst for InGuardians that specializes in network and web app pentests. Another one of our Senior Analysts is Kevin Johnson, who is the author and lead instructor for the SANS 542 "Web App Penetration Testing and Ethical Hacking" course.
http://www.sans.org/security08/description.php?tid=1722
Here is something else to help you out, regardless of who you go with. Kevin and I have a few OSS community projects, one that you'd probably be interested in is our live pentest CD called "Samurai-WTF". It is a live Linux environment that has been pre-configured with the best open source and free tools for testing and attacking websites. Feel free to go download a copy from our website. It works great running from any of the virtual machine products out there, and also works great if you burn it to a DVD. Once you get it running, the login is "samurai" with the password "samurai".
http://samurai.inguardians.com/
I'd love to draft up a proposal for Kevin and I to pentest your website and the network it is sitting on. Please feel free to email me at justin (at) inguardians.com to set up a time to talk about your needs in more detail.
Check out our website if you would like to learn more about our company, the other services we offer, and the other members of our team.
http://inguardians.com/
isec partners
whitehat (already mentioned)
inguardians (already mentioned)
matasano
Stay away from the big auditing firms like KPMG and Deloitte.
I've worked with each of those companies, and can say that the smart people are at the boutique security firms, the MBAs with their MCSE are at the big auditing firms.
Well, the problem with big and famous ones is that they may have very good people working for them, but in practice you never get the senior people in your audit, but some junior ones that really don't know that much. They'll run a set of tools and provide a report that covers whatever the tools found. Some value, true.
That's how we got one major deal in the banking sector (good word of mouth by just showing how good we were in conferences etc.). We did the job very well, and now are getting a ton of follow-up deals (they gave us an extraordinary reference too, they made it even more flattering than the one we suggested!). They used to be happy with a well known consultancy agency, but...
Oh well, once we go above just a few good guys we'll probably start sucking too. C'est la vie.
I'll add another plug in the parade of shameless plugs.
My employer is Fortify Software; we make a static analyzer that performs good quality cross tier analysis of popular languages like Java, JavaScript and PHP.
In addition to the static analysis, we also have a QA assistance tool that uses Java bytecode instrumentation to follow taints dynamically through the application and correlate with the static findings.
Doug
Take off every 'sig' !!
We use Mandalorian for our penetration testing and I highly recommend them. They're very good with web apps and web services and aren't too expensive.