I've been doing this (reviewing security and being reviewed myself) for a long time. For what it's worth, here' a few thoughts on getting your
money's worth:
What kind of review?
Basically, there are three types: penetration tests (ethical
hacking), assessments (white box technical reviews), and audits
(process/procedure reviews). These are very different from one another,
as (typically) are the firms who perform each type. Ira Winkler wrote a
good article
on this subject.
Although a pen test is sexy, you almost certainly want an assessment based on your description.
Know the goal.
Unfortunately, much of this market is driven by "Good
Housekeeping Seals of Approval" -- inexpensive rubber stamp reviews
designed to limit liability and make partners feel good (e.g., we followed the best practices
and even had 3rd party auditors, they just didn't find this hole). Unfortunately, this creates a disincentive to actually finding problems since that's not what the customer ordered.
If you're really concerned about your security, you want a
confidential report for your internal consumption that takes a good hard
look at your real security and is clear about all of the problems, even
less critical ones (though of course you want them prioritized). Stay
far away from "certification" oriented reviews.
Make it attorney-client work product.
If your organization is structured such that this works (and in this
case — a state agency — it may not be), it can be useful to
have the report be protected by attorney-client privilege, to manage the
legal liability caused by the findings in the report. You especially
want this if you follow the previous step and get a good, hard look.
Go independent.
This was already mentioned in another post, but bears repeating.
Don't get a review from a company whose primary business involves
selling anything other than security reviews. First, they often
consciously try to sell you their product (or service). Second, they
are generally unconsciously biased by their own efforts on their product
and are looking at problems from a more limited perspective. Same goes
from companies who resell network and security products for other
vendors, taking a cut of every sale. Get an independent review from
someone who's earning their keep based on their professional opinion,
not leveraging follow-on sales.
Also, look out for the one-two punch from auditing firms: a cheap initial pen test to prove how insecure you are (typically with lots of grandstanding to upper management), follwed by a really expensive audit where they actually make their money.
Hire the individuals, not the company.
There are good people at mediocre companies and vice versa. The
quality of the output depends most on who did the work and least on what
company employs them. Although the larger firms have more structure and
quality control, the odds of getting a great reviewer rather than a room
full of talking heads from a Big 5 are less.
This doesn't mean don't hire a Big 5, it means hire a specific team
from a reputable company.
If at all possible, make the hiring decision based on face-to-face
discussions with the actual team that will do the work, and ensure the
contract allows you to approve changes in the team. Look for people who
five or more years technical experience outside security before they
started doing security (e.g., was a hard-core sys admin for five years
before they started consulting others on systems security).
This also means evaluating potential firms like a job interview, to
some degree. The most effective, yet cooperative way to accomplish this
is to invite them over and start describing a couple of your problems
that you've already carefully considered. If the potential team rolls
up their sleeves and starts solving your problems — in the sales
call — with good, obviously experienced approaches, then they're
worth considering. If they only talk in broad generalities or don't
grasp issues that are widely understood, then they're not worth your
time.
On the other hand, ensure that they are bi-lingual. Not English and
Hindi, but Technical and Management. They need to be able to find
problems, propose practical solutions. Then they have to document this
in the report, so that the technical staff understands the problem and
solution well enough to fix it and the management team can grasp the
level of risk, cost of remediation, and gauge priorities.
Try to get a sanitized report from a job performed by the same team
for review. Evaluate whether you would be happy with those results and,
if so, ensure they know that you expect even better.
Be specific.
When developing the scope of work, be specific about what is and is
not included in the review. Don't accept a vague statement of work that
isn't clear which or how many systems will be reviewed, the structure of
the report, or other details. Ensure you know what you're paying for
and what will be performed.
Be prepared
Although you're overworked and have a hard enough time keeping up
with your day-to-day tasks, the results also depend on your preparation,
responsiveness, and organization. Have network diagrams, org charts,
and device/system configs ready for the reviewers. When they need more
information, get it too them in a timely fashion — it'll keep your
costs down and result in a more detailed report with fewer guesses on
the part of the reviewer.
Don't hold back.
Although it may be tempting to not tell them about things you know
are a problem to gauge how long it takes them to find the problem, this
approach is simply a waste of your own money during the review. If you
evaluated the team well before hiring them, tell them everything you
already know is a problem so they don't spend time rediscovering those
issues. Sure, they'll end up in the report even though you already knew
about them, but it'll again save money and result in a better more
detailed product.
More small reviews.
It's quite likely that you'll get better results getting several
smaller reviews from carefully chosen teams than one single large
review. This is especially true if you choose well rounded teams with
different backgrounds. While they should all be competent across the
board, if one team comes from an application development background
while another team comes from a system administration background,
they're likely to find different results.
Ultimately, it's your security.
Opinions vary. Accept the report as one person's opinion on how they
would prioritize the issues and fix them. After you receive the report,
review it and then prioritize the issues and develop fixes based on your
knowledge of the environment and business goals.
If you've done you're
homework, your prioritization and solutions will match those in the
report. If they clash, then figure out what went wrong an know to look for those indicators next time.
Slashdot Mirror
Basically, there are three types: penetration tests (ethical hacking), assessments (white box technical reviews), and audits (process/procedure reviews). These are very different from one another, as (typically) are the firms who perform each type. Ira Winkler wrote a good article on this subject.
Although a pen test is sexy, you almost certainly want an assessment based on your description.
Unfortunately, much of this market is driven by "Good Housekeeping Seals of Approval" -- inexpensive rubber stamp reviews designed to limit liability and make partners feel good (e.g., we followed the best practices and even had 3rd party auditors, they just didn't find this hole). Unfortunately, this creates a disincentive to actually finding problems since that's not what the customer ordered.
If you're really concerned about your security, you want a confidential report for your internal consumption that takes a good hard look at your real security and is clear about all of the problems, even less critical ones (though of course you want them prioritized). Stay far away from "certification" oriented reviews.
If your organization is structured such that this works (and in this case — a state agency — it may not be), it can be useful to have the report be protected by attorney-client privilege, to manage the legal liability caused by the findings in the report. You especially want this if you follow the previous step and get a good, hard look.
This was already mentioned in another post, but bears repeating. Don't get a review from a company whose primary business involves selling anything other than security reviews. First, they often consciously try to sell you their product (or service). Second, they are generally unconsciously biased by their own efforts on their product and are looking at problems from a more limited perspective. Same goes from companies who resell network and security products for other vendors, taking a cut of every sale. Get an independent review from someone who's earning their keep based on their professional opinion, not leveraging follow-on sales.
Also, look out for the one-two punch from auditing firms: a cheap initial pen test to prove how insecure you are (typically with lots of grandstanding to upper management), follwed by a really expensive audit where they actually make their money.
There are good people at mediocre companies and vice versa. The quality of the output depends most on who did the work and least on what company employs them. Although the larger firms have more structure and quality control, the odds of getting a great reviewer rather than a room full of talking heads from a Big 5 are less.
This doesn't mean don't hire a Big 5, it means hire a specific team from a reputable company.
If at all possible, make the hiring decision based on face-to-face discussions with the actual team that will do the work, and ensure the contract allows you to approve changes in the team. Look for people who five or more years technical experience outside security before they started doing security (e.g., was a hard-core sys admin for five years before they started consulting others on systems security).
This also means evaluating potential firms like a job interview, to some degree. The most effective, yet cooperative way to accomplish this is to invite them over and start describing a couple of your problems that you've already carefully considered. If the potential team rolls up their sleeves and starts solving your problems — in the sales call — with good, obviously experienced approaches, then they're worth considering. If they only talk in broad generalities or don't grasp issues that are widely understood, then they're not worth your time.
On the other hand, ensure that they are bi-lingual. Not English and Hindi, but Technical and Management. They need to be able to find problems, propose practical solutions. Then they have to document this in the report, so that the technical staff understands the problem and solution well enough to fix it and the management team can grasp the level of risk, cost of remediation, and gauge priorities.
Try to get a sanitized report from a job performed by the same team for review. Evaluate whether you would be happy with those results and, if so, ensure they know that you expect even better.
When developing the scope of work, be specific about what is and is not included in the review. Don't accept a vague statement of work that isn't clear which or how many systems will be reviewed, the structure of the report, or other details. Ensure you know what you're paying for and what will be performed.
Although you're overworked and have a hard enough time keeping up with your day-to-day tasks, the results also depend on your preparation, responsiveness, and organization. Have network diagrams, org charts, and device/system configs ready for the reviewers. When they need more information, get it too them in a timely fashion — it'll keep your costs down and result in a more detailed report with fewer guesses on the part of the reviewer.
Although it may be tempting to not tell them about things you know are a problem to gauge how long it takes them to find the problem, this approach is simply a waste of your own money during the review. If you evaluated the team well before hiring them, tell them everything you already know is a problem so they don't spend time rediscovering those issues. Sure, they'll end up in the report even though you already knew about them, but it'll again save money and result in a better more detailed product.
It's quite likely that you'll get better results getting several smaller reviews from carefully chosen teams than one single large review. This is especially true if you choose well rounded teams with different backgrounds. While they should all be competent across the board, if one team comes from an application development background while another team comes from a system administration background, they're likely to find different results.
Opinions vary. Accept the report as one person's opinion on how they would prioritize the issues and fix them. After you receive the report, review it and then prioritize the issues and develop fixes based on your knowledge of the environment and business goals.
If you've done you're homework, your prioritization and solutions will match those in the report. If they clash, then figure out what went wrong an know to look for those indicators next time.