Slashdot Mirror
Do Data Center Audits Mean Anything?
1sockchuck writes "Data center service providers often tout certifications such as SAS 70, SSAE 16 and SOC 2 as evidence that they meet lofty operational standards. But some of these certifications are based on self-defined standards, and the entire situation is confusing and frustrating to customers, according to one critic, who says data center shoppers are poorly served by the jumble of acronyms and standards. Do these certifications matter when users are seeking data center space? Should they?"
Now, if you get your hands at the detail reports, the audit result may actually tell you something, at least if the auditors are good. But the certifications pretty much only ensure minimal standards low enough to be meaningless.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
It's like when any of the US government agencies "audit" themselves for instance, The Federal Reserve. Remember that "audit" recently?
not entirely unlike MSCE, but less so.
without data center audits thousands of datacenters across the country would have to forego tiny wooden plaques with things like "SAS70 CERTIFIED!" and "SSAE 16 READY!"
and I as a sysadmin would have to stop making the joke, "SAS70? oh thats for when we change the motor oil in the cloud."
Good people go to bed earlier.
A feeling of having made the same mistake before: Deja Foobar
The fact of the matter is a lot of stupid certification acronyms were specifically designed to allow spenders to make decisions without being actually informed in any way about what they're spending their money on. That's actually the *point* here. The problem isn't the certifications, the problem is that to make an informed decision about which ISP should host your servers you shouldn't be the type of CTO who insists on using outlook express and ie6 still and can't even configure their own email client. You need to know bandwidth from ass-width.
Yeah, I paid the bribe. Now I am certified!
I'm a work at a somewhat large financial services company that provides customer information to various other large financial institutions (chase, wells fargo, capital one, amex, discover, just to name a few). We receive this customer information from pretty much everywhere - those self same banks, government agencies, credit card companies, universities. Basically, if you've ever had a loan or grant, credit card, bank account, paid a utility bill, child support or been in prison then we have that data. Your address, phone number, social security number, bank account information, etc.
The majority of this information is stored unencrypted on systems that are accessible to any employee, often with 777 permissions. While the majority of the systems are patched pretty regularly, many aren't. I recently had to convert over an old apache 1.3 server that hadn't been patched since 2006 - there's another similar server that is regularly used by outside contributors to drop off customer information.
We have customer facing IPlanet servers that haven't been patched since 2004 - the software isn't even under support anymore.
We have session recording software on our unix servers that is so ridiculously trivial to bypass that the company that sells it (centrify) should be ashamed to sell it.
Yet we've had PCI certification for 3 years, we've passed the SAS70 certification every time - they are rubber-stamps, nothing more.
Well, it certainly matter for regulation purpose. If you handle data that need to be covered under a specific standard (say, PCI), you'll seek out a certified data center. In this context, the certification isn't about security, it's about risk transfer. It's the provider who become liable if there's a breach if it can't show to have respected the standard properly.
Now as security references, they certainly have their problems. We can take solace in the thought that they help enforce the bare minimum at the very least. As a security professional, I would say their best benefit is how well they can be used as a big stick, "encouraging" management to perform necessary changes. It's a hard sell to convince an average manager to invest in security for the sake of security. But if there's a legal penalty associated with whatever standard must be put in place, as well as a big dollar sign attached to it, they'll suddenly start to listen. That's a language they understand.
I've always been amazed at things like SAS 70 which, as the poster states, is based on self-defined criteria. The most shocking part, if I recall correctly, is that the criteria are not publicly consumable! This is the worst part of it all and the key part which needs to change.
The problem is that they are trying to get certifications when what they really need are Achievements! Just ask Microsoft, it worked for the XBOX, they are throwing it into Visual Studio, it will work for data centers.
Include a Facebook Like button and a Twitter link and your done!
Just like when hiring a new employee, you look for certifications and credentials. When trying to separate the legitimate companies from the fly-by-nights, seeing audits every year going back 2, 3, 4 years can help verify that they've been around for a while. Datacenter space isn't cheap, and if you find a good deal you want to make sure that your server with all the company data on it is still there on monday morning, and not on a plane to China.
You put your money in an FDIC-insured bank account because it's registered with someone who's taken the task of keeping an eye on their registrants and staked their reputation on it. If you don't agree with that, I have an offshore bank account you can transfer some money in to for the International Bank Of Hadlock, we just opened yesterday, but we offer 3000% interest daily and don't keep money laundering records.
moox. for a new generation.
I'd say that data center audits, just like financial audits, are only as good as the auditor. If you're a big enough client, the auditor will say pretty much any thing you want:
http://en.wikipedia.org/wiki/Arthur_Andersen#Demise
But still, I look for the certifications to cover my butt. Of course, that's what all of these standards are about - just saying that you've implemented procedures to cover your butt. It doesn't matter whether or not the procedure actually does anything worthwhile or even if there are big gaping holes elsewhere. As long as you can say you've implemented it fully, then you're covered.
There is a number of problems with how data centers make these statements and what people interpret.
The main problem is that people say things like "SAS70 Certified". That is terribly bad wording. There is no such thing. The SAS70 (now SSAE16 or SOC1 report) is not a certification. There is no preset/predetermined criteria that is universal to all companies that receive such a report. Each report is specific to that particular company/data center. It's almost like saying I have a diploma as an independent study major.
The next thing is that these reports are not intended for public use. These are auditor-to-auditor reports. They are meant for the auditor for a company that uses said data center (or other service provider) to rely on and not need to audit the data center itself. That is why auditors review these reports to make sure it contains the provisions it's looking for. Otherwise, they're going to go in and audit the data center.
Companies that get such reports tend to use it as a marketing tool to show potential customers, when that isn't the purpose. To reduce some blame, I've known auditors guilty of telling data centers that they can do that so that they could convince the data center to pay for the service.
Also, SAS70 was designed to reflect controls at a service provider that impact or relate to the processing of financial data, which would have an effect on the financial statements that the auditor is reviewing. Most data centers don't process data (the customers that host stuff there do and they need the SAS70). However, over the years, people have convinced themselves that because the data physically resides at the data center, they impact the financial statements and so they should get a SAS70. This is however, not really true, since with good security controls around the data, the physical hosting of it won't materially misstate the financials. It was for this reason that the AICPA split the old SAS70 into 3 separate services: SOC 1 (SSAE16) which is what the old SAS70 was meant to be, SOC 2, SOC 3. The latter 2 are geared more toward data centers and technology firms that don't impact financial data.
The seals that are issued by the AICPA just state that you've had a report done. They do not speak to the content of the report. I could get a SOC report that just says "All employees are entitled to free breakfast". The auditor I hire will come in and test/verify that and then will sign-off saying that they agree. I now have such a report and can boast "SAS70 Certified" everywhere, which doesn't mean squat.
It only matters to the company itself, the company that uses their services (depending on context), and the auditors of the company that uses their services.
There is a number of problems with how data centers make these statements and what people interpret. The main problem is that people say things like "SAS70 Certified". That is terribly bad wording. There is no such thing. The SAS70 (now SSAE16 or SOC1 report) is not a certification. There is no preset/predetermined criteria that is universal to all companies that receive such a report. Each report is specific to that particular company/data center. It's almost like saying I have a diploma as an independent study major. The next thing is that these reports are not intended for public use. These are auditor-to-auditor reports. They are meant for the auditor for a company that uses said data center (or other service provider) to rely on and not need to audit the data center itself. That is why auditors review these reports to make sure it contains the provisions it's looking for. Otherwise, they're going to go in and audit the data center. Companies that get such reports tend to use it as a marketing tool to show potential customers, when that isn't the purpose. To reduce some blame, I've known auditors guilty of telling data centers that they can do that so that they could convince the data center to pay for the service. Also, SAS70 was designed to reflect controls at a service provider that impact or relate to the processing of financial data, which would have an effect on the financial statements that the auditor is reviewing. Most data centers don't process data (the customers that host stuff there do and they need the SAS70). However, over the years, people have convinced themselves that because the data physically resides at the data center, they impact the financial statements and so they should get a SAS70. This is however, not really true, since with good security controls around the data, the physical hosting of it won't materially misstate the financials. It was for this reason that the AICPA split the old SAS70 into 3 separate services: SOC 1 (SSAE16) which is what the old SAS70 was meant to be, SOC 2, SOC 3. The latter 2 are geared more toward data centers and technology firms that don't impact financial data. The seals that are issued by the AICPA just state that you've had a report done. They do not speak to the content of the report. I could get a SOC report that just says "All employees are entitled to free breakfast". The auditor I hire will come in and test/verify that and then will sign-off saying that they agree. I now have such a report and can boast "SAS70 Certified" everywhere, which doesn't mean squat. It only matters to the company itself, the company that uses their services (depending on context), and the auditors of the company that uses their services.
How much is PHB speak that what people who don't know that they are running come up with BS to make it look like they do.
Security and reliability are processes, they are not something you can do once and then forget about. So, yes, I would say that having regular audits are a useful thing. As far as whether these specific standards are useful, the facility we have most of our servers in we have been in since before their SAS 70 audit, and their procedures were good before, but there's a noticeable improvement after. Things like a man-trap with a live security person comparing you with your on-file photo before you enter the raised floor, 2-factor auth on all doors rather than just on the key doors, maintenance lock-outs displayed more prominently, EPOs installed (not a benefit to me, but they did put alarmed doors around the EPOs to prevent the common problems).
As far as it being "based on self-defined standards", I'm ok with that. I'm ok with the requirement being that they *HAVE* standards for certain things rather than dictating what exactly those standards are. One size does not fit all, but having standards for what you do, I have found in my own business, improves quality.
SAS70 and the new SSAE 16 require that the assessor and the reader both recognize the limitations and the scope of the work. Like many have said, its dependent on the quality of the auditor. This is why there are a "Big 4" in the audit world, their name carries weight to the quality of review. We will exclude the perceptions about the quality that may actually be provided. When I did SAS70 and similar audits, one of the first things i look at is the company that performed the review, the quality of the finished product (format, style, mistakes, sentence construction.) SAS70, HIPAA (HIPPA was when it was in draft) certified or similar "certified" is bs. No such thing exists. The controls selected for testing are decided by the hosting company, not by the reviewer. This requires the secondary auditor to evaluated the the scope of controls and quality of the controls tested against the controls they need to evaluate. Since many that receive a SAS70 are not actually the data processors, the assessment usually only provides some comfort in the effectiveness to the physical access and system availability (UPS, AC, redundancy).
rarely
Me:"So you don't care if the written procedure is a valid method to accomplish practical outcome, as long as I follow the procedure as written." Auditor: "Yes." That pretty much sums up their real world value.
Right here, pure gold: http://www.gartner.com/it/page.jsp?id=1400813
Read that 5 times, carefully, and then get your bosses to do the same. Seriously.
SAS70 is a *questionnaire* that the vendor completes, and then the auditors just go in and confirm that their answers are correct.
So I could say "we don't do backups" in my answer to the questionnaire, the auditors would verify that I didn't do backups, and I'd "complete" the SAS70 process (not a certification!) successfully.
It is the client that is resoponsible for reviewing the questionnaire and ensuring that the audited answers are sufficient for the needs of their business. That's called "vendor management" and is a core practice area in ITIL.
The presence of a certification unaccompanied by an invitation to come and look over the place yourself should be a pretty good warning.
Birds are not dinosaur descendants;birds are dinosaurs, for all useful meanings of "birds", "are" and "dinosaurs"
in my humble experience, especially on the continuity area, certs mean something if management assertions are really tested in real life like scenarios... a data center may have adequate UPS and 2 generators but it may not protect the "system" from pure human idiocity. So in a real life case, once, when power failed, UPS kicked in until generators started. However BOTH generators started and due to high power surge current protectors shut both down at the same time... what do you say to that?
Certainly at the worker bee level we all can agree that most certs are not worth the paper that they're printed on, however when you start looking at what the company needs and requires then, yes, the certs do provide some value.
Imagine signing a multi-million dollar contract with a data center. Part of the contract are clauses for things like QOS, DR, and a whole host of other very tiny details which are so, so important in the contract. Now, lets say that the data center goes down (fire for example), along with it your business. Now you're out major bucks, you will be turning to your lawyers and asking if you can sue for breach of contract. The lawyers are going to review the contract, see what the certs mean, review the certs and the data center's answers, and then tell you if you do or do not have a case.
And that's why the certs are important. Because they spell out exactly what the data center will do in case of [fill in the blank], and gives the data center coverage which is then enforced by your contract.
III.IIVIVIXIIVIVIIIVVIIIIXVIIIXIIIIIIIIVIIIIVVIII
DaCurryman has it right. SAS 70 (dead) and the new standards (SOC 1, SOC 2, SOC 3) are not certifications. SOC 1 (SSAE 16) is not intended to provide assurance over security. It is intended to provide financial statement auditors (not management of prospective customers) with an understanding of controls in place at a service organization that impact the financial reporting of their audit clients (the data center customers). None of the AICPA "standards" have anything to do with the "lofty operational standards" mentioned in the original post of this thread. The AICPA standards are standards for conducting the attestations, NOT standards for data center security and operations. If your intended audience is management of existing customers and of prospective customers, then SOC 2 and SOC 3 are infinitely better attestations (NOT CERTIFICATIONS!!!) to request. Many data centers are helpless at this point because existing customers all want SSAE 16 reports. They don't understand why, they just know that their auditors will ask for the report during the next audit cycle just like they asked for a SAS 70 report in years past. It is up to those on this board to educate your management and your customers about the differences and ensure you provide your customers with the correct report.
In a Type 1 audit, all the auditors look for is whether the company has policies/procedures/controls in effect to obtain the objectives of the company (whatever those may be)
In a Type 2 audit, the auditors will attempt to determine whether the policies and procedures in place are being followed. Whether the controls are effective in achieving the objectives that have been stated.
I work for a software company that recently went through a Type 2 audit. In our case most of what was looked at was our SDLC (software development life cycle) process, version control, etc. They went through our work ticket system & spent a week following more than a few tickets through the entire process: code check out, work produced, QA testing, user testing, peer review, code check in. They spent several weeks over a three month period driving our internal audit & software staff nuts.
Does it mean anything? From our point of view, yes. But, not only does the audit depend on the quality of the auditors, but on the quality & detail of those process & procedure documents that they are auditing.