Slashdot Mirror
DSS/HIPPA/SOX Unalterable Audit Logs?
analogrithems writes "Recently I was asked by one of the suits in my company to come up with a method to comply with the new PCI DSS policy that requires companies to have write once, read many logs. In short the requirement is for a secure method to make sure that once a log is written it can never be deleted or changed. So far I've only been able to find commercial and hardware-based solutions. I would prefer to use an open source solution. I know this policy is already part of HIPPA and soon to be part of SOX. It seems like there ought to be a way to do this with cryptography and checksums to ensure authenticity. Has anyone seen or developed such a solution? Or how have you made compliance?"
Optical media are great for write once, read many.
Lawyers love paper. (A magistate once asked me if a printout I presented in a case was an "original email". I said it was as close as you could get.) In all likelihood, no one will ever refer to it, so don't worry about that it might take 10 minutes to find a page. Once a month, ship it to a secure storage. For real paranoia, have two printers making two simultaneous copies.
I preface this by saying I know I will get flamed for not recommending Open Source but SOX is a Federal mandate (Federal equals PMITA)
EMC's Centera is my personal favorite, it isn't cheap but it does exactly what you need and is auditable and recognized by all the third party audit compmaies as well as the Federal government.
I have worked in IT for 15 years and 5 of those have been for a LARGE financial institution. When it comes to audit and SOX go with something standard, tested and commercial, unless you want to spend the next 6 months explaining to your auditors how your homegrown solution works and then the next 6 months building something new that your auditors do understand (or worse, like losing your job).
Besides logging locally to disk, also add a line to /etc/syslog.conf to log to a remote machine.
If syslog can write to a remote machine, then a compromised syslog can overwrite a file on a remote machine. I suspect thats not even remotely close to enough read-only.
As others have suggested, print your logs on a line printer.
There are shills on slashdot. Apparently, I'm one of them.
Cryptography, digital signatures, and checksums can only take you so far. They can detect tampering pretty easily. However, crypto can't prevent someone from deleting a file, although by checksumming or signing a whole bunch of files you could at least detect deletion of one of them. Ultimately, if you really want permanence, you need to write it out (as an above poster suggested) to some sort of write-once media. CD-Rs or DVD-Rs would obviously fit the bill here, although one can indeed delete a CD-R by simply throwing it out, of course.
:)
Another cheap write-only medium is paper; I suppose you could purchase a laser printer (or even a line printer), and have it spit out the logs as they occur. If you kept the printer in a locked transparent box, nobody but people with the keys would have access to the output.
You could burn the logs onto PROMs as well, that's pretty permanent
Anything on magnetic or flash media can be erased or tampered with somehow, unless the drive controller hardware itself prohibited overwriting existing data. Even then you're relying on someone not being able to replace the drive controller or take the drive apart and diddle the platters/flash chips directly (although I suppose a decent amount of epoxy could thwart this). Any software-based solution can be tampered with in theory. One hacker favorite (which may be a legend or not) is that people used to get root on other people's boxes and then replace their copy of PGP with an instrumented copy. Thus, even the encryption software became compromised.
For compliance, though, I'm not sure what kind of oversight you have to have. At the end of the day, somebody has to be trusted with these logs, and that person would almost assuredly have the power to destroy them, or at least portions of them.
I use syslog-ng to relay information from several different datacenters to a centralized and secure location hosting all of the syslog information. Each DC has its own syslog-ng system acting as the local relay, transporting syslog information from local clients using TCP over a VPN to the centralized host. The logs are written on the central syslog sever organized by on date and hostname, and each file that is created is then assigned an 'append-only' bit using chattr. It works really well.
FeeBSD supports append only files via the chflags command.
"To those who are overly cautious, everything is impossible. "
At USENIX/EVT06 last year a team from the University of Iowa presented a cheap one-way data cable you could make with off-the-shelf parts from Radio Shack. Total cost is about $5 (for bulk, maybe $10 if you're buying single units) and it is provably, auditably, one-way. It was originally developed for electronic voting, to allow for counting computers to communicate with webservers that post election results. An attacker compromising the webserver cannot attack the counting computer, because there is literally no return path.
It works with very high reliability up to about 9600 baud.
You may be able to use this to your benefit. Have an isolated system air-gapped from the rest of the network which listens for log events on a one-way data cable. While you're no longer guaranteed to be safe (since if a logging PC is compromised, an attacker could send compromised data to the syslog PC and perhaps cause some sort of mayhem), but the lack of a return path makes interactive attacks infeasible.
ObDisclosure: I am a graduate student at UI and know the guy who invented the data cable, although I am not associated with the gadget.
But that doesn't really scale very well, and then you have the problem of dealing with retention/storage requirements.
A house divided against itself cannot stand.
That's all you need
Opus: the Swiss army knife of audio codec
I would dump it in GIT or the likes.... any change of it will be recorded ;)
Seriously, many version controlling systems already contain the data integrity and authenticity checks that you need
Given sufficient resources, time, and dedication, ANY log can be altered.
If the "unalterable" log is maintained in software, it's a comparatively simple matter of hoisting it up on a VM. Since we're presumably talking about white-collar crime, it's a fair bet they have or can get root access to the machine to install the VM and rootkit to hide it. At that point, the CEO can do anything and the system can't fight back. Capture passwords of people logging in, alter data, you name it.
A hardware system would be more robust, but still vulnerable. I imagine the most likely attack vector would be Man in the Middle - Just take over the box that guards/drives the logger machine.
Some of the work I do may require something like this, so I'm considering implementing Guy Fawkes over syslog.
http://www.cl.cam.ac.uk/~rja14/Papers/fawkes.pdf
From the paper:
6.2 Tamper-evident audit trails
It is a well known problem that an intruder can often acquire root status by using well known operating system weaknesses, and then alter the audit and log information to remove the evidence of the intrusion. In order to prevent this, some Unix systems require that operations on log and audit data other than reads and appends be carried out from the system console. Others do not, and it could be of value to arrange alternative tamper-evidence mechanisms.
A first idea might be to simply sign and timestamp the audit trail at regular intervals, but this is not sufficient as a root intruder will be able to obtain the private signing key and retrospectively forge audit records. In addition, the intervals would have to be small (of the order of a second, or even less) and the computation of RSA or DSA signatures at this frequency could impose a noticeable system overhead.
In this application, the Guy Fawkes protocol appears well suited because of the low computational overhead (two hash function computations per signature) and the fact that all secrets are transient; this second's secret codeword is no use in forging a signature of a second ago.
Not with a properly configured syslog. You're not supposed to just use a remote logfile, but a remote logging daemon (RFC 3164). That way you can add entries to a remote log, but not change or delete any (unless you make the logfile directly accessible over the network, which I wouldn't recommend).
I honestly don't know about DSS or SOX, but I have had plenty of fun with HIPAA.
Unalterable logs as a matter of compliance does not mean "absolutely unalterable under any circumstances". There should be no way for an end user to modify audit trails. There should be no preconceived way for an administrator to alter audit trails - i.e. no utilities for doing so. That does not mean that an admin can't go directly into the DB and alter the data from behind the application layer.
Under every circumstance when I have run into audit logs involving HIPAA compliance, they have been written by an application directly into a SQL database (oracle, ms sql, informix, and one time db2). It used to be that they were written in a fairly easy to decipher format within a single text column on a per record basis - which made for a fairly-difficult-to-alter audit trail because within that easy to decipher format were non-printable characters that you would at least have to know to look for them. With current implimentations, however, the records are stored in a separate table with a many-to-one relationship with the audit-required records, in varchar fields, as plain text - much easier to alter or get rid of single entries. There is still a level of obfuscation as far as table names and column names but thats really a side effect of other things that are going on.
These systems have been reviewed by auditors and certified as compliant. In the older system, there was no application interface to delete audit records. In the newer system, there is an application interface to delete records in any given application table - and therefore there is one for the audit tables as well. Admin level access is required to delete or alter the records, though.
Personally, I would expect more as far as HIPAA compliance goes - from both a customer standpoint and an auditor standpoint. My experience (and it is pretty extensive across several high profile enterprises) - is that the customer will demand a better system only when the auditors demand a better system. I haven't run into an auditor yet who has even given more than a casual glance at the 'back door' scenario. I suppose it's because there is no true way to keep things absolutely secure and application level audit log security is only one layer of the onion.
Before you get too far into an overly complex and potentially expensive solution, talk with your auditors about the requirements for your specific scenarios. They've seen it before and can tell you exactly what they are looking for from an audit compliance standpoint. They are usually pretty easy to work with and open with their knowledge.
Seriously, when it comes to legal requirements, do not skimp!
Go for something that is guarentee'd to fulfill your legal compliance requirements.
Yeah, optical media is great for WORM, but you dont want something your going to have to manage day to day. The legal req's of sox and so forth are beyond that of traditional optical drives in terms of life span in any case. Do not go with optical for compliance unless its something specifically designed for compliance (Again, thats $$$).
As someone suggested, centera is a good option - but all the storage vendors have good options (from emc, netapp, hds, sun, falconstor, mimosa the list is endless) and they'll all tell you how theirs is better than anyone else (and why). At the end of the day, you want a compliance solution with someone's stamp on it, and a throat you can cut when it goes wrong.
If your absolutely determined to go the compliance route on OSS - go with ext3cow (www.ext3cow.com) IMHO, a fully versioning COW fs with a non-erasable past and the best OSS solution for the job - backup on to optical if you like, but dont make optical your only option. If it only had policy-based management (i.e. snapshot whenever user X or group y writes a file) rather then crontab'ing its snapshot agent it would almost be perfect for a start-point solution for compliance. It has a big benifit along with it though, you can show users how to get files "from yesterday".
Keep in mind, WORM means policy-based write-once, not necessarily immutable storage! And almost every compliance worm product out there depends on that fact.
"Little does he know, but there is no 'I' in 'Idiot'!"
I work for a big pharma company as a sysadmin, and we have to abide by similar rules and laws. Our data recording and data logging has to be proven to be unalterable.
Go with a commercial solution unless you want to battle with the QA and Validation departments for haf a year. And even if you would get the go-ahead (unlikely) you'd get in a hell of a lot of trouble during an audit because auditors a) don't know your solution and b) they will quickly see that it is not certified.
There are specified requirements (don't know the names and numbers by heart) that your solution has to proven to fulfill, and certified by some external party.
Just saying 'Yeah but I know it cannot be altered because it is syslog / ' will not cut it.
And non-compliance can eend up costing your company millions if not hundreds of millions.
Open source or home grown has it's place, but in a regulated environment you go with commercial for certain things because that is the only option where you get certification with your device / software.
ext3cow was written with compliance in mind (i.e. with an untouchable past), and so its AFAIK the ONLY solution that can fit in compliance (keeping in mind that this only covers part of compliance). svn, git, cvs - im sorry, but thats just a non-solution for compliance. It also gives you no-mess management with a very easy interface to make sure you are being compliant (this is important, and its something YOU dont have to be involved in, your lawyers can "look at the past" to make sure "discovery" is going to be consistent).
... which has never been proven or certified by anyone".
The second thing is, compliance is (ridiculously) complex - the compliance vendors have spent many hours with lawyers getting it together, they know the requirements and they know they fullfill them - this is important. It also means their solutions come with an implicit warranty - "hey, your using netapp worm, we know it works" as apposed to "what software is that? how do you know it works?". At the end of the day a lawyer is going to either go "well i cant argue with the compliance solution" when your with a well-known or "your honor, the defendant is using
Compliance is the only time i will say to someone - "get a throat to cut", get a solution you know works, written by people who know what they are doing and its all because compliance req's were written by lawyers for lawyers (i.e. scum) and so their scum is going to make you have to act like scum.
The API is also geared up so you can choose what 'mode' you want it to operate in. In the most secure mode, the API and OS built in (it's Suse based) won't let you delete anything. Which, basically means you have to pull out the individual drives that 'clip' is stored on, to trash it. Data will be gone, which isn't great, but ... well, pretty much impossible to prevent for any system. Modifying data retroactively though, is much much harder - recreating the right md5sum is a non trivial task. Impossible? Perhaps not, but ... well, EMC have done quite well with 'selling' this product in a 'it will meet your compliance needs' which is considered good enough for our auditors.
We have 'financial organisation' regulations, for retention of emails, and a Centerra is what we settled on as the solution.
So you find it easier to kill people than to run computer programs... Remind me not to get on your shit list. :p
Experience teaches only the teachable. -AH
Centerras don't count as the original post, of a 'cheap solution'. They're not all that expensive by 'enterprise standards' but that's ... well not quite the same as 'affordable for most people'.
Also, our data centre is under fairly intensive scrutiny and control of physical access. My employer and customer are well aware that physical access means all bets are off, so in order to get physical access you need escorting, and authorization in advance, including documentation of what you're changing, why, and which grid squares in the datacentre you need access to.
I and the rest of my team are admins on this Centerra don't get access to the datacentre. If we have a need to enter, then we can fill in the paperwork and do so, but ... well, we're based 100 miles away. Most 'hands on' is done by someone else.
Now, combine that with the fact that each 'clip' (file) is stored 4 times, on 4 separate physical devices (2 of each, on 2 different sites) it would require ... well quite a few people to be complicit to even be able to destroy (or tamper with) data, physically. And a hell of a lot more to do so without leaving great big footprints all over the place screaming to the world what you've done.
I think you'd need 2 people on each site (one to actually tamper, and one to 'not notice' as he was escorting), plus an admin person offsite to identify which drives need 'doing', on both sites, and to mess with the 'self healing' replication so that one site didn't just restore the other. (You'd have to be fairly quick on the drives too, as soon as one goes down, the healing starts to replicate to other 'spare' drives).
And then you'd need some other people to mess with the entry logs to site, CCTV footage, change authorization....
You'd have to be pretty damn serious to pull that off. I mean, it's not even a case of some pointy haired one seeing their career on the line, and demanding immediate sabotage.
That's easy; feed the paper coming out of the printer through some sort of OCR machine.
Take life easy: one bit at a time.
Let me first state that I'm a PCI Qualified Security Assessor. That means I am certified by PCI to perform audits and report back to banks whether or not a company is compliant or not. In other words, consider me authoritative on this matter.
When dealing with any PCI requirement the most important thing to think about is the INTENT. Is the intent of the logging requirements in section 10 of the PCI DSS to prevent anyone, anywhere, from EVER being able to modify log files? No! The intent is to prevent a compromised system from altering its own log files--hiding the fact that it has been compromised. As long as your logging solution handles this situation effectively you really don't have anything to worry about.
In my role as auditor I would never fail a syslog host just because it was writing to a standard ext3 volume. I *would* fault a company if their logging solution was poorly configured (insecure: say, running telnetd) or was write-accessible by the same admins that send all their log data to it (unless they were a small company--if you only have one or two admins there's only so much separation of privilege you can get away with). I'd also have problems with a syslog host that wasn't backing itself up on a regular basis (90 days online, 3 year archive).
If I were you I'd be more concerned with your logging system meeting the other requirements of the PCI DSS. If it is inherently insecure or fails to implement proper access controls (say, shared root account) who cares how the logging solution is configured?
Remember: Intent is everything. If in doubt, call your acquirer (i.e. your bank). They're the ones who ultimately have to decide whether or not your implementation is good enough anyway. The auditor just writes a report--the bank has to sign off on it.
-Riskable
"Those who choose proprietary software will pay for their decision!"