DSS/HIPPA/SOX Unalterable Audit Logs?

analogrithems writes "Recently I was asked by one of the suits in my company to come up with a method to comply with the new PCI DSS policy that requires companies to have write once, read many logs. In short the requirement is for a secure method to make sure that once a log is written it can never be deleted or changed. So far I've only been able to find commercial and hardware-based solutions. I would prefer to use an open source solution. I know this policy is already part of HIPPA and soon to be part of SOX. It seems like there ought to be a way to do this with cryptography and checksums to ensure authenticity. Has anyone seen or developed such a solution? Or how have you made compliance?"

Comments

use a line printer

[1u3hr]

Connect a line printer to mirror the log file as it's created. Use continuous fanfold paper. Get staff to sign and date first and last page.

Lawyers love paper. (A magistate once asked me if a printout I presented in a case was an "original email". I said it was as close as you could get.) In all likelihood, no one will ever refer to it, so don't worry about that it might take 10 minutes to find a page. Once a month, ship it to a secure storage. For real paranoia, have two printers making two simultaneous copies.

Go with commercial hardware solution

[jsimon12]

I preface this by saying I know I will get flamed for not recommending Open Source but SOX is a Federal mandate (Federal equals PMITA)

EMC's Centera is my personal favorite, it isn't cheap but it does exactly what you need and is auditable and recognized by all the third party audit compmaies as well as the Federal government.

I have worked in IT for 15 years and 5 of those have been for a LARGE financial institution. When it comes to audit and SOX go with something standard, tested and commercial, unless you want to spend the next 6 months explaining to your auditors how your homegrown solution works and then the next 6 months building something new that your auditors do understand (or worse, like losing your job).

Re:Go with commercial hardware solution

[feepness]

unless you want to spend the next 6 months explaining to your auditors how your homegrown solution works and then the next 6 months building something new that your auditors do understand (or worse, like losing your job). I dunno, I can lose my job WAY faster than 6 months.

Re:Syslog

[Whiney Mac Fanboy]

Besides logging locally to disk, also add a line to /etc/syslog.conf to log to a remote machine.

If syslog can write to a remote machine, then a compromised syslog can overwrite a file on a remote machine. I suspect thats not even remotely close to enough read-only.

As others have suggested, print your logs on a line printer.

Syslog + chattr

[ethzer0]

I use syslog-ng to relay information from several different datacenters to a centralized and secure location hosting all of the syslog information. Each DC has its own syslog-ng system acting as the local relay, transporting syslog information from local clients using TCP over a VPN to the centralized host. The logs are written on the central syslog sever organized by on date and hostname, and each file that is created is then assigned an 'append-only' bit using chattr. It works really well.

One-way data cable

[rjh]

At USENIX/EVT06 last year a team from the University of Iowa presented a cheap one-way data cable you could make with off-the-shelf parts from Radio Shack. Total cost is about $5 (for bulk, maybe $10 if you're buying single units) and it is provably, auditably, one-way. It was originally developed for electronic voting, to allow for counting computers to communicate with webservers that post election results. An attacker compromising the webserver cannot attack the counting computer, because there is literally no return path.

It works with very high reliability up to about 9600 baud.

You may be able to use this to your benefit. Have an isolated system air-gapped from the rest of the network which listens for log events on a one-way data cable. While you're no longer guaranteed to be safe (since if a logging PC is compromised, an attacker could send compromised data to the syslog PC and perhaps cause some sort of mayhem), but the lack of a return path makes interactive attacks infeasible.

ObDisclosure: I am a graduate student at UI and know the guy who invented the data cable, although I am not associated with the gadget.

Write-Only Memory

[jmv]

That's all you need

Guy Fawkes Protocol

[LilBlackKittie]

Some of the work I do may require something like this, so I'm considering implementing Guy Fawkes over syslog.

http://www.cl.cam.ac.uk/~rja14/Papers/fawkes.pdf

From the paper:

6.2 Tamper-evident audit trails

It is a well known problem that an intruder can often acquire root status by using well known operating system weaknesses, and then alter the audit and log information to remove the evidence of the intrusion. In order to prevent this, some Unix systems require that operations on log and audit data other than reads and appends be carried out from the system console. Others do not, and it could be of value to arrange alternative tamper-evidence mechanisms.

A first idea might be to simply sign and timestamp the audit trail at regular intervals, but this is not sufficient as a root intruder will be able to obtain the private signing key and retrospectively forge audit records. In addition, the intervals would have to be small (of the order of a second, or even less) and the computation of RSA or DSA signatures at this frequency could impose a noticeable system overhead.

In this application, the Guy Fawkes protocol appears well suited because of the low computational overhead (two hash function computations per signature) and the fact that all secrets are transient; this second's secret codeword is no use in forging a signature of a second ago.

Re:Syslog

[Anonymous Coward]

a compromised syslog can overwrite a file on a remote machine

Not with a properly configured syslog. You're not supposed to just use a remote logfile, but a remote logging daemon (RFC 3164). That way you can add entries to a remote log, but not change or delete any (unless you make the logfile directly accessible over the network, which I wouldn't recommend).

From Experience

[Evets]

I honestly don't know about DSS or SOX, but I have had plenty of fun with HIPAA.

Unalterable logs as a matter of compliance does not mean "absolutely unalterable under any circumstances". There should be no way for an end user to modify audit trails. There should be no preconceived way for an administrator to alter audit trails - i.e. no utilities for doing so. That does not mean that an admin can't go directly into the DB and alter the data from behind the application layer.

Under every circumstance when I have run into audit logs involving HIPAA compliance, they have been written by an application directly into a SQL database (oracle, ms sql, informix, and one time db2). It used to be that they were written in a fairly easy to decipher format within a single text column on a per record basis - which made for a fairly-difficult-to-alter audit trail because within that easy to decipher format were non-printable characters that you would at least have to know to look for them. With current implimentations, however, the records are stored in a separate table with a many-to-one relationship with the audit-required records, in varchar fields, as plain text - much easier to alter or get rid of single entries. There is still a level of obfuscation as far as table names and column names but thats really a side effect of other things that are going on.

These systems have been reviewed by auditors and certified as compliant. In the older system, there was no application interface to delete audit records. In the newer system, there is an application interface to delete records in any given application table - and therefore there is one for the audit tables as well. Admin level access is required to delete or alter the records, though.

Personally, I would expect more as far as HIPAA compliance goes - from both a customer standpoint and an auditor standpoint. My experience (and it is pretty extensive across several high profile enterprises) - is that the customer will demand a better system only when the auditors demand a better system. I haven't run into an auditor yet who has even given more than a casual glance at the 'back door' scenario. I suppose it's because there is no true way to keep things absolutely secure and application level audit log security is only one layer of the onion.

Before you get too far into an overly complex and potentially expensive solution, talk with your auditors about the requirements for your specific scenarios. They've seen it before and can tell you exactly what they are looking for from an audit compliance standpoint. They are usually pretty easy to work with and open with their knowledge.

Re:Syslog

[dkf]

You're not supposed to just use a remote logfile, but a remote logging daemon. Another thing you can do is to send the logging messages over a non-IP connection (e.g. a serial line) so that even a standard network failure won't disrupt the logging and a hacked machine will continue to generate a track-able log. And the last I heard, you can't unsend bits sent down a serial line.

Re:Write them to a DVD jukebox

[arivanov]

Not quite.

They are not very good at tasks which involve writing a lot in small increments like a log. The sector size is quite big so if you guarantee that each log entry has finished physically on disc without caching till the sector is full the disc will be eaten in no time.

You probably need a custom writer/reader (most normal ones cannot alter sector size) and custom formatted media along with something different from isofs. Not rocket science really, but definitely beyond the scope of DIY.

Re:Write them to a DVD jukebox

[//rhi]

I always thought that WORM stood for "Write Once, Read Maybe"
//rhi - Enjoy the American Dream - You have to be asleep to believe it.

Re:How odd

[Sobrique]

I should add:

Centerras don't count as the original post, of a 'cheap solution'. They're not all that expensive by 'enterprise standards' but that's ... well not quite the same as 'affordable for most people'.

Also, our data centre is under fairly intensive scrutiny and control of physical access. My employer and customer are well aware that physical access means all bets are off, so in order to get physical access you need escorting, and authorization in advance, including documentation of what you're changing, why, and which grid squares in the datacentre you need access to.

I and the rest of my team are admins on this Centerra don't get access to the datacentre. If we have a need to enter, then we can fill in the paperwork and do so, but ... well, we're based 100 miles away. Most 'hands on' is done by someone else.

Now, combine that with the fact that each 'clip' (file) is stored 4 times, on 4 separate physical devices (2 of each, on 2 different sites) it would require ... well quite a few people to be complicit to even be able to destroy (or tamper with) data, physically. And a hell of a lot more to do so without leaving great big footprints all over the place screaming to the world what you've done.

I think you'd need 2 people on each site (one to actually tamper, and one to 'not notice' as he was escorting), plus an admin person offsite to identify which drives need 'doing', on both sites, and to mess with the 'self healing' replication so that one site didn't just restore the other. (You'd have to be fairly quick on the drives too, as soon as one goes down, the healing starts to replicate to other 'spare' drives).

And then you'd need some other people to mess with the entry logs to site, CCTV footage, change authorization....

You'd have to be pretty damn serious to pull that off. I mean, it's not even a case of some pointy haired one seeing their career on the line, and demanding immediate sabotage.

Re:Write them to a DVD jukebox

[jabuzz]

Or you could just use a DLT/lTO drive with WORM media. Works just fine for appending, no special software needed. Admitedly the drives are not cheap, but it is an easy solution. In fact the WORM media for DLT/LTO where developed specifically for this sort of application.

Re:Write them to a DVD jukebox

[ajs]

Optical is the right choice here, but you need to understand the PCI requirements and their most common interpretation VERY clearly. What you will probably end up with is something like this:

• Logs are written over the network (e.g. syslog)
  
• Logging host, which is locked down, and has no access from the infrastructure that it's performing logging for other than the incoming log data itself.
  
• Logging host writes the logs locally to files which are marked as append-only by the OS (Linux can do this)
  
• The logs are then written periodically (e.g. once per hour) to optical media.
  
• Add redundant logging hosts to taste (3 is a nice number for validation purposes).
  

Re:Syslog

[kars]

That's easy; feed the paper coming out of the printer through some sort of OCR machine.

Re:Write them to a DVD jukebox

[alcourt]

Append only files have not been required in my experience. What is required is that there be no ability to overwrite a previously written file by the team that is sending the log data. This can be done a number of ways, but the easiest method is to transmit the data in a way that the server chooses the filename, not the client. Add a date string into the filename and you can (with a few other details I've worked at but am here waving a wand at) avoid the problem.

syslog works for most data, but not all. Linux is one of the only Unix based systems that puts sulog through syslog. The failed logins log is much more difficult, as is the wtmp data. wtmp data is especially annoying as it is one of the only ways to semi-reliably record both login and logout regardless of login type (including ssh), and can't really handle real time data streaming. The other annoying item is the command line history of all commands with EUID 0. I'm hoping to hear some news soon on a solution to that problem, but it is really difficult, especially since a lot of SAs become root via `sudo -s` or `su` (as opposed to `su -`, which would not modify their HISTFILE variable. Many root shells do not support direct sending of HISTFILE over the network.

As to writing periodically to a optical media, I wouldn't worry quite so much about that. I would instead worry more about the encrypting all that security data while in network transit. (Sorry, can't recall if that is a firm requirement of PCIDSS 1.1 or not). Unfortunately, this makes use of syslog a less trivial solution. Authenticity is also an issue to be concerned with. How do you know that the event that got inserted into the log really came from that box, and not some random other server? Traditionally, syslog has not concerned itself with such issues, but a PCI system may care a great deal.

Once the data is on the central logging host, it is already in a state that the author of the data (the SAs for the PCI impacted box) cannot modify it. That eliminates at least in the interpretation of PCI I've been working on, the need for writing to optical media. Immutable is not so much immutable by anyone, but immutable by the server in question.

The point of the central copy of the logs is so that modification on either side can be readily detected and investigated. But if you cannot trust your central log host to have an accurate copy of the logs because you are receiving log data from anyone who chooses to pretend they are your PCI impacted server, then your central log host does not give you as much value as it may seem. The audit requirements aren't just for making lives miserable, they usually have a valid point behind them.

When working with PCI, know which DSS you are on, 1.0 or 1.1. (I don't know the release schedule for the next PCIDSS.) The requirements do differ, as do even the interpretations. Reference https://www.pcisecuritystandards.org/ for the information.