DSS/HIPPA/SOX Unalterable Audit Logs?

analogrithems writes "Recently I was asked by one of the suits in my company to come up with a method to comply with the new PCI DSS policy that requires companies to have write once, read many logs. In short the requirement is for a secure method to make sure that once a log is written it can never be deleted or changed. So far I've only been able to find commercial and hardware-based solutions. I would prefer to use an open source solution. I know this policy is already part of HIPPA and soon to be part of SOX. It seems like there ought to be a way to do this with cryptography and checksums to ensure authenticity. Has anyone seen or developed such a solution? Or how have you made compliance?"

Comments

Question... What's to stop

[Anonymous Coward]

What's to stop someone from reading in one of the WORM tapes, modifying the log file data and then writing it back to another blank WORM tape and claiming it's the unaltered tape?

Do all logs have to be encrypted and signed with a seperate super sekret key/cert before being recorded?

Re:Question... What's to stop

[beyondkaoru]

i don't know much about the laws/regulations in question here, but yes, there isn't anything stopping someone from making a new 'worm' storage device and claiming it to be new unless there's a third party who will remember identifying information on the data.

if i really wanted to make sure my archives weren't tampered with, i'd bring my data (in whatever medium, the 'worm' thing wouldn't be necessary to ensure non-tampering, though it'd be good for storage purposes) to a trustworthy and hopefully vaguely computer savvy notary. then, i and the notary would hash the data, write up a form that says "i, name here, declare that on this date data with this hash value, some hexadecimal, was filed. signed, signature".

storage aside, this means that for someone to tamper with it they'd have to either bribe/coerce/kill people who saw this form (difficult) or reverse a cryptographic hash (even more difficult). so, pick a good notary (or submit the hash value to the gov maybe?) and a good hash function (like a larger sha or whirlpool) and i think you're tamperproof.

of course, i don't know the regulation so i don't know if this matches the needs of the article.

Re:Question... What's to stop

[beyondkaoru]

well, by being 'tamper evident' as you say, you are in fact tamper proof, so long as the data is well stored (and tape drives, cd or dvd jukeboxes, etc can do a good job of this). an iron-clad 'tamper-proof' box is not, in fact, tamper proof if one can simply substitute another iron-clad box in its place. this is the reason that only having a dvd jukebox wouldn't be secure (though again, i don't know what the regulation's requirements are). a nefarious company could simply juxtapose dvd's. remember that the ones who would be interested in tampering with the data are also the same folks who are storing and maintaining it.

use a line printer

[1u3hr]

Connect a line printer to mirror the log file as it's created. Use continuous fanfold paper. Get staff to sign and date first and last page.

Lawyers love paper. (A magistate once asked me if a printout I presented in a case was an "original email". I said it was as close as you could get.) In all likelihood, no one will ever refer to it, so don't worry about that it might take 10 minutes to find a page. Once a month, ship it to a secure storage. For real paranoia, have two printers making two simultaneous copies.

Re:use a line printer

[pla]

Wouldn't work in Australia, compliance penalties apply if you can't dredge up the data within a specified period of time.

The hardcopy just "proves" that no one has tampered with the normal (and searcheable) logs. You would never actually use the hardcopy, you would just have it in storage somewhere. You would still use the standard system (or application) logs for day-to-day auditing.

And if/when you find yourself in court, answering the question "can you prove this as the real logged data?", you can confidently say "yes" and have the dumptruck full of printouts pull up in front of the courthouse.


Though, personally, I don't see the problem with using a non-rewriteable DVD. Yes, each log entry would take up a full sector, but that still gives you, per disc, 2.3 million log entries of up to 2k each. That should last at least a few days (more likely, a few months) at any company small enough to balk at the cost of an OEM solution.

How odd

[Anonymous Coward]

It looks like that commercial offering is a piece of hardware with a network API (web service) that you write to which doesn't provide any network APIs for deleting or modifying records. Presumably it has a read-only view of the data.

Now, assuming that they use harddrives, we all know that someone could extract mount the file system and change records. They could hide their tracks by recalculating cryptographic hashes. So it's simply a lie to say that the only way to modify or indeed delete the data the data is through physically destroying the hardware.

However having a dedicated hardware for a write-only, read-many system is a good idea, but I can't imagine that this would necessarily be more complex than a locked down box running some web scripting language that appends http post data to a log, or a database.

If there is more to it then please could someone elabourate on what exactly one must do?

Re:How odd

[Sobrique]

Sort of, but not quite. A Centerra is a Content Addressed Storage thingy. Which basically means it's file identifiers are md5 sums. It's a multi node thingummy too, which replicates stuff about. Is it impossible to tamper with? Well, no, nothing is. But it's pretty hard. Simply because it has implicit 'tamper detection'.

The API is also geared up so you can choose what 'mode' you want it to operate in. In the most secure mode, the API and OS built in (it's Suse based) won't let you delete anything. Which, basically means you have to pull out the individual drives that 'clip' is stored on, to trash it. Data will be gone, which isn't great, but ... well, pretty much impossible to prevent for any system. Modifying data retroactively though, is much much harder - recreating the right md5sum is a non trivial task. Impossible? Perhaps not, but ... well, EMC have done quite well with 'selling' this product in a 'it will meet your compliance needs' which is considered good enough for our auditors.

We have 'financial organisation' regulations, for retention of emails, and a Centerra is what we settled on as the solution.

Re:Syslog

[Whiney Mac Fanboy]

Besides logging locally to disk, also add a line to /etc/syslog.conf to log to a remote machine.

If syslog can write to a remote machine, then a compromised syslog can overwrite a file on a remote machine. I suspect thats not even remotely close to enough read-only.

As others have suggested, print your logs on a line printer.

Sometimes, the old ideas are the best

[polymath69]

Nothing is an unalterable as a line printer which lacks reverse-vertical-paging capability. Just make sure it doesn't run out of ink or paper.

Line Printer won't scale

[jsimon12]

If you only had a single machine or maybe even a couple lightly used boxes a printer might work. But even that would be near impossible to go back and sort through and if you ever ran out of paper or ink you would be SOL. And trust me the last thing you want is to to be SOL when it comes to SOX. If you have the money don't half ass an audit solution.

Re:Line Printer won't scale

[mrsteveman1]

Yes, so you have a law requiring accountability for companies, most of which break the law constantly and again when they try to cover it up, and it gets to be too much of a burden on these completely dishonest companies, yes lets just repeal the law.

Moron

Syslog + chattr

[ethzer0]

I use syslog-ng to relay information from several different datacenters to a centralized and secure location hosting all of the syslog information. Each DC has its own syslog-ng system acting as the local relay, transporting syslog information from local clients using TCP over a VPN to the centralized host. The logs are written on the central syslog sever organized by on date and hostname, and each file that is created is then assigned an 'append-only' bit using chattr. It works really well.

The Linux way

[professorfalcon]

tail -f thelog.txt >> /mnt/cdrom/thelog.txt

FreeBSD to the rescue

[stox]

FeeBSD supports append only files via the chflags command.

Re:Syslog

[pedestrian crossing]

As others have suggested, print your logs on a line printer.

But that doesn't really scale very well, and then you have the problem of dealing with retention/storage requirements.

Re:Syslog

[Anonymous Coward]

Sure. but what keeps you from making a copy of the CD-ROM with certain info deleted or altered, and putting that in the archives instead of the original?

This request is impossible.

[The Master Control P]

Given sufficient resources, time, and dedication, ANY log can be altered.

If the "unalterable" log is maintained in software, it's a comparatively simple matter of hoisting it up on a VM. Since we're presumably talking about white-collar crime, it's a fair bet they have or can get root access to the machine to install the VM and rootkit to hide it. At that point, the CEO can do anything and the system can't fight back. Capture passwords of people logging in, alter data, you name it.

A hardware system would be more robust, but still vulnerable. I imagine the most likely attack vector would be Man in the Middle - Just take over the box that guards/drives the logger machine.

Re:This request is impossible.

[hazem]

Given sufficient resources, time, and dedication, ANY log can be altered.

What really matters is if there is any case law that actually interprets the laws and provides standards for due diligence.

The law might say "unalterable" or "lasting indefinitely" we all know there are practical physical limits - given enough time, anything is alterable and nothing lasts forever. We could come up with outrageous methods like using satellites with lasers to etch logs on the surface of the moon, but there's not much point in going to such expense until the case law suggests that is what is needed to be "due diligence".

Until you have that case law each organization will simply have to do a cost x risk analysis and determine how much they're willing to risk in order to "do enough" to keep out of trouble with the auditors. Something like:

chance of being audited x cost of failing an audit vs. cost of maintaining an "unalterable" logging system

Then you just wait to see what organizations get skewered and adjust your analysis and practices accordingly... and just hope you're not the first to get audited. OR... we can work on a way to etch logs onto spheres of aluminum that are then launched into orbit so they can be read with a telescope. Though that will probably lead to an increase in insurance premiums.

Re:Syslog

[Anonymous Coward]

a compromised syslog can overwrite a file on a remote machine

Not with a properly configured syslog. You're not supposed to just use a remote logfile, but a remote logging daemon (RFC 3164). That way you can add entries to a remote log, but not change or delete any (unless you make the logfile directly accessible over the network, which I wouldn't recommend).

Re:Syslog

[cerberusss]

If syslog can write to a remote machine, then a compromised syslog can overwrite a file on a remote machine.

I don't think so; the receiving syslog machine will be "add-only" and won't have rights to overwrite files. Of course, you can print your logs and that would be a good second defense. But searching through printed logs manually is a pain in the butt.

Physical Destruction

[unsubscribe]

In short the requirement is for a secure method to make sure that once a log is written it can never be deleted or changed.
Although it is possible to prevent logs from being modified (using write-once media) or undetectably tampered with (using crypto, possibly with a TPM module for the ultra-paranoid), any log can be 'deleted' by physically destroying the device/media on which it is stored.

Dont skimp...

[pjr.cc]

Seriously, when it comes to legal requirements, do not skimp!

Go for something that is guarentee'd to fulfill your legal compliance requirements.

Yeah, optical media is great for WORM, but you dont want something your going to have to manage day to day. The legal req's of sox and so forth are beyond that of traditional optical drives in terms of life span in any case. Do not go with optical for compliance unless its something specifically designed for compliance (Again, thats $$$).

As someone suggested, centera is a good option - but all the storage vendors have good options (from emc, netapp, hds, sun, falconstor, mimosa the list is endless) and they'll all tell you how theirs is better than anyone else (and why). At the end of the day, you want a compliance solution with someone's stamp on it, and a throat you can cut when it goes wrong.

If your absolutely determined to go the compliance route on OSS - go with ext3cow (www.ext3cow.com) IMHO, a fully versioning COW fs with a non-erasable past and the best OSS solution for the job - backup on to optical if you like, but dont make optical your only option. If it only had policy-based management (i.e. snapshot whenever user X or group y writes a file) rather then crontab'ing its snapshot agent it would almost be perfect for a start-point solution for compliance. It has a big benifit along with it though, you can show users how to get files "from yesterday".

Keep in mind, WORM means policy-based write-once, not necessarily immutable storage! And almost every compliance worm product out there depends on that fact.

Don't Build Your Own Device

[Interfacer]

I work for a big pharma company as a sysadmin, and we have to abide by similar rules and laws. Our data recording and data logging has to be proven to be unalterable.

Go with a commercial solution unless you want to battle with the QA and Validation departments for haf a year. And even if you would get the go-ahead (unlikely) you'd get in a hell of a lot of trouble during an audit because auditors a) don't know your solution and b) they will quickly see that it is not certified.

There are specified requirements (don't know the names and numbers by heart) that your solution has to proven to fulfill, and certified by some external party.
Just saying 'Yeah but I know it cannot be altered because it is syslog / ' will not cut it.

And non-compliance can eend up costing your company millions if not hundreds of millions.
Open source or home grown has it's place, but in a regulated environment you go with commercial for certain things because that is the only option where you get certification with your device / software.

Re:Syslog

[arth1]

Why, just tee it with one copy going to the remote syslogd on the remote secured machine, and the other copy being stored locally for easy access. Use the local insecure copy for day-to-day read-many access, and the remote secure copy for archival and legal purposes.

Yes, this will be write-twice, read-many, but that's much easier to implement than true WORM.

we did it ourselves in the end

[instantiator]

as it was quickest. writing: 1. append latest entry as plaintext 2. sign [signature of previous entry + plaintext of latest entry], and append as signature of latest entry

This is audit, not perfection

[nicolaiplum]

First of all, this is a requirement to satisfy an audit. It would be nice if whatever solution you come up with is actually good, but you only have to satisfy the auditors.
Auditors want processes and records to raise the barrier to someone doing something wrong or unrecorded. They know systems aren't perfect. Nerds all go "but someone could just make a change some other way so this system is no use". If it raises the barrier high enough, it is useful. All accounting books can be cooked, the point of auditing is to make it harder (and to make it require more people).
Some write-once methods are more accepted by auditors than others, because they have seen them before. Of course you can rewrite a WORM disc's contents onto another disc and edit the contents on the way, but most auditors think that is sufficiently difficult that they think WORM media is OK. And you can write several copies and send them to different places, then the person wanting to edit the log has to get to all the discs.
The commercial solutions like those from EMC and Network Appliance aren't foolproof. You could take all the discs and edit the data on the discs if you want. It's harder for you to do that than simply "su root; vi logfile" because you have to work at a lower level. All the EMC or NetApp software is promising the auditors is that it would be hard work to do it undetectably and that is what they are looking for.
An Open Source solution which has no API to change data once it is written would work the same as the EMC or NetApp, but if you are the first to use it then you have to persuade your auditors (and possibly a court down the line) that it really doesn't have any such API.
Usually Open Source is good for your business because you have control of it, can change it, and can see how it works.
Auditable logs are about you not having control of them, not being able to change them.
The easiest way to get your auditors to agree your logs meet the audit requirements is to use some solution that they have seen before. Buy a Netapp (I won't, personally, recommend EMC) or buy a WORM drive. DVD-R is WORM, it's cheap.

Re:Write them to a DVD jukebox

[itwerx]

Wish there was a way to mod an entire thread Offtopic.

From TFA:
      "So far I've only been able to find commercial and hardware-based solutions. I would prefer to use an open source solution."
FP:
      Write them to a DVD jukebox

      Hmm, yeah, I'm sure there's dozens of open source hardware designs for DVD jukeboxes - I'll have that Googled by the time my soldering iron heats up!
      Only on Slashdot can the First Post get modded to +5 for a reply which is so completely Offtopic it's Funny, obviously written by somebody who didn't RTFA, followed by dozens of posts debating the merits of the "answer" without noticing that it's not what the submitter is looking for!!
      And we wonder why the rest of the world thinks IT people can't communicate...

Re:Syslog

[Albanach]

Sure. but what keeps you from making a copy of the CD-ROM with certain info deleted or altered, and putting that in the archives instead of the original?

You miss the point. The submitter is not asking for a way to produce a secure write once logging system, he's asking for a way to make a write once logging system that is compliant with the legislation he is operating under.

While we may be able to see endless faults with any proposed solution that doesn't matter. He just needs to implement one that covers him and the suits above him should they be audited.

Re:Write them to a DVD jukebox

[Anonymous Coward]

How did this troll get modded insightful. I guess that the processor, motherboard, firmware, cd, dvd, controllers and everything else has to be open source. Linux can talk to a dvd jukebox, write your app on top of that.