DSS/HIPPA/SOX Unalterable Audit Logs?

analogrithems writes "Recently I was asked by one of the suits in my company to come up with a method to comply with the new PCI DSS policy that requires companies to have write once, read many logs. In short the requirement is for a secure method to make sure that once a log is written it can never be deleted or changed. So far I've only been able to find commercial and hardware-based solutions. I would prefer to use an open source solution. I know this policy is already part of HIPPA and soon to be part of SOX. It seems like there ought to be a way to do this with cryptography and checksums to ensure authenticity. Has anyone seen or developed such a solution? Or how have you made compliance?"

Comments

Write them to a DVD jukebox

[Anonymous Coward]

Optical media are great for write once, read many.

Re:Write them to a DVD jukebox

[arivanov]

Not quite.

They are not very good at tasks which involve writing a lot in small increments like a log. The sector size is quite big so if you guarantee that each log entry has finished physically on disc without caching till the sector is full the disc will be eaten in no time.

You probably need a custom writer/reader (most normal ones cannot alter sector size) and custom formatted media along with something different from isofs. Not rocket science really, but definitely beyond the scope of DIY.

Re:Write them to a DVD jukebox

[//rhi]

I always thought that WORM stood for "Write Once, Read Maybe"
//rhi - Enjoy the American Dream - You have to be asleep to believe it.

Re:Write them to a DVD jukebox

[jabuzz]

Or you could just use a DLT/lTO drive with WORM media. Works just fine for appending, no special software needed. Admitedly the drives are not cheap, but it is an easy solution. In fact the WORM media for DLT/LTO where developed specifically for this sort of application.

Re:Write them to a DVD jukebox

[Jah-Wren Ryel]

They are not very good at tasks which involve writing a lot in small increments like a log. The sector size is quite big so if you guarantee that each log entry has finished physically on disc without caching till the sector is full the disc will be eaten in no time. I seem to recall that DVD+R was designed to work around that problem. The thinking at the time was that people would used DVD+R media like they used VHS tapes, to record tv with the ability to pause and or stop/restart recording frequently. They wanted to avoid the inefficiency of CD-R and DVD-R which are very wasteful on start/stop record operations as you indicated.

I really can't dig up the link, it was years ago that I read this and google ain't cooperating right now, but I recall that whereas a recording pause could waste up to an entire track (once around the disc) with DVD-R, a DVD+R recorder would waste at most one sector (one the order of a few Kbytes).

Re:Write them to a DVD jukebox

[ajs]

Optical is the right choice here, but you need to understand the PCI requirements and their most common interpretation VERY clearly. What you will probably end up with is something like this:

• Logs are written over the network (e.g. syslog)
  
• Logging host, which is locked down, and has no access from the infrastructure that it's performing logging for other than the incoming log data itself.
  
• Logging host writes the logs locally to files which are marked as append-only by the OS (Linux can do this)
  
• The logs are then written periodically (e.g. once per hour) to optical media.
  
• Add redundant logging hosts to taste (3 is a nice number for validation purposes).
  

Re:Write them to a DVD jukebox

[netglen]

What about the old school method of dumping the log directly to a line printer?

Re:Write them to a DVD jukebox

[itwerx]

Wish there was a way to mod an entire thread Offtopic.

From TFA:
      "So far I've only been able to find commercial and hardware-based solutions. I would prefer to use an open source solution."
FP:
      Write them to a DVD jukebox

      Hmm, yeah, I'm sure there's dozens of open source hardware designs for DVD jukeboxes - I'll have that Googled by the time my soldering iron heats up!
      Only on Slashdot can the First Post get modded to +5 for a reply which is so completely Offtopic it's Funny, obviously written by somebody who didn't RTFA, followed by dozens of posts debating the merits of the "answer" without noticing that it's not what the submitter is looking for!!
      And we wonder why the rest of the world thinks IT people can't communicate...

Re:Write them to a DVD jukebox

[CastrTroy]

I remember working with VHS tapes. We had to lay down a "control track" by recording a continuous stream over the entire disc before using it. Maybe it's just because my highschool had bad video editing hardware, but I remember that this control track was important if you wanted the editing machines to be able to properly align with single frames when editing, and for the time and frame number to be consistent.

Re:Write them to a DVD jukebox

[ThinkingInBinary]

If I want to alter records all I need to do is rip the DVD, edit it on my HD, and burn it to a new DVD+R, abd destroy the old one.

Maybe I'm missing something, but wouldn't that be possible with unsigned data on any media? If you can obtain the media and a writer, and the data isn't authenticated somehow, you can always simply write a new version and toss the old. Unalterable does not mean impossible to destroy, just impossible to modify once written. Cryptographically signing data before writing it to any write-once medium (like DVD+R) would seem to solve this problem, because you'd need the signing key to "modify" it as you suggest.

Re:Write them to a DVD jukebox

[alcourt]

Append only files have not been required in my experience. What is required is that there be no ability to overwrite a previously written file by the team that is sending the log data. This can be done a number of ways, but the easiest method is to transmit the data in a way that the server chooses the filename, not the client. Add a date string into the filename and you can (with a few other details I've worked at but am here waving a wand at) avoid the problem.

syslog works for most data, but not all. Linux is one of the only Unix based systems that puts sulog through syslog. The failed logins log is much more difficult, as is the wtmp data. wtmp data is especially annoying as it is one of the only ways to semi-reliably record both login and logout regardless of login type (including ssh), and can't really handle real time data streaming. The other annoying item is the command line history of all commands with EUID 0. I'm hoping to hear some news soon on a solution to that problem, but it is really difficult, especially since a lot of SAs become root via `sudo -s` or `su` (as opposed to `su -`, which would not modify their HISTFILE variable. Many root shells do not support direct sending of HISTFILE over the network.

As to writing periodically to a optical media, I wouldn't worry quite so much about that. I would instead worry more about the encrypting all that security data while in network transit. (Sorry, can't recall if that is a firm requirement of PCIDSS 1.1 or not). Unfortunately, this makes use of syslog a less trivial solution. Authenticity is also an issue to be concerned with. How do you know that the event that got inserted into the log really came from that box, and not some random other server? Traditionally, syslog has not concerned itself with such issues, but a PCI system may care a great deal.

Once the data is on the central logging host, it is already in a state that the author of the data (the SAs for the PCI impacted box) cannot modify it. That eliminates at least in the interpretation of PCI I've been working on, the need for writing to optical media. Immutable is not so much immutable by anyone, but immutable by the server in question.

The point of the central copy of the logs is so that modification on either side can be readily detected and investigated. But if you cannot trust your central log host to have an accurate copy of the logs because you are receiving log data from anyone who chooses to pretend they are your PCI impacted server, then your central log host does not give you as much value as it may seem. The audit requirements aren't just for making lives miserable, they usually have a valid point behind them.

When working with PCI, know which DSS you are on, 1.0 or 1.1. (I don't know the release schedule for the next PCIDSS.) The requirements do differ, as do even the interpretations. Reference https://www.pcisecuritystandards.org/ for the information.

Re:Write them to a DVD jukebox

[ajs]

Append only files have not been required in my experience. What is required is that there be no ability to overwrite a previously written file by the team that is sending the log data. The one is a way to get the other... at least partially. Physical and electronic security and partitioning of roles gets you the rest of the way.

This can be done a number of ways, but the easiest method is to transmit the data in a way that the server chooses the filename, not the client. I'm not sure how filenames enter into it, since you don't give the application people access to the log host anyway.

syslog works for most data, but not all. Linux is one of the only Unix based systems that puts sulog through syslog. The failed logins log is much more difficult, as is the wtmp data. There is a "syslog Working Group" that's working on that and other problems. I don't know if syslogng supports any of their proposals yet, though.

As to writing periodically to a optical media, I wouldn't worry quite so much about that. It's very important to be able to talk about your risk exposure profile. When you can say that the exposure to electronic subversion of the logs (regardless of how hard you make that, via electronic security) ends when the data is written to optical disk, you can make a much stronger case for the data being functionally write-once.

I would instead worry more about the encrypting all that security data while in network transit. That would only be necessary if you transmit sensitive data in the logs. For example, if a healthcare company wrote client data to their in-house application server's logs, then the logs would have to be subject to the same security constraints as every other piece of sensitive data. This is as far as I know, but my PCI exposure is tangential, and I haven't read the requirements first-hand.

Authenticity is also an issue to be concerned with. How do you know that the event that got inserted into the log really came from that box, and not some random other server? Typically, you don't care (as long as you have the valid entries, any invalid ones are typically just noise), sometimes you do. I'm not aware of any PCI requirement for authentication in general, but for some purposes that may be there. I do think that syslogng might provide a means of minimal authentication, but I'm not certain about that.

When working with PCI, know which DSS you are on, 1.0 or 1.1. A fair point.

Re:Write them to a DVD jukebox

[beckerist]

^^not related to the parent, but not off-topic^^

It's HIPAA, not HIPPA, though we in the medical biz do refer to it as the giant hippo on our back!

Don't get me wrong, patient privacy is important, but these regulations are incredibly strict! If I receive an email containing just ONE identifiable record for a medical patient, I'm to report it and the sender is to be cited. This includes birthdays, phone numbers or billing IDs (which are completely internal to the facility!)

Two citations and you're canned... Yeah, that's right, send me an email containing a patients phone number and birthday and you are fired, no questions asked.

Re:Write them to a DVD jukebox

[alcourt]

Append only files have not been required in my experience. What is required is that there be no ability to overwrite a previously written file by the team that is sending the log data.

The one is a way to get the other... at least partially. Physical and electronic security and partitioning of roles gets you the rest of the way.

Agreed. I just find that the lack of support for append only files makes it hard to use as a solution on most platforms.

This can be done a number of ways, but the easiest method is to transmit the data in a way that the server chooses the filename, not the client.

I'm not sure how filenames enter into it, since you don't give the application people access to the log host anyway.

The solution I'm familiar with receives a datastream and writes to a file. If I allowed the sender to select a filename to write, they could hypothetically corrupt or worse, delete log data. It's a little easier to set up than most solutions to transmit the data securely.

syslog works for most data, but not all. Linux is one of the only Unix based systems that puts sulog through syslog. The failed logins log is much more difficult, as is the wtmp data.

There is a "syslog Working Group" that's working on that and other problems. I don't know if syslogng supports any of their proposals yet, though.

I don't see how this can help. The issue isn't so much how to handle data that has gone into the syslog stream, but how to grab critical log data that doesn't normally enter into the syslog mechanism in the first place. Maybe I am missing something? I am however interested in hearing more about the working group, especially if they are likely to be able to update the standards so that the commercial Unix vendors will be able to seriously implement an improved syslog. Sun's comment on why Solaris 10 didn't have a better syslog was they wanted to, but they felt bound by POSIX. True or not, there is at least the impression by some of the vendors that they aren't allowed to use a better syslog by default. Replacing every single box's syslog would be problematic in larger shops.

As to writing periodically to a optical media, I wouldn't worry quite so much about that.

It's very important to be able to talk about your risk exposure profile. When you can say that the exposure to electronic subversion of the logs (regardless of how hard you make that, via electronic security) ends when the data is written to optical disk, you can make a much stronger case for the data being functionally write-once.

I wouldn't say the risk ends, just that the risk for modification effectively ends. But I very much agree, one needs to look at the threat profile.

I would instead worry more about the encrypting all that security data while in network transit.

That would only be necessary if you transmit sensitive data in the logs. For example, if a healthcare company wrote client data to their in-house application server's logs, then the logs would have to be subject to the same security constraints as every other piece of sensitive data. This is as far as I know, but my PCI exposure is tangential, and I haven't read the requirements first-hand.

Since the logs in question are security logs, my inclination is to always encrypt them just in case sensitive data does enter it. It need not be true credit card data, it could be other items that increase attack exposure. Knee-jerk? Maybe. One thing to think about though, the security logs of one box may not be critical enough to justify encrypting, but the security logs of lots of systems together may be that sensitive.

Authenticity is also an issue to be concerned with. How do you know that the event that got inserted into the log really came from that box, and not some ra

Re:Write them to a DVD jukebox

[einhverfr]

Append only files have not been required in my experience. What is required is that there be no ability to overwrite a previously written file by the team that is sending the log data. This can be done a number of ways, but the easiest method is to transmit the data in a way that the server chooses the filename, not the client. Add a date string into the filename and you can (with a few other details I've worked at but am here waving a wand at) avoid the problem.

Sure, and append-only is not foolproof. It is just one step in the right direction. Defense in depth.

syslog works for most data, but not all. Linux is one of the only Unix based systems that puts sulog through syslog. The failed logins log is much more difficult, as is the wtmp data. wtmp data is especially annoying as it is one of the only ways to semi-reliably record both login and logout regardless of login type (including ssh), and can't really handle real time data streaming. The other annoying item is the command line history of all commands with EUID 0. I'm hoping to hear some news soon on a solution to that problem, but it is really difficult, especially since a lot of SAs become root via `sudo -s` or `su` (as opposed to `su -`, which would not modify their HISTFILE variable. Many root shells do not support direct sending of HISTFILE over the network.

Allowing multiple people to log in as root (through su or otherwise) violates the PCI-DSS requirements according to my reading. In my view the *only* acceptable option is to either limit root to one person (for small environments) or require that everyone use sudo exclusively for executing root commands.

As to writing periodically to a optical media, I wouldn't worry quite so much about that. I would instead worry more about the encrypting all that security data while in network transit. (Sorry, can't recall if that is a firm requirement of PCIDSS 1.1 or not). Unfortunately, this makes use of syslog a less trivial solution. Authenticity is also an issue to be concerned with. How do you know that the event that got inserted into the log really came from that box, and not some random other server? Traditionally, syslog has not concerned itself with such issues, but a PCI system may care a great deal.

PCI-DSS only requires that certain information (useful in creating credit card transactions) is encrypted in transit and that this information may *not* be stored in the logs. So logs are not considered to be sensitive enough to require encryption. However, if you need to do this, there are a number of options including IPSec (which will give you the host-based security controls).

Once the data is on the central logging host, it is already in a state that the author of the data (the SAs for the PCI impacted box) cannot modify it. That eliminates at least in the interpretation of PCI I've been working on, the need for writing to optical media. Immutable is not so much immutable by anyone, but immutable by the server in question.

Agreed. The main purpose is to prevent an attacker from covering his tracks by screwing with the logs. The PCI-DSS is largely a standard to ensure that people who are processing credit card transactions are not storing overly sensitive data, are using encryption appropriately for somewhat sensitive data that they must retain, and are generally following industry-accepted best security practices.

The point of the central copy of the logs is so that modification on either side can be readily detected and investigated. But if you cannot trust your central log host to have an accurate copy of the logs because you are receiving log data from anyone who chooses to pretend they are your PCI impacted server, then your central log host does not give you as much value as it may seem. The audit requirements aren't just for making lives miserable, they usually have a valid point behind them.

When working with PCI, know which DSS you are o

Question... What's to stop

[Anonymous Coward]

What's to stop someone from reading in one of the WORM tapes, modifying the log file data and then writing it back to another blank WORM tape and claiming it's the unaltered tape?

Do all logs have to be encrypted and signed with a seperate super sekret key/cert before being recorded?

Re:Question... What's to stop

[beyondkaoru]

i don't know much about the laws/regulations in question here, but yes, there isn't anything stopping someone from making a new 'worm' storage device and claiming it to be new unless there's a third party who will remember identifying information on the data.

if i really wanted to make sure my archives weren't tampered with, i'd bring my data (in whatever medium, the 'worm' thing wouldn't be necessary to ensure non-tampering, though it'd be good for storage purposes) to a trustworthy and hopefully vaguely computer savvy notary. then, i and the notary would hash the data, write up a form that says "i, name here, declare that on this date data with this hash value, some hexadecimal, was filed. signed, signature".

storage aside, this means that for someone to tamper with it they'd have to either bribe/coerce/kill people who saw this form (difficult) or reverse a cryptographic hash (even more difficult). so, pick a good notary (or submit the hash value to the gov maybe?) and a good hash function (like a larger sha or whirlpool) and i think you're tamperproof.

of course, i don't know the regulation so i don't know if this matches the needs of the article.

Re:Question... What's to stop

[beyondkaoru]

well, by being 'tamper evident' as you say, you are in fact tamper proof, so long as the data is well stored (and tape drives, cd or dvd jukeboxes, etc can do a good job of this). an iron-clad 'tamper-proof' box is not, in fact, tamper proof if one can simply substitute another iron-clad box in its place. this is the reason that only having a dvd jukebox wouldn't be secure (though again, i don't know what the regulation's requirements are). a nefarious company could simply juxtapose dvd's. remember that the ones who would be interested in tampering with the data are also the same folks who are storing and maintaining it.

Re:Question... What's to stop

[More_Cowbell]

/kill people who saw this form (difficult) or reverse a cryptographic hash (even more difficult).

So you find it easier to kill people than to run computer programs... Remind me not to get on your shit list. :p

use a line printer

[1u3hr]

Connect a line printer to mirror the log file as it's created. Use continuous fanfold paper. Get staff to sign and date first and last page.

Lawyers love paper. (A magistate once asked me if a printout I presented in a case was an "original email". I said it was as close as you could get.) In all likelihood, no one will ever refer to it, so don't worry about that it might take 10 minutes to find a page. Once a month, ship it to a secure storage. For real paranoia, have two printers making two simultaneous copies.

Re:use a line printer

[Nefarious Wheel]

Wouldn't work in Australia, compliance penalties apply if you can't dredge up the data within a specified period of time. YMMV but it'd be worth checking what the regs actually require. A good reference is this little PDF I found http://www.ironport.com/pdf/ironport_email_complia nce_guide.pdf/

Personally I'd think about a hardware solution, block replication off-site to a third party registry. When you're talking compliance (especially fiduciary compliance) it's usually easy to come up with the bucks, so dream up something right and propose it.

Re:use a line printer

[pla]

Wouldn't work in Australia, compliance penalties apply if you can't dredge up the data within a specified period of time.

The hardcopy just "proves" that no one has tampered with the normal (and searcheable) logs. You would never actually use the hardcopy, you would just have it in storage somewhere. You would still use the standard system (or application) logs for day-to-day auditing.

And if/when you find yourself in court, answering the question "can you prove this as the real logged data?", you can confidently say "yes" and have the dumptruck full of printouts pull up in front of the courthouse.


Though, personally, I don't see the problem with using a non-rewriteable DVD. Yes, each log entry would take up a full sector, but that still gives you, per disc, 2.3 million log entries of up to 2k each. That should last at least a few days (more likely, a few months) at any company small enough to balk at the cost of an OEM solution.

Re:use a line printer

[The Mysterious X]

Despite being "broken", in this case, sha1 would be acceptable.

All SHA1 being broken means is that it is easy to find a collision, or 2 values that match. If you are using it to verify the integrity of a file, then even if a collision is found, it's going to be plainly evident.

Though it's easy to find a collision, it is *impossible* to choose the content of that collision.

The importance of SHA1 being broken is when it is used for say, obfuscating passwords. If a system is compromised, and the cracker gets a list of password hashes, they can then generate from that list of password hashes, a list of valid plaintext sequences that would generate that hash.

So in the former case, the cracker would find a matching set of plaintext to the logfile, but due to the contents of the alternative plaintext probably being a psuedo-random jumble of data, anyone who looks in any detail at the fake log file will instantly see that it is falsified. The cracker may as well create a false logfile, and lie about the hash.

In the latter, it would allow the cracker to get a list of passwords that could be used to compromise his target systems much more quickly than he could have without them.

Re:use a line printer

[amiga500]

The NASD has a requirement that a firm must keep a copy of all email sent and received for three years. We figured the NASD must have the same requirement, so a simple solution would be to forward copies of all our email to the NASD, and let them worry about retaining it. I was proposing solutions for another bank on how they could meet the PCI DSS requirements, and the business users decided it would just be easier if we didn't log anything at all. That we we didn't have to worry about them getting tampered or falling into the wrong hand.

Re:use a line printer

[sjames]

Though it's easy to find a collision, it is *impossible* to choose the content of that collision.

In this case, "easy" means not utterly impossible to accomplish in a lifetime if you have unlimited funds.

The significance isn't that the new attack is "practical". It's more that given those results, the odds of an even better attack coming along in the next decade or two went up.

All the same, for a brand new application, why not just use SHA256? That's what Jon Callas meant by "walk, but not run, to the fire exits". No need to panic over data already protected by SHA1 or even to run around replacing all uses of SHA1 this instant, but if you're writing code anyway, why not choose a safer option?

As you say, you don't get to choose the collision, that's why it's not time to panic.

USB Card punch

[www.sorehands.com]

And you thought there was no use for a USB card punch.

Hard to change punched cards. Just don't trip with your box of cards.

Tattoos

[Anonymous Coward]

Start tattooing everyone in the office with data. Encode it in some nice optical way (a la barcodes) for easy reading later.

Ontop of the obvious benefits, it provides a good deal of job security, if they get fired, they take away some important data, your employees will be thrilled with their newfound sense of security.

Go with commercial hardware solution

[jsimon12]

I preface this by saying I know I will get flamed for not recommending Open Source but SOX is a Federal mandate (Federal equals PMITA)

EMC's Centera is my personal favorite, it isn't cheap but it does exactly what you need and is auditable and recognized by all the third party audit compmaies as well as the Federal government.

I have worked in IT for 15 years and 5 of those have been for a LARGE financial institution. When it comes to audit and SOX go with something standard, tested and commercial, unless you want to spend the next 6 months explaining to your auditors how your homegrown solution works and then the next 6 months building something new that your auditors do understand (or worse, like losing your job).

Re:Go with commercial hardware solution

[feepness]

unless you want to spend the next 6 months explaining to your auditors how your homegrown solution works and then the next 6 months building something new that your auditors do understand (or worse, like losing your job). I dunno, I can lose my job WAY faster than 6 months.

How odd

[Anonymous Coward]

It looks like that commercial offering is a piece of hardware with a network API (web service) that you write to which doesn't provide any network APIs for deleting or modifying records. Presumably it has a read-only view of the data.

Now, assuming that they use harddrives, we all know that someone could extract mount the file system and change records. They could hide their tracks by recalculating cryptographic hashes. So it's simply a lie to say that the only way to modify or indeed delete the data the data is through physically destroying the hardware.

However having a dedicated hardware for a write-only, read-many system is a good idea, but I can't imagine that this would necessarily be more complex than a locked down box running some web scripting language that appends http post data to a log, or a database.

If there is more to it then please could someone elabourate on what exactly one must do?

Re:How odd

[arivanov]

Now, assuming that they use harddrives, we all know that someone could extract mount the file system and change records.

Not if they have done it properly. If it is designed as an audit solution it is likely to have a hardware crypto module, a device specific key and have all data written out to disks at least signed with it. More likely - encrypted with it. In either case even if the fs is standard you cannot do jack sh*** with it after taking the drives out.

By the way - implementing the above using OSS is trivial as all free OS-es nowdays provide a TPM API so you can have unique machine keys. In fact you can implement this on top of any Free OS and integrate it with any standard MTA and most applications with minimal effort. The implementation would also most likely pass audit scrutiny as it is trivial. The only sticking point will be the crypto procedures and especially escrow. While proving that the app and the design is compliant is not hard, proving that your CA procedures are solid is a phenomenal pain in the a***. Also, you need to prove that you have an effective escrow and taking a hammer to the log machine does not prevent reading the compliance logs later on. The vendor has already done that and the auditors are happy. Compared to that it will take you on average 4-6 months to get this done with the help of external consluttants. Now, if you have done it anyway for a different project that is an entirely differnet ball game. You always have to prove to auditors that your app does what it says on the tin anyway and the apps are often internal. So one more or one less item is not going to turn the boat if the main sticking points (the CA and the escrow) have already been done.

Re:How odd

[Sobrique]

Sort of, but not quite. A Centerra is a Content Addressed Storage thingy. Which basically means it's file identifiers are md5 sums. It's a multi node thingummy too, which replicates stuff about. Is it impossible to tamper with? Well, no, nothing is. But it's pretty hard. Simply because it has implicit 'tamper detection'.

The API is also geared up so you can choose what 'mode' you want it to operate in. In the most secure mode, the API and OS built in (it's Suse based) won't let you delete anything. Which, basically means you have to pull out the individual drives that 'clip' is stored on, to trash it. Data will be gone, which isn't great, but ... well, pretty much impossible to prevent for any system. Modifying data retroactively though, is much much harder - recreating the right md5sum is a non trivial task. Impossible? Perhaps not, but ... well, EMC have done quite well with 'selling' this product in a 'it will meet your compliance needs' which is considered good enough for our auditors.

We have 'financial organisation' regulations, for retention of emails, and a Centerra is what we settled on as the solution.

Re:How odd

[Sobrique]

I should add:

Centerras don't count as the original post, of a 'cheap solution'. They're not all that expensive by 'enterprise standards' but that's ... well not quite the same as 'affordable for most people'.

Also, our data centre is under fairly intensive scrutiny and control of physical access. My employer and customer are well aware that physical access means all bets are off, so in order to get physical access you need escorting, and authorization in advance, including documentation of what you're changing, why, and which grid squares in the datacentre you need access to.

I and the rest of my team are admins on this Centerra don't get access to the datacentre. If we have a need to enter, then we can fill in the paperwork and do so, but ... well, we're based 100 miles away. Most 'hands on' is done by someone else.

Now, combine that with the fact that each 'clip' (file) is stored 4 times, on 4 separate physical devices (2 of each, on 2 different sites) it would require ... well quite a few people to be complicit to even be able to destroy (or tamper with) data, physically. And a hell of a lot more to do so without leaving great big footprints all over the place screaming to the world what you've done.

I think you'd need 2 people on each site (one to actually tamper, and one to 'not notice' as he was escorting), plus an admin person offsite to identify which drives need 'doing', on both sites, and to mess with the 'self healing' replication so that one site didn't just restore the other. (You'd have to be fairly quick on the drives too, as soon as one goes down, the healing starts to replicate to other 'spare' drives).

And then you'd need some other people to mess with the entry logs to site, CCTV footage, change authorization....

You'd have to be pretty damn serious to pull that off. I mean, it's not even a case of some pointy haired one seeing their career on the line, and demanding immediate sabotage.

Re:Syslog

[Whiney Mac Fanboy]

Besides logging locally to disk, also add a line to /etc/syslog.conf to log to a remote machine.

If syslog can write to a remote machine, then a compromised syslog can overwrite a file on a remote machine. I suspect thats not even remotely close to enough read-only.

As others have suggested, print your logs on a line printer.

Sometimes, the old ideas are the best

[polymath69]

Nothing is an unalterable as a line printer which lacks reverse-vertical-paging capability. Just make sure it doesn't run out of ink or paper.

WORM Device

[passthecrackpipe]

What you need is a Write Once Read Many (WORM) device. Unless EMC started shipping Open Source hardware (hahaha) I don't think you will be able to find this as Open Source. There may be some software solution, but you would most likely need some certification for it anyway ("no, officer, it _really_ is unalterable, trust me....."). Granted, most "hardware" solutions implement WORM through software, but I know from experience that it is impossible to change the data on WORM.

Technically, a CD-R with some checksumming would work to be compliant - these guys have some more info, but if you need it for formal compliance use, you are better off talking to your friendly neighbourhood storage vendor to save you lots of legal hassle should you ever need the WORM thing for evidence. It is the difference between a lengthy legal process where you have to explain exactly why your homebrew solution is legal and simply saying "talk to NetApp"

How sure do you need to be?

[edashofy]

Cryptography, digital signatures, and checksums can only take you so far. They can detect tampering pretty easily. However, crypto can't prevent someone from deleting a file, although by checksumming or signing a whole bunch of files you could at least detect deletion of one of them. Ultimately, if you really want permanence, you need to write it out (as an above poster suggested) to some sort of write-once media. CD-Rs or DVD-Rs would obviously fit the bill here, although one can indeed delete a CD-R by simply throwing it out, of course.

Another cheap write-only medium is paper; I suppose you could purchase a laser printer (or even a line printer), and have it spit out the logs as they occur. If you kept the printer in a locked transparent box, nobody but people with the keys would have access to the output.

You could burn the logs onto PROMs as well, that's pretty permanent :)

Anything on magnetic or flash media can be erased or tampered with somehow, unless the drive controller hardware itself prohibited overwriting existing data. Even then you're relying on someone not being able to replace the drive controller or take the drive apart and diddle the platters/flash chips directly (although I suppose a decent amount of epoxy could thwart this). Any software-based solution can be tampered with in theory. One hacker favorite (which may be a legend or not) is that people used to get root on other people's boxes and then replace their copy of PGP with an instrumented copy. Thus, even the encryption software became compromised.

For compliance, though, I'm not sure what kind of oversight you have to have. At the end of the day, somebody has to be trusted with these logs, and that person would almost assuredly have the power to destroy them, or at least portions of them.

Re:How sure do you need to be?

[Maximum Prophet]

Anything can be destroyed or altered and as with any security issue this a matter of making the cost of doing so more then anyone is willing to pay.

Absolutely true in principle, but in practice it can be hard to put a proper dollar value on any small piece of information. Here's an extreme example. Suppose you have a manufacturer that makes widgets that are worth $N. They implement access controls on the doors, so they know who is coming and going. If one employee can carry M widgets, then they can estimate the value of one record at $N*M. Now, let's say that one day an employee comes in during the off hours, kills another employee, then decides to spend $N*Z dollars to remove the record that he was there, where Z >> M.
The trouble is most times we figure out what the data might be worth to us, not taking into account what it might be worth to the bad guys. The opposite scenario is more likely, where a company spends much more to protect a piece of data than it would ever be worth. In that case they are wasting money that would be better spent doing real work.

The best thing you can do with your cryptographic hashes is to have copies away from the actual logs. Make sure that the people who have access to the remote hashes are different than the people who have access to the logs. Then it takes at least two people working together to muck things up.

Line Printer won't scale

[jsimon12]

If you only had a single machine or maybe even a couple lightly used boxes a printer might work. But even that would be near impossible to go back and sort through and if you ever ran out of paper or ink you would be SOL. And trust me the last thing you want is to to be SOL when it comes to SOX. If you have the money don't half ass an audit solution.

Syslog + chattr

[ethzer0]

I use syslog-ng to relay information from several different datacenters to a centralized and secure location hosting all of the syslog information. Each DC has its own syslog-ng system acting as the local relay, transporting syslog information from local clients using TCP over a VPN to the centralized host. The logs are written on the central syslog sever organized by on date and hostname, and each file that is created is then assigned an 'append-only' bit using chattr. It works really well.

The Linux way

[professorfalcon]

tail -f thelog.txt >> /mnt/cdrom/thelog.txt

FreeBSD to the rescue

[stox]

FeeBSD supports append only files via the chflags command.

Re:FreeBSD to the rescue

[moosesocks]

Yes, but the tricky thing about this situation is that it's a "who will guard the guards" type of deal.

If the root user can set that attribute, he can just as easily unset it, modify the data, and clean up after himself before re-setting it.

Remotely spitting your logs out to a line printer managed by a trusted 3rd party would seem to be a reasonable solution.

One-way data cable

[rjh]

At USENIX/EVT06 last year a team from the University of Iowa presented a cheap one-way data cable you could make with off-the-shelf parts from Radio Shack. Total cost is about $5 (for bulk, maybe $10 if you're buying single units) and it is provably, auditably, one-way. It was originally developed for electronic voting, to allow for counting computers to communicate with webservers that post election results. An attacker compromising the webserver cannot attack the counting computer, because there is literally no return path.

It works with very high reliability up to about 9600 baud.

You may be able to use this to your benefit. Have an isolated system air-gapped from the rest of the network which listens for log events on a one-way data cable. While you're no longer guaranteed to be safe (since if a logging PC is compromised, an attacker could send compromised data to the syslog PC and perhaps cause some sort of mayhem), but the lack of a return path makes interactive attacks infeasible.

ObDisclosure: I am a graduate student at UI and know the guy who invented the data cable, although I am not associated with the gadget.

Re:Syslog

[pedestrian crossing]

As others have suggested, print your logs on a line printer.

But that doesn't really scale very well, and then you have the problem of dealing with retention/storage requirements.

Write-Only Memory

[jmv]

That's all you need

Why not store it in a version control system

[Ptur]

I would dump it in GIT or the likes.... any change of it will be recorded ;) Seriously, many version controlling systems already contain the data integrity and authenticity checks that you need

This request is impossible.

[The Master Control P]

Given sufficient resources, time, and dedication, ANY log can be altered.

If the "unalterable" log is maintained in software, it's a comparatively simple matter of hoisting it up on a VM. Since we're presumably talking about white-collar crime, it's a fair bet they have or can get root access to the machine to install the VM and rootkit to hide it. At that point, the CEO can do anything and the system can't fight back. Capture passwords of people logging in, alter data, you name it.

A hardware system would be more robust, but still vulnerable. I imagine the most likely attack vector would be Man in the Middle - Just take over the box that guards/drives the logger machine.

Re:This request is impossible.

[hazem]

Given sufficient resources, time, and dedication, ANY log can be altered.

What really matters is if there is any case law that actually interprets the laws and provides standards for due diligence.

The law might say "unalterable" or "lasting indefinitely" we all know there are practical physical limits - given enough time, anything is alterable and nothing lasts forever. We could come up with outrageous methods like using satellites with lasers to etch logs on the surface of the moon, but there's not much point in going to such expense until the case law suggests that is what is needed to be "due diligence".

Until you have that case law each organization will simply have to do a cost x risk analysis and determine how much they're willing to risk in order to "do enough" to keep out of trouble with the auditors. Something like:

chance of being audited x cost of failing an audit vs. cost of maintaining an "unalterable" logging system

Then you just wait to see what organizations get skewered and adjust your analysis and practices accordingly... and just hope you're not the first to get audited. OR... we can work on a way to etch logs onto spheres of aluminum that are then launched into orbit so they can be read with a telescope. Though that will probably lead to an increase in insurance premiums.

Guy Fawkes Protocol

[LilBlackKittie]

Some of the work I do may require something like this, so I'm considering implementing Guy Fawkes over syslog.

http://www.cl.cam.ac.uk/~rja14/Papers/fawkes.pdf

From the paper:

6.2 Tamper-evident audit trails

It is a well known problem that an intruder can often acquire root status by using well known operating system weaknesses, and then alter the audit and log information to remove the evidence of the intrusion. In order to prevent this, some Unix systems require that operations on log and audit data other than reads and appends be carried out from the system console. Others do not, and it could be of value to arrange alternative tamper-evidence mechanisms.

A first idea might be to simply sign and timestamp the audit trail at regular intervals, but this is not sufficient as a root intruder will be able to obtain the private signing key and retrospectively forge audit records. In addition, the intervals would have to be small (of the order of a second, or even less) and the computation of RSA or DSA signatures at this frequency could impose a noticeable system overhead.

In this application, the Guy Fawkes protocol appears well suited because of the low computational overhead (two hash function computations per signature) and the fact that all secrets are transient; this second's secret codeword is no use in forging a signature of a second ago.

Re:Syslog

[Boricle]

Probably the same thing that stops you from making scanning in the old print out, modifying, printing it out again and putting it in the stack.

i.e. Nothing really.

However, if you have the CD or tapes signed and dated by the ops staff, then shipped to off site security, you've made it harder to falsify.

The interesting issue is that if you are organized enough, what's to stop you from intercepting the messages on the way to the printer / CDR?

The only way I could see around this is some kind of trusted computing style initiative.

Re:Syslog

[Anonymous Coward]

a compromised syslog can overwrite a file on a remote machine

Not with a properly configured syslog. You're not supposed to just use a remote logfile, but a remote logging daemon (RFC 3164). That way you can add entries to a remote log, but not change or delete any (unless you make the logfile directly accessible over the network, which I wouldn't recommend).

Re:Syslog

[cerberusss]

If syslog can write to a remote machine, then a compromised syslog can overwrite a file on a remote machine.

I don't think so; the receiving syslog machine will be "add-only" and won't have rights to overwrite files. Of course, you can print your logs and that would be a good second defense. But searching through printed logs manually is a pain in the butt.

Clicky

[mritunjai]

WORM media with HIPPA compliance in mind...

WORM on wiki

From Experience

[Evets]

I honestly don't know about DSS or SOX, but I have had plenty of fun with HIPAA.

Unalterable logs as a matter of compliance does not mean "absolutely unalterable under any circumstances". There should be no way for an end user to modify audit trails. There should be no preconceived way for an administrator to alter audit trails - i.e. no utilities for doing so. That does not mean that an admin can't go directly into the DB and alter the data from behind the application layer.

Under every circumstance when I have run into audit logs involving HIPAA compliance, they have been written by an application directly into a SQL database (oracle, ms sql, informix, and one time db2). It used to be that they were written in a fairly easy to decipher format within a single text column on a per record basis - which made for a fairly-difficult-to-alter audit trail because within that easy to decipher format were non-printable characters that you would at least have to know to look for them. With current implimentations, however, the records are stored in a separate table with a many-to-one relationship with the audit-required records, in varchar fields, as plain text - much easier to alter or get rid of single entries. There is still a level of obfuscation as far as table names and column names but thats really a side effect of other things that are going on.

These systems have been reviewed by auditors and certified as compliant. In the older system, there was no application interface to delete audit records. In the newer system, there is an application interface to delete records in any given application table - and therefore there is one for the audit tables as well. Admin level access is required to delete or alter the records, though.

Personally, I would expect more as far as HIPAA compliance goes - from both a customer standpoint and an auditor standpoint. My experience (and it is pretty extensive across several high profile enterprises) - is that the customer will demand a better system only when the auditors demand a better system. I haven't run into an auditor yet who has even given more than a casual glance at the 'back door' scenario. I suppose it's because there is no true way to keep things absolutely secure and application level audit log security is only one layer of the onion.

Before you get too far into an overly complex and potentially expensive solution, talk with your auditors about the requirements for your specific scenarios. They've seen it before and can tell you exactly what they are looking for from an audit compliance standpoint. They are usually pretty easy to work with and open with their knowledge.

Re:From Experience

[georgewilliamherbert]

Second the "ask the auditors what they are looking for"... not everyone gets audited the same.

Financial company I know passed audit fine with syslog -> a secure system which the normal sysadmins didn't have access to. The people whose actions were being logged couldn't get to the logs (well, presumably someone could break the system, but it was well secured and had non-overlapping sysadmin staff).

That was good enough. As long as it took two compromised people to hide any given event, that passed audit.

Re:From Experience

[sjames]

Unalterable logs as a matter of compliance does not mean "absolutely unalterable under any circumstances". There should be no way for an end user to modify audit trails. There should be no preconceived way for an administrator to alter audit trails - i.e. no utilities for doing so. That does not mean that an admin can't go directly into the DB and alter the data from behind the application layer.

That's VERY important to keep in mind. A lot of the wailing, hype, and FUD around all of the various auditing and retention laws comes from people who do not understand that fundamentally absolutely ANY audit trail can be altered given sufficient determination and resources. Even if the logs are chiseled into stone slabs it is not absolutely inconceivable that someone might produce a slab thgat is identical in every way except for a changed digit or 2.

WORM media can be duplicated as well. Whole vaults of WORM media can be duplicated. if you save hashes of the data seperately, the media containing the hashes can be swapped just like the main logs.

So, making it absolutely impossible to alter data is out of the question, it's really a matter of how hard you can make it without bankrupting the company in the process (cheapest solution is fold the company. No company means no data means no alterations).

Dumping the lot to a line printer in real time AND storing to a log file is one answer. In some cases, just keeping the logfiles in ext[23] and setting the append only attribute may be enough, at least until enough has accumulated to burn another sector onto a WORM device.

For a while, I ran a patched kernel that would allow the immutable or append only bits to be set by root but not cleared. Clearing the bits when necessary required booting into another kernel (which would trigger many alarms when the machine went down). Doing so was a regular procedure for maintenance, but that was scheduled and all admins were notified. An unscheduled event would NOT go un-noticed.

It's also useful to note that most of the requirements are that fraud be merely detectable. That is, the data need not be unalterable so long as the alteration is detectable. It's MUCH easier to detect that data was changed than it is to allow for reconstruction of the original data. One viable scenerio (given that fraud is a rare to non-existant event for the company) is to detect the alteration in the electronic data and then reconstruct the real data by following the paper audit trail.

Re:From Experience

[Akatosh]

For a while, I ran a patched kernel that would allow the immutable or append only bits to be set by root but not cleared. Clearing the bits when necessary required booting into another kernel (which would trigger many alarms when the machine went down). FYI you don't need a special kernel for that for linux,

lcap CAP_LINUX_IMMUTABLE CAP_SYS_RAWIO CAP_SYS_MODULE CAP_MKNOD CAP_SYS_BOOT
unlink /dev/sda1 (or whatever, after fscking/mounting)

that will disable the ability to alter immutable bits, access /dev/mem, kmem, etc, load kernel modules, access storage devices directly or reboot the system.

Now lets put said unalterable log on an encrypted partition that requires a key on a usb dongle to mount, split the key into a few parts and give the parts out to different people.

Also there are hardware crypto based document storage solutions out there that supposedly make things totally unalterable short of an act of god (embeded in laquer kind of thing). Ncipher makes some stuff like that, google them. (I don't have any vested interest, it's just the only company I know of that makes that sort of thing).

Physical Destruction

[unsubscribe]

In short the requirement is for a secure method to make sure that once a log is written it can never be deleted or changed.
Although it is possible to prevent logs from being modified (using write-once media) or undetectably tampered with (using crypto, possibly with a TPM module for the ultra-paranoid), any log can be 'deleted' by physically destroying the device/media on which it is stored.

Dont skimp...

[pjr.cc]

Seriously, when it comes to legal requirements, do not skimp!

Go for something that is guarentee'd to fulfill your legal compliance requirements.

Yeah, optical media is great for WORM, but you dont want something your going to have to manage day to day. The legal req's of sox and so forth are beyond that of traditional optical drives in terms of life span in any case. Do not go with optical for compliance unless its something specifically designed for compliance (Again, thats $$$).

As someone suggested, centera is a good option - but all the storage vendors have good options (from emc, netapp, hds, sun, falconstor, mimosa the list is endless) and they'll all tell you how theirs is better than anyone else (and why). At the end of the day, you want a compliance solution with someone's stamp on it, and a throat you can cut when it goes wrong.

If your absolutely determined to go the compliance route on OSS - go with ext3cow (www.ext3cow.com) IMHO, a fully versioning COW fs with a non-erasable past and the best OSS solution for the job - backup on to optical if you like, but dont make optical your only option. If it only had policy-based management (i.e. snapshot whenever user X or group y writes a file) rather then crontab'ing its snapshot agent it would almost be perfect for a start-point solution for compliance. It has a big benifit along with it though, you can show users how to get files "from yesterday".

Keep in mind, WORM means policy-based write-once, not necessarily immutable storage! And almost every compliance worm product out there depends on that fact.

Re:Syslog

[dkf]

You're not supposed to just use a remote logfile, but a remote logging daemon. Another thing you can do is to send the logging messages over a non-IP connection (e.g. a serial line) so that even a standard network failure won't disrupt the logging and a hacked machine will continue to generate a track-able log. And the last I heard, you can't unsend bits sent down a serial line.

Don't Build Your Own Device

[Interfacer]

I work for a big pharma company as a sysadmin, and we have to abide by similar rules and laws. Our data recording and data logging has to be proven to be unalterable.

Go with a commercial solution unless you want to battle with the QA and Validation departments for haf a year. And even if you would get the go-ahead (unlikely) you'd get in a hell of a lot of trouble during an audit because auditors a) don't know your solution and b) they will quickly see that it is not certified.

There are specified requirements (don't know the names and numbers by heart) that your solution has to proven to fulfill, and certified by some external party.
Just saying 'Yeah but I know it cannot be altered because it is syslog / ' will not cut it.

And non-compliance can eend up costing your company millions if not hundreds of millions.
Open source or home grown has it's place, but in a regulated environment you go with commercial for certain things because that is the only option where you get certification with your device / software.

Re:Don't Build Your Own Device

[timmarhy]

"Our data recording and data logging has to be proven to be unalterable."

no such thing exists. given enough time and a mediocure amount of money, i'm 100% certain i could alter anything your storing your information on and make it look real.

the toughest system i've ever seen as far as audit trails goes is using cdr's in a machine that makes a hash of the data on the cdr AND reads the serial number on the cd and stores that on a geographically seperate cdr system. it's similar to those automated cd turnstyle things you can buy, only beefy with steel casing and alarms on it and what not.

Re:Dont skimp... two other things.

[pjr.cc]

ext3cow was written with compliance in mind (i.e. with an untouchable past), and so its AFAIK the ONLY solution that can fit in compliance (keeping in mind that this only covers part of compliance). svn, git, cvs - im sorry, but thats just a non-solution for compliance. It also gives you no-mess management with a very easy interface to make sure you are being compliant (this is important, and its something YOU dont have to be involved in, your lawyers can "look at the past" to make sure "discovery" is going to be consistent).

The second thing is, compliance is (ridiculously) complex - the compliance vendors have spent many hours with lawyers getting it together, they know the requirements and they know they fullfill them - this is important. It also means their solutions come with an implicit warranty - "hey, your using netapp worm, we know it works" as apposed to "what software is that? how do you know it works?". At the end of the day a lawyer is going to either go "well i cant argue with the compliance solution" when your with a well-known or "your honor, the defendant is using ... which has never been proven or certified by anyone".

Compliance is the only time i will say to someone - "get a throat to cut", get a solution you know works, written by people who know what they are doing and its all because compliance req's were written by lawyers for lawyers (i.e. scum) and so their scum is going to make you have to act like scum.

Re:Syslog

[cerberusss]

Hmyeah I agree a line printer is good as an addition, however paper is hardly searchable. I bet one of the requirements would be to have an auditing interface searchable by user, date, et cetera.

Re:Syslog

[arth1]

Why, just tee it with one copy going to the remote syslogd on the remote secured machine, and the other copy being stored locally for easy access. Use the local insecure copy for day-to-day read-many access, and the remote secure copy for archival and legal purposes.

Yes, this will be write-twice, read-many, but that's much easier to implement than true WORM.

we did it ourselves in the end

[instantiator]

as it was quickest. writing: 1. append latest entry as plaintext 2. sign [signature of previous entry + plaintext of latest entry], and append as signature of latest entry

This is audit, not perfection

[nicolaiplum]

First of all, this is a requirement to satisfy an audit. It would be nice if whatever solution you come up with is actually good, but you only have to satisfy the auditors.
Auditors want processes and records to raise the barrier to someone doing something wrong or unrecorded. They know systems aren't perfect. Nerds all go "but someone could just make a change some other way so this system is no use". If it raises the barrier high enough, it is useful. All accounting books can be cooked, the point of auditing is to make it harder (and to make it require more people).
Some write-once methods are more accepted by auditors than others, because they have seen them before. Of course you can rewrite a WORM disc's contents onto another disc and edit the contents on the way, but most auditors think that is sufficiently difficult that they think WORM media is OK. And you can write several copies and send them to different places, then the person wanting to edit the log has to get to all the discs.
The commercial solutions like those from EMC and Network Appliance aren't foolproof. You could take all the discs and edit the data on the discs if you want. It's harder for you to do that than simply "su root; vi logfile" because you have to work at a lower level. All the EMC or NetApp software is promising the auditors is that it would be hard work to do it undetectably and that is what they are looking for.
An Open Source solution which has no API to change data once it is written would work the same as the EMC or NetApp, but if you are the first to use it then you have to persuade your auditors (and possibly a court down the line) that it really doesn't have any such API.
Usually Open Source is good for your business because you have control of it, can change it, and can see how it works.
Auditable logs are about you not having control of them, not being able to change them.
The easiest way to get your auditors to agree your logs meet the audit requirements is to use some solution that they have seen before. Buy a Netapp (I won't, personally, recommend EMC) or buy a WORM drive. DVD-R is WORM, it's cheap.

Re:Syslog

[kars]

That's easy; feed the paper coming out of the printer through some sort of OCR machine.

Compliance is risk-avoidance, don't be cheap.

[Bright Apollo]

I work in a regulated industry, and this is an ongoing topic at pharmaceuticals.
Basically, you weigh the cost of non-compliance versus compliance, figure out
what that risk is worth to your business, then try to spend as little as possible
to mitigate the risk until the cost is acceptable.

There is no such thing as 100% compliance or security. Oracle makes a big deal out
of their data vault tech, but there's someone out there who can circumvent it. You
need to figure out your comfort level for the risk, and in big corps, this is a
financial decision.

Which leads me to this: there is no "roll your own" compliance software. You do not
want to assume the responsibility of proving to auditors that your software is correct
and fully-functional. That is a difficult process to behold, and it will make your
dev team crazy with paperwork. This is why people buy commercial off-the-shelf (COTS)
software and then configure it, as they can then point to the COTS vendor and say
"He vouches for the software". Auditors already versed in the COTS solution will
then look to see examples of your configuration to see if it's sufficient, then
move on.

Sure, it's a nice intellectual exercise and certainly worthy of development, by a
dedicated team willing to tackle all of the issues around securing the data, providing
secure authentication and controls, proving non-repudiation and temporal consistency,
etc, all of which a one-man show cannot achieve, all of which a half-assed token effort
cannot achieve.

Really, it boils down to this: you wanna roll the dice on your company being under a
consent decree from the DoJ because you were too cheap to buy a system? That cost can
shutter your doors.

-BA

Re:But delete is still easy to do

[fimbulvetr]

It's not about "deleting" the data, it's about trusting the data you have. Just like there's a difference between disinformation and no information - though I admit there should be procedures in place to keep said things away from your super powerful 1200 watt coffee maker.

Re:Syslog

[Albanach]

Sure. but what keeps you from making a copy of the CD-ROM with certain info deleted or altered, and putting that in the archives instead of the original?

You miss the point. The submitter is not asking for a way to produce a secure write once logging system, he's asking for a way to make a write once logging system that is compliant with the legislation he is operating under.

While we may be able to see endless faults with any proposed solution that doesn't matter. He just needs to implement one that covers him and the suits above him should they be audited.

Re:Syslog

[Nos.]

I recently attended a SANS Summit on Logging. Its not about making it impossible to overwrite logs... there's basically no way to do that. Every suggestion here pretty much has a reply to it about how to get around it. Its not a technical problem to solve, its a policy one.

Given a non-tiny operation, its fairly simple to reach compliance (IANAL). The group that runs the payment gateway is NOT the group that runs the centralized logging system. Use syslog-ng to send the logs to a central server. The payment gateway guys don't get access to the centralized logging server, at least not write access. If you want, store the logs in a DB, and give them read access. They'll still have local logs for troubleshooting and such anyways, so they don't really need it, unless they need to go farther back then the local server logs are stored. Backup the centralized logs regularly to tape, or whatever your backup setup is. If you're paranoid, store checksums in a separate area, email them out, whatever.

You can't make the logs unalterable. What you do is put policies in place to make sure that they are secure if the infrastructure you are logging is compromised, internally or externally. For example, the systems you are trying to protect don't need full access to your logging servers, port 514 (or whatever you pick) is enough.

It's HIPAA not HIPPA

[opkool]

Hi,

It's HIPAA not HIPPA.

See Wikipedia, among others:

http://en.wikipedia.org/wiki/Health_Insurance_Port ability_and_Accountability_Act.

Peace

PCI-DSS logging and transmission requirements

[einhverfr]

Ok, I am not saying that one shouldn't encrypt. I think one generally should, but here are the requirements and relevant sections paraphrased:

3.1: Store only what you must.
3.2: Do not store sensitive authentication data (CVV, CVV2, magnetic stripe) subsequent to authorization.
3.3: Mask the credit card number when it must be displayed whenever possible
3.4: Render PAN (Credit Card account number) unreadible anywhere it is stored. If this is not possible, see appendix B for acceptable compensating controls.

Note that 3.4 does apply to backup media, logs, etc. The simple approach is don't log credit card numbers. And don't store them in your MySQL database in plain text (as does OSCommerce in at least one configuration).

4.1, cardholder data must be encripted when transmitted across "open, public networks." Presumably corporate intranets are excluded from this requirement but I would just as soon encrypt everywhere.

4.2: Never send credit card numbers via email.

10.1: Establish practices that allow you to track any administrative or root action and associate that with each individual user. In other words, you must be able to show not only what root did, but also which individual did it. I suspect that restricting root to *one* person and giving others access to sudo would be sufficient provided that sudo -s and su are prohibited from being used.

10.5: Protect audit trails against unauthorized modifications. This does not mean write once. It simply requires that the media be "difficult to alter." However, periodically backing recent logs up to optical disks would likely be a good practice.

10.7: log data must be retained for at least one year, and at least three months must be available online.

I'm a PCI QSA - You've got it all wrong

[Riskable]

Let me first state that I'm a PCI Qualified Security Assessor. That means I am certified by PCI to perform audits and report back to banks whether or not a company is compliant or not. In other words, consider me authoritative on this matter.

When dealing with any PCI requirement the most important thing to think about is the INTENT. Is the intent of the logging requirements in section 10 of the PCI DSS to prevent anyone, anywhere, from EVER being able to modify log files? No! The intent is to prevent a compromised system from altering its own log files--hiding the fact that it has been compromised. As long as your logging solution handles this situation effectively you really don't have anything to worry about.

In my role as auditor I would never fail a syslog host just because it was writing to a standard ext3 volume. I *would* fault a company if their logging solution was poorly configured (insecure: say, running telnetd) or was write-accessible by the same admins that send all their log data to it (unless they were a small company--if you only have one or two admins there's only so much separation of privilege you can get away with). I'd also have problems with a syslog host that wasn't backing itself up on a regular basis (90 days online, 3 year archive).

If I were you I'd be more concerned with your logging system meeting the other requirements of the PCI DSS. If it is inherently insecure or fails to implement proper access controls (say, shared root account) who cares how the logging solution is configured?

Remember: Intent is everything. If in doubt, call your acquirer (i.e. your bank). They're the ones who ultimately have to decide whether or not your implementation is good enough anyway. The auditor just writes a report--the bank has to sign off on it.