Forensics Tool Finds Headerless Encrypted Files

gurps_npc writes "Forensics Innovations claims to have for sale a product that detects headerless encrypted files, such as TrueCrypt Dynamic files. It does not decrypt the file, just tells you that it is in fact an encrypted file. It works by detecting hidden patterns that don't exist in a random file. It does not mention steganography, but if their claim is true, it seems that it should be capable of detecting stenographic information as well."

Comments

Plausible Denial?

[telchine]

I'm am a citizen of the United Kingdom. Amongst many odd laws we have here, there's one that basically means that you can go to jail if you refuse to hand the police your encryption keys if they ask for them. The one saviour was Truecrypt's plausible denial. If they don't know you have encryption they can't ask for keys!

Now they do know I have encryption... ...and I've forgotten my password.

Can someone please give me tips on how to avoid dropping soap in the shower?

Re:Plausible Denial?

[wjh31]

practice holding soap between your cheeks, that should prepare you well.

Re:Plausible Denial?

[DeadDecoy]

Your computer is infected by a rootkit that dumps payloads on your system?
Sadly, this is my case where I have randomly named files on my system that I just cannot remove short of reinstalling the OS.
However, giving the extent of viruses, who's to say that you placed the content on your computer or even knew it was there.

Re:Plausible Denial?

[jroysdon]

I thought one feature of TrueCrypt was the ability to have two passwords. One password unlocks your "non-secret" data. The other password unlocks your "secret" data in a hidden volume.

http://www.truecrypt.org/docs/plausible-deniability

The point is both sets of data are stored in one big binary blob. It'll all look like one big fat encrypted mess. In fact, if you are not careful, your non-secret data can overrun your secret data.

To get around this "randomness" problem, after creating your non-secret partition, fill the partition completely with something (copy a few public domain books over and over until the partition is full). All the "randomness" will be gone with encrypted data. Then delete everything and put back in just the smallest amount of non-secret data you need to store in order to appear legit. The "randomness" is still there, as only the FAT entries are deleted, but all the encrypted data is still filling up that whole binary blob.

Now, create your secret partition and use it. Be sure to use it just short of the non-secret data's amount (as they fill from the opposite end), otherwise your non-secret partition will be corrupted.

This link helps with the graphics:
http://www.truecrypt.org/docs/hidden-volume

The one downside is that the non-secret side, if it fills up with too much data, will override your secret side. That's why your have backups and this is just for transport anyway, right?

Re:Plausible Denial?

[Lumpy]

you got it. It's called hiding in the noise.

Format your drive, now plug it in as usb and create a full size truecrypt encryption on it and fill it with junk.

now take the drive, delete that file and then use it as your new drive whatever. any encrypted files will be hidden in the noise of the background encrypted file that is in the blank area of the drive.

Re:Plausible Denial?

[Animaether]

"That's cute, sir - now give us the other password"
- "what other password?"
"for the hidden truecrypt volume"
- "what hidden truecrypt volume??"
"the one that's being referred to by half a dozen applications' most recently used files lists"
- "oh err.. that's uh.. another drive entirely"
"very well, then hand us that other drive"
- "err uhm.. my dog ate it?"

If you're really, really serious about these things, maybe you could work super-diligently to prevent leaving any clues as to that hidden volume's existence.. odds are something's going to bite you in the behind somewhere though.

Re:Plausible Denial?

[gurps_npc]

The tool claims to be able to detect patterns that indicate files. So if you give them your first password, they can look for said patterns within the first encrypted file, thereby displaying that a second level of encryption exists.

Re:Plausible Denial?

[PMuse]

...you can go to jail if you refuse to hand the police your encryption keys if they ask for them.

Interesting. Does anyone know if there are similar laws concerning assisting the police in non-digital searches? In the UK? In the States?

For example, suppose that a 9mm handgun was used to kill your husband. The police have records indicating that you own such a gun and they have your empty gun case, but your gun is missing. A ballistic analysis of your gun would be vital evidence, but you remain silent. A trial later acquits you of murder. Can the police charge you with Failure to Assist in an Authorized Search and send you to jail for not telling them where your gun was?

Re:Plausible Denial?

[Kjella]

The one downside is that the non-secret side, if it fills up with too much data, will override your secret side. That's why your have backups and this is just for transport anyway, right?

It has a protection option where you can enter the hidden password along with the normal password so the hidden partition will be protected, the outer container will be frozen on a write attempt to hidden data. I think it's unnatural that you must ensure that there's no data written to the end of the disk though, it leads to some peculiar disk format choices and so on. A better implementation would be more like a transparent file system layer, where the outer partition could write anywhere it wants and the encryption software would move any encrypted data already stored there. It'd make it more difficult to locate the header but maybe a pseudo-random sector based on password. That way the outer container could look really natural. Today they tend to seem so very staged which tends to bring you don't from plausible deniablitity to "you can't PROVE it" deniability.

Re:Plausible Denial?

[Randle_Revar]

>recently used files lists

strange, my cli apps don't seem to have that

Re:Plausible Denial?

[causality]

Your computer is infected by a rootkit that dumps payloads on your system? Sadly, this is my case where I have randomly named files on my system that I just cannot remove short of reinstalling the OS. However, giving the extent of viruses, who's to say that you placed the content on your computer or even knew it was there.

In a way that's pretty funny.

"Windows is insecure and has all of these virus problems."
"It's not a bug, it's a feature! This way you have plausible deniability."

Re:Plausible Denial?

[Fulcrum of Evil]

Can the police charge you with Failure to Assist in an Authorized Search and send you to jail for not telling them where your gun was?

In the US, you have no duty to assist the police, only a duty not to impede them. I think an analogous situation would be whether they could charge you for refusing to open the safe. Is remembering the combo the same as giving up the passcode? Is it possible self incrimination to do so? You aren't telling them things that implicate you, just how to get at things that might do that. What if you're a lawyer and there are client files in that safe/encrypted volume?

Re:Plausible Denial?

[RiotingPacifist]

The other problem is the fact your using truecrypt, used to be an indication that you were using stenography, it seams popular enough to get away with now.
What would be nice would be the ability to have multiple stenographic partitions in the same file, so you could setup schemes like: 5.7Gb porn, 100mb fake documents * 3, 100mb real docs. By even if they get the real docs they'll never really be sure.

Re:Plausible Denial?

[Lord Ender]

This is a great point. Suppose you're being investigated for tax fraud. The UK police order you to provide a password to your encrypted volume. You give the password to the dummy drive. They say "your registry lists S:\foreign accounts.xls as a recently-opened document, yet the password you provided opens a partition with no such file. You're using software which supports two partitions, therefore you must have a second password."

The legal system is not a mathematical proof, and I suspect this would be sufficient imprison you for failure divulge your encryption key (self-incrimination, we call it over here).

Re:Plausible Denial?

[Zumbs]

How about a third password that decrypt the non-secret part, but deletes the secret part?

Re:Plausible Denial?

[at_slashdot]

Use one-time pads :D

Re:Plausible Denial?

[piojo]

The tool claims to be able to detect patterns that indicate files. So if you give them your first password, they can look for said patterns within the first encrypted file, thereby displaying that a second level of encryption exists.

No, no, no... this isn't hiding an encrypted blob in another encrypted blob. This is just one encrypted filesystem, where some files can be unlocked with one password, and other files can be unlocked with another. As far as I know, there is no way to get at or analyze partially decrypted data.

Re:Plausible Denial?

[piojo]

How about a third password that decrypt the non-secret part, but deletes the secret part?

Data is normally backed up before it is analyzed, so this would not help--besides, this would depend entirely on the client program, because files can't delete themselves.

Re:Plausible Denial?

[FineWolf]

TrueCrypt hidden volumes... You give them the key to the main volume and store your data in the hidden volume.

Re:Plausible Denial?

[Amazing Quantum Man]

Simple. Make your password, "what hidden truecrypt volume?"

Re:Plausible Denial?

[Amazing Quantum Man]

When I bought my 8GB Corsair Flash Voyager, it came *bundled* with TrueCrypt.

Re:Plausible Denial?

[FatdogHaiku]

...this would depend entirely on the client program, because files can't delete themselves.

Wouldn't the client program have to be running to insert the password?

Re:Plausible Denial?

[Trapick]

I think an analogous situation would be whether they could charge you for refusing to open the safe.

This seems analogous, but really isn't from the court's point of view. Cracking a safe is trivial - locksmith with torch and tools can open it in a day - the security of a safe is even given that way - time with torch+tools. And it's generally in hours. Cracking well encrypted drives, however...little bit different.

Re:Plausible Denial?

[amRadioHed]

...because files can't delete themselves

Since when?

pete@lenore ~/tmp $ ls
rm*
pete@lenore ~/tmp $ ./rm rm
pete@lenore ~/tmp $ ls
pete@lenore ~/tmp $

Re:Plausible Denial?

[shutdown -p now]

If you are actually seriously using TrueCrypt so that the NSA (or law enforcement in general) won't get ya, you'd be an idiot to do so from Windows, or even your typical desktop Linux. I'd probably make a separate Linux (or BSD) install just for that, with home directory mounted in ramfs by default. Then make an image of its clean untainted state, and then everytime I need to access the encrypted drive, dd the image to a USB flash stick, boot from that, and only then mount the TrueCrypt volume and work with it. Once done, `shred` the stick.

Re:Plausible Denial?

[Joe U]

And your non-virtual OS's swap file? You can turn it off, but that'd raise big suspicions.

Like what? Having 4GB of RAM? I have no swap on some systems, don't need it, why should I thrash my HDD?

Re:Plausible Denial?

[Jane Q. Public]

That might work in the UK but not in the United States. Even the recent ruling that got all the bad press made it clear that they could only demand the key because they already KNEW the data in question was in the encrypted file.

In the U.S., they still can't demand a password unless they KNOW there is illegal material hiding in a SPECIFIC encrypted file. Suspicion is not sufficient.

Re:Plausible Denial?

[piojo]

That's cute. Another way is to start a file with "#!/bin/rm", and then it auto-deletes. But it still needs to be opened in a certain way in order to be deleted...

Re:Plausible Denial?

[piojo]

...this would depend entirely on the client program, because files can't delete themselves.

Wouldn't the client program have to be running to insert the password?

It depends. If the client program is TrueCrypt, I am sure that the version that a forensic examiner uses will not contain the deletion code. TrueCrypt is open source, after all--there is no reason why the government's version will contain features intended to destroy data.

Re:Plausible Denial?

[el americano]

"...an indication that you were using stenography."

They can always just ask the stenographer if she did any work for you, and then she rats you out.

Lesson: Don't use stenographers. Typing is fast enough.

Re:Plausible Denial?

[HTH NE1]

This identifies the presence of encrypted data. It does not establish providence.

Don't admit to the encrypted data being yours, or even to knowing about it.

How can you provide decryption keys for encrypted data that isn't yours?

It helps if your drive was bought off eBay (or better yet in an anonymous trash heap), is used as part of a RAID, and the encrypted data is in the excess space not usable by your striping method.

Unless they can determine the age of the most recent writes. Your system then better have some back door open and remote activity coinciding with those modifications.

Re:Plausible Denial?

[Jane Q. Public]

That still doesn't answer the question, however. This does (IANAL, but I have some familiarity):

If there is a warrant for the contents of the safe based on probable cause, or other probably cause (they saw you put the gun in there), then a court can subpoena the password or combination. But there must be probable cause. Under no circumstances can a court legally subpoena your safe combination without it. Suspicion is not sufficient.

In a recent court case, the court ruled that the defendant must supply the password to an encrypted file on a computer siezed by U.S. Customs agents. However, the details of the circumstances are very important. The man initially gave the customs officials permission to look at the contents of the computer. At that time, one of the customs agents found some depictions of child pornography (apparently in encrypted storage that had been unlocked). It was only later that the man encrypted the file(s). On this basis, the court stated that since they KNEW that illegal file(s) were contained in the encrypted file, it was within the power of the court to demand the password.

It is important to note that according to the court, if they had not already known that illegal material was there, they would have been powerless to demand the password.

Re:Plausible Denial?

[FilterMapReduce]

What we really could use is a distro meant specifically to prevent this this, with (among other security features) default configurations that don't save any data about what your applications have been doing. Perhaps Paranoid Linux, if it matures.

Re:Plausible Denial?

[Sancho]

Why not just dd if=/dev/urandom of=/dev/yourhardriveblockdevice?

Re:Plausible Denial?

[Lord Ender]

My understanding of the ruling in the US was that the guy was forced to give up his key only because he initially cooperated with the investigation.

We still have a right against self-incrimination in the states. UK citizens have no such right.

Re:Plausible Denial?

[amRadioHed]

It's not meant to be cute, I don't know what you are getting at. What certain way does it need to be opened to allow this?

Re:Plausible Denial?

[adolf]

Because the noise would be too random?

Re:Plausible Denial?

[geekmux]

Simple. Make your password, "what hidden truecrypt volume?"

Dammit man! Now I've got to change my password.

Cripes, every time I come up with a good one...

Re:Plausible Denial?

[Fjandr]

You invoke your right to remain silent. In the case of a safe, their option is to crack it. In the case of an encrypted file, their option it also to crack it. They cannot compel you to speak in order to assist their investigation, unless you are in the military and being investigated by the military.

Re:Plausible Denial?

[piojo]

Okay, sorry. What I meant was that if a file is read, it cannot delete itself. The case you mentioned is if the file is executed--crypto-files are generally never executed. Truecrypt (or other forensic software) just writes and reads to it, and the file cannot force truecrypt to do something (like delete it).

Re:Plausible Denial?

[Jane Q. Public]

It wasn't his cooperation per se. This is a common misconception. You can invoke the 5th Amendment at any time, even if you have previously "cooperated".

The difference is that they found the illegal files while he was cooperating -- so their find was legal, and they knew the files were there. The judge ruled, in essence, that they already knew there was a violation of law, and where it could be found. So his refusing to relinquish the password was tatamount to obstruction.

Even if he had previously cooperated, if they had not found illegal material, they would still not be able to demand his password if he refused to give it.

Re:Plausible Denial?

[Fulcrum of Evil]

I'll just point out that we're also discussing UK, where the 5th is not a valid defense.

Re:Plausible Denial?

[PReDiToR]

After watching too much TV and reading not nearly enough books I gather that there are circumstances where this privilege can be maintained by having a third party open the safe who has been sworn under an NDA for anything that isn't part of the searched for items.

I'm sure that someone could confirm this who is A) American and B) a lawyer. I'm neither.

Re:Plausible Denial?

[Fjandr]

I figured anyone who was paying attention would realize my comment applied only to the US side of the discussion, but I suppose it's good to point that out specifically. This is Slashdot after all. :)

Re:Plausible Denial?

[amRadioHed]

No, I still don't think that's the case. POSIX unlink works pretty much regardless of the state of the file. I just wrote a little test program that opens itself O_RDONLY and unlinks itself while the file descriptor is still open. The executable deletes itself with no errors.

Re:Plausible Denial?

[JRIsidore]

Or simply use a Live CD.

Re:Plausible Denial?

[hesaigo999ca]

LMAO!

Re:Plausible Denial?

[Joe U]

If you're really paranoid, you create a hidden os partition in truecrypt, then create a VM in the hidden os, then inside the vm in the hidden os you create a truecrypt volume and inside that you create a hidden volume.

Don't make all the passwords 'l33t h@x0r'.

Re:Plausible Denial?

[skeeto]

Then you have to hire guards to watch your computer, flash drive, and "clean image" at all times, then be able to trust those guards. Otherwise, one or more of those can be compromised in order to leak your key.

Re:Plausible Denial?

[zipherx]

Thats gota be the most paranoia approach i have heard of haha ;)

Re:Plausible Denial?

[Hurricane78]

That's what 2-factor authentication is for.

Have cheap USB stick, and put the key for your encrypted stuff on it. But encrypt *that* key with your password. Then make a second one, using a different password, and store it somewhere really far away and really safe. (Something like burying it in a box at a specific location.)

Now get a giant hammer or melting pot of acid or something, and as soon as the cops kick your door is, destroy that stick completely with it.
Then tell them, that this was your key, and that your password is useless without it. Tell them "Heck, I will even give you the password right now.". Make it clear that fixing that pulverized or liquefied stick is the only way to ever get to that data. (Except for brute-force of course.)

Now the only problem is, that they could beat you until you reveal that there is a second stick. (You could read anti-torture manuals of the CIA to fight it a bit.)
But the chance that this happens is much smaller.

About the jail: Tell them you have 1. AIDS, and 2. some really nasty STD with pox and crap all over. And hope they'd not hit that anyway. :P

Re:Plausible Denial?

[Jeruvy]

Soap on a rope.

Re:Plausible Denial?

[piojo]

I just meant that the file that needs to be deleted is never executed (so it can't delete itself). With an encrypted file, gpg, truecrypt, dm-crypt, or some other program is executed. That program will most likely not be configured to delete the encrypted file, under any condition.

Patterns?

[causality]

It works by detecting hidden patterns that don't exist in a random file.

I should first say that I'm rather ignorant about encryption but I hope someone will be able to explain this. I was under the impression that any sort of good-quality encrypted data is indistinguishable from completely random data. That seems to directly contradict the ability to determine whether a volume contains encrypted data by means of locating patterns. Is this really a contradiction?

Re:Patterns?

[feld]

If you have a file and encrypt it with an algorithm, it will become encrypted and look like nonsense.

If you take another copy of the original file and encrypt it, it will become encrypted and look like nonsense... but still be identical to the first encrypted file.

Encryption doesn't = random. It just means indecipherable.

Re:Patterns?

[42forty-two42]

And how many completely-random files do you have on your computer?

Re:Patterns?

[Firethorn]

The fact that there's order in the encrypted information doesn't change the fact that, to an outside observer that doesn't know the original information or the key can't tell the difference between the encrypted information and true random noise. That's part of the point.

If they can tell that it's not random, that's a start on cracking the encryption and gaining the original information.

Re:Patterns?

[causality]

And how many completely-random files do you have on your computer?

One, and a second file that's pretty close. /dev/random and /dev/urandom.

Dear mods, that's meant to be facetious. Some of you seem to be a little trigger-happy so you won't understand why I shouldn't have to explain that.

Re:Patterns?

[gnick]

Right - If your encrypted file is distinguishable from noise, your algorithm is broken. Maybe they were using this to find things encrypted using the ROT-13 or XOR cipher? I find it very hard to believe that this thing could distinguish between an AES encrypted file and a file of the same size full of random bits. Its site claims that it, "Detects Encrypted Files, including TrueCrypt", but my guess is that they just look for headerless files full of noise.

They give no details about how they do it other than "There actually is a pattern to it. You have to know how to extract that pattern.", but I'm still calling snake oil.

Re:Patterns?

[Andy Dodd]

Actually, if you use the wrong block cipher mode, it's easy to distinguish between an encrypted file and random noise. AES-256 encrypts 128 bits of data at a time (with a 256-bit key). If you use the same key and the same block of data (ECB mode), you get the same output and can determine that there's something there.

If you modify each block with some known quantity that is different from block to block, then the output becomes much less patterned. For example, Counter (CTR) mode XORs or adds an increasing count to each block of cleartext, so that if you have two identical blocks of cleartext, the output is very different. Cipher Block Chaining (CBC) takes the encrypted output of block N and XORs it with the cleartext of block N+1 before encrypting that block.

Re:Patterns?

[AnotherBlackHat]

I knew testing that hardware random number generator was a bad idea.

Re:Patterns?

[SerpentMage]

No...

Encryption is supposed to indicate random noise. But encryption in a grand sense is about writing, and rewriting data.

Let's say I have data which is number 2...

My key is 4,4,4

My encryption is:

Value1 + number -> * Value3 -> - Value4

So it is 4 + 2 * 4 - 4... And I get some number...

I do this multiple times and I get a bunch of other others. Put all of these numbers together and I get what looks like giberish (assuming the algorithm is good enough).

But here is the problem, underneath the data is a pattern. And the calculations are a pattern, as a result a pattern emerges. The pattern is called human language.

For example one strategy for passwords is to use random data. Then you have no patterns because the resulting encryption is random noise.

To give you an understanding, I deal with random numbers and I cannot use a computer based random number generator because they generate patterns.

I subscribe to a random number service which is connected to a quantum lab and space noise...

Now, to say if it is not random you can start cracking it. Guess what you are right, but what if your numbers have 500 hundred thousand places. Going in reverse to figure out what those numbers are is actually pretty hard. That is why you have these issues of finding prime numbers...

Re:Patterns?

[Andy Dodd]

I think it goes with the whole "Innocent until proven guilty beyond a reasonable doubt" mantra.

If a file is indistinguishable from random noise, then a court can't prove that it's encrypted data.

That said, in reality they can make your life a living hell.

Re:Patterns?

[SerpentMage]

Now I would actually agree with them. I think you could find out.

Though the way to fool this system quite easily is make the entire drive appear like an encrypted file. That way they can't distinguish between where it starts and where it ends.

Then when asked to pull up the data they can't prove you one way or the other... (and you can pull up non-critical information)

Re:Patterns?

[Jah-Wren Ryel]

Dear mods, that's meant to be facetious. Some of you seem to be a little trigger-happy so you won't understand why I shouldn't have to explain that.

Make your joke and take the moderations like a man.
If you are going to explain that it is a joke, you might as well not bother in the first place since explaining takes away all the fun.

Re:Patterns?

[MeanMF]

This is why it's a good idea to use a different initialization vector every time you encrypt something. Ideally your encryption software will do this for you automatically.

Re:Patterns?

[thehickcoder]

Good point. My guess is that is how this tool actually works. It relies on the assumption that any statistically psuedorandom files (or maybe even partitions) must be encrypted, since every other file will contain some sort of pattern.

Re:Patterns?

[geekboy642]

Another thing would be Truecrypt's refusal to overwrite certain parts of that "random" data inside the not-hidden container. Gives it away that it's protecting the integrity of a hidden container.
Why do people constantly make this mistake?
TrueCrypt cannot know a hidden partition exists, *unless* you enter the inner volume password. It will cheerfully let you write right over the inner volume without so much as a by-your-leave, if you only give it the first password. It is true deniability, assuming this tool can't distinguish "encrypted blank space" and "encrypted data".

Re:Patterns?

[Dan Ost]

Another thing would be Truecrypt's refusal to overwrite certain parts of that "random" data inside the not-hidden container.

Actually, according to several other posts on the subject, Truecrypt will happily overwrite the hidden container if you're not careful.

Re:Patterns?

[DriedClexler]

Perfectly compressed data is indistinguishable from random noise. Well-compressed data is "close" to random noise. So I suspect any file type I have that uses a good compression method (jpeg, most mpeg codecs) looks close enough to completely random.

(Be nice, I'm just an information theory hobbyist.)

Re:Patterns?

[Lehk228]

well my /b/ folder is pretty random

Re:Patterns?

[marc.andrysco]

Well, having taken a graduate class on crypto, you're guaranteed to leak some kind of information unless your using a system with perfect secrecy, but that requires that your key be as long as your message. With modern cryptosystems, short keys (short compared to the message) are used, so some type of information is being leaked, although it is greatly obfuscated through the algorithm of the cryptosystem. I don't believe you can say for certainty that no information is leaked, but the premise is that a robust system ought not to leak noticeable information unless the attackers discovered the key. My professor said that designing a system that fit this description is closer to an art since the mathematics says some information should be leaked.

Disclaimer: This is what I know from a limited amount of experience with, so a lot of the details are fuzzy or incomplete. Please feel free to correct me (politely).

Re:Patterns?

[Threni]

I thought the first step of most encryption systems is to compress what you're encrypting, so it's more random, and there's less to encrypt!

Re:Patterns?

[Sloppy]

If you use the same key and the same block of data (ECB mode), you get the same output and can determine that there's something there.

So, modify your file deleter or filesystem initializer to occasionally spew the same random junk into unused blocks. Now your truly-not-encrypted free space looks just like maybe-it's-encrypted space.

Whatever this "forensic tool" picks up on, can be duplicated by noise makers to create false positives.

Re:Patterns?

[Kjella]

That's called a known cleartext attack. If they already have the original file then the point of encryption is moot.

1. It's usually called a "known plaintext" attack.
2. Detecting patterns in ECB mode encrypted data is not a known plaintext attack.
3. Known plaintext attacks are most definately not moot.

A known plaintext attack means that you can derive the key or some intermediate to decrypt other data encrypted with the same material, and is highly useful. For example, you could send someone a mail, an instant message, upload a file to a server or whatever and if stored on an encrypted disk you have a known plaintext. If that'll let you figure out the key, big uh-oh. I actually used this on some encrypted (standard password protected) zip files, they have a known plaintext attack. Basicly I had one zip file with contents I already had, and other zip files with contents that I didn't have. But from having both plaintext and ciphertext from one file, I could decrypt all the other files too.

Re:Patterns?

[slinches]

It relies on the assumption that any statistically psuedorandom files (or maybe even partitions) must be encrypted, since every other file will contain some sort of pattern.

Wouldn't formatting the empty space on a drive with similar pseudo-random data create false positives?

Re:Patterns?

[maxume]

Just toss any suspects at decent decompression and image decoding programs and you can filter out quite a bit.

Re:Patterns?

[jcochran]

Only if the encryption method used is broken.... badly....

If you have a file and encrypt it with an algorithm, it will become encrypted and look like nonsense.

True. After you encrypt it, it looks like nonsense.

If you take another copy of the original file and encrypt it, it will become encrypted and look like nonsense... but still be identical to the first encrypted file.

True and False. It's true if you use Electronic Code Book with your encryption. And that's reason EBC is not recommended for use. If you use Cipher Block Chaining, you can encrypt the same file thousands of times and every time, the encrypted output would be different.

Re:Patterns?

[daveime]

Yup, I was going to comment something similar.

Compression removes entropy, the better the compression method, the more entropy is removed and the more random the data appears.

So surely the best way to go is encryption followed by compression (or vice versa). This guarantees the file is indistinguishable from random data. Of course you'd have to strip out any headers that the compression program itself might leave etc etc ...

Re:Patterns?

[evanbd]

If you are going to explain that it is a joke, you might as well not bother in the first place since explaining takes away all the fun.

Make your snide remarks and take your moderations like a man.

If you're going to try to avoid the downmods and explain your reasoning, you might as well not bother in the first place since explaining takes away all the fun.

Re:Patterns?

[DriedClexler]

Don't you mean compression increases the entropy? Since it makes subsequent bits harder to predict?

Re:Patterns?

[Beryllium Sphere(tm)]

You're absolutely right. In fact some secure random number generators are based on the output of block ciphers. ANSI X9.17 isn't a pure example, since it uses a random input, but Fortuna simply uses a cipher in counter mode.

Any statistically significant pattern in a cipher's output would be considered a flaw in the cipher. Unless, of course, these people have just "discovered" that ECB mode is bad.

Re:Patterns?

[chill]

Make your joke and take the moderations like a man.

Wouldn't the joke be on you if the the GP was the one female posting to Slashdot?

Re:Patterns?

[AliasMarlowe]

It works by detecting hidden patterns that don't exist in a random file.

I should first say that I'm rather ignorant about encryption but I hope someone will be able to explain this. I was under the impression that any sort of good-quality encrypted data is indistinguishable from completely random data. That seems to directly contradict the ability to determine whether a volume contains encrypted data by means of locating patterns. Is this really a contradiction?

It rather depends on the encryption algorithm. ECB is fast, and much favoured where performance is critical (maybe a file system counts). However, it merely transforms the patterns in the cleartext data into other patterns, and a periodicity based on key length may also be discernible if the key length is only a small fraction of the data length (as is usually the case).
On the other hand, if a better algorithm is used, such as CBC or cypher streaming with unique initial states, then there is no periodicity for any nontrivial key, and the encrypted data is essentially random (unless it repeats in a pattern related to the key). Unfortunately these methods require decryption of all preceding data in a file just to read a single byte, so there can be a significant performance hit.

Re:Patterns?

[42forty-two42]

However most compressed formats have magics or other headers to identify them. Truly encrypted data won't - so just do a pass looking for 'random' data, and filter out anything that has a known header.

Re:Patterns?

[42forty-two42]

Use noatime. It improves disk performance too, so plenty of plausible deniability there.

Re:Patterns?

[AliasMarlowe]

I forgot to mention that Wikipedia is your friend: http://en.wikipedia.org/wiki/Block_cipher_modes_of_operation has a good example of the weakness of ECB relative to CBC etc. and shows exactly the kinds of pattern visible in ECB ciphertext of patterned plaintext.

Re:Patterns?

[supersat]

It's usually possible to distinguish an encrypted sequence from a random one, but with a good cipher, that requires more work than just brute-forcing the key. You can also do things like turning a block cipher into a system of equations, but again, with a good cipher, solving that system of equations is harder than brute-forcing the key.

Re:Patterns?

[againjj]

It works by detecting hidden patterns that don't exist in a random file.

I should first say that I'm rather ignorant about encryption but I hope someone will be able to explain this. I was under the impression that any sort of good-quality encrypted data is indistinguishable from completely random data. That seems to directly contradict the ability to determine whether a volume contains encrypted data by means of locating patterns. Is this really a contradiction?

You are basically correct, but there is more to the picture.

First, properly encrypted data does pass randomness tests. However, arbitrary data generally does not. Basically nothing on your disk contains data that will test random. The thing that will come closest is compressed data, since perfect compression produces something that can not be compresses further (because there are no patterns exploitable to compress). However, no compression is perfect, and compressed data has metadata, so that is also detectable. Free space on disk also has this property, unless overwritten with random data after deletion.

Further, encrypted data generally has other properties. One common property of encrypted data is that it is often padded out to the nearest multiple of some particular size. Truecrypt, for example, apparently always produces volume files of some multiple of 512.

Last, note that the detection merely purports to detect an encrypted block of data, not reveal any information about the data itself.

To analogize, imagine a large graphic, say 10,000x10,000 (your hard disk), and somewhere on that graphic is a 100x100 block of perfectly 50% grey (encrypted data). That block will stick out like a sore thumb, unless the graphic contains a bunch of blocks of 50% grey throughout (other random or encrypted data).

Re:Patterns?

[poopdeville]

His point is that you have no evidence to back up your assertions regarding his character or motives. Ergo, you are talking shit.

A "real man" doesn't care what you think, let alone do what you say.

Still, he got trolled. You both lost this round.

Re:Patterns?

[Onymous Coward]

Idiotic moderation is also a bit of a killjoy.

Re:Patterns?

[geekboy642]

"The right way to do this is simply to have the encryption code silently destroy access to the data when given a special, secondary password."

Have you never heard of "dd"? Bit-for-bit copying of an entire drive renders your special booby-trapped key completely pointless. Who are you trying to defend yourself against, Inspector Clouseau?

Re:Patterns?

[Firethorn]

if you use the wrong block cipher mode, it's easy to distinguish between an encrypted file and random noise.

Maybe I should have said 'properly encrypted'. If you're using the 'wrong' cipher mode, or a weak encryption, you are indeed going to be leaving patterns - patterns which will assist in breaking the code.

The 'ideal' encryption, a one time pad, will leave a result that can't be told from random noise of the quality the pad was made from.

That's the goal. Other encryption systems that are more complicated mathematically(OTP by default is a simple XOR), but with shorter keys, should imitate this ideal, and generally quite well.

Re:Patterns?

[Firethorn]

But here is the problem, underneath the data is a pattern. And the calculations are a pattern, as a result a pattern emerges. The pattern is called human language.

I think you're getting a bit confused here? The 'pattern' in my mind is the data - whether it be a text, a picture, or a multimedia stream. The ciphers today only see a stream of bits.

Many encryption programs will zip/compress text files before encryption due to the extra order caused by large numbers of zeros, and relatively low number of patterns per character.

To give you an understanding, I deal with random numbers and I cannot use a computer based random number generator because they generate patterns.

I'm fully aware of those. Part of the reason for those patterns is the encryption isn't perfect - just good enough that it'd take forever and a day to find the original data if you lack the key.

Still, would you be able to tell the difference between a file full of flawed computer generated random numbers and an encrypted file?

Re:Patterns?

[Jane Q. Public]

Using XOR with a "sufficiently" random key will not yield a noticeable pattern. The only way XOR will reveal a pattern is if a poorly chosen key is used.

Re:Patterns?

[Jane Q. Public]

Someone else here had the same idea and it may be a good one.

Re:Patterns?

[NotBornYesterday]

If they do ...

dd if=/dev/yourharddrive of=/my/drive/copy.dd
cp /my/drive/copy.dd /my/drive/copy2.dd
mount -r -o loop /my/hard/drive/copy2.dd /mountpoint

... your wild "killalldata" application and tamper-resistant hardware gets you nowhere, unless your special hardware doesn't support block-level access. You're not planning on turning off block-level access with you tamper-resistant hardware, are you? Because that would really piss off your average filesystem.

Re:Patterns?

[daveime]

s/remove/increase/

My bad, just woken up.

Re:Patterns?

[Jah-Wren Ryel]

His point is that you have no evidence to back up your assertions regarding his character or motives.

Right, because people write "dear mods" to give moderators a special education intended only for them because they are such special people with educational needs distinct from everybody else on slashdot, not because they'll mod him down out of their own ignorance unlike non-moderators who can't act on their own ignorance.

Re:Patterns?

[Garse Janacek]

If you take another copy of the original file and encrypt it, it will become encrypted and look like nonsense... but still be identical to the first encrypted file.

Actually, with many (most?) encryption schemes used in practice, this isn't the case. Google "known plaintext attack."

Re:Patterns?

[gnick]

That's true as long as your key is the same size as your message. Otherwise, applying an XOR cipher won't make the data look much more like noise than it started out as.

Imagine a really short XOR key - 1 bit. The data will be either identical to the original, or an exact inverse. In either case, it is exactly as close to noise as it started, it just may be harder to notice as data to the naked eye - No matter how random your key was. Now think forward to a 2-bit key - Better, but not great (or even good). Until you get to a 1-time-pad where key_length==file_length, you've not ensured noise. With XOR, the randomness of the key is a very minor point unless you're really far from random.

Side note - If you're going to use an XOR cipher, apply it 3 times for added security. For a small fee, I'll provide you with an algorithm that can accomplish this nearly as quickly as applying it once.

Re:Patterns?

[Jane Q. Public]

"That's true as long as your key is the same size as your message."

Of course. That is a given. That is the strength of the one-time pad, and also one of its weaknesses... you need a key as long as your text.

Re:Patterns?

[Jane Q. Public]

"Side note - If you're going to use an XOR cipher, apply it 3 times for added security. For a small fee, I'll provide you with an algorithm that can accomplish this nearly as quickly as applying it once."

You mean like Rot39? :o)

Re:Patterns?

[poopdeville]

That's so truthy!

Re:Patterns?

[Repossessed]

The encrypted stuff being random might be the point.

Stuff on your hard drive isn't random, it'll either be 0s left over from its new state, or lots of pieces of files that were written and then deleted. So look for large chunks of truly random data, thats probably an encrypted file.

Re:Patterns?

[DMUTPeregrine]

Compress -> Encrypt IS used instead of Encrypt -> Compress, but not for that reason. Encrypted data is effectively random, and thus very, very hard to compress. Compressed data is just as easy to encrypt as uncompressed. So to save space you MUST compress before encrypting.

Re:Patterns?

[TechyImmigrant]

To give you an understanding, I deal with random numbers and I cannot use a computer based random number generator because they generate patterns.

I understand that you are not using a good enough PRNG. A high cryptographic quality PRNG seeded with high quality entropy will meet your needs.

Re:Patterns?

[sFurbo]

Wouldn't your example be a chosen plaintext attack? Or perhaps I'm just confused about different uses of "you" in your post :-)

Re:Patterns?

[calmofthestorm]

Mmmmm...salt...

Re:Patterns?

[Andy Dodd]

Where did I say ANYTHING about having the original file?

ECB mode does make a "guessed plaintext" cryptanalysis attack more feasible. For example, if you encrypt empty space (all zeroes) with ECB, an attacker will see lots of repeating ciphertext and guess that it's probably the ciphertext for a zero block.

Now for at least one value, you know the cleartext and ciphertext, which in theory makes it easier to discover the key. (With AES, it's still Really Damn Hard, and at best makes it easier to know you guessed right. With weaker block ciphers it can kill you - For example, some attacks against CSS used "guessed plaintext" attacks I believe, where they took advantage of a few things:
1) The CSS block cipher was weak and you gained a lot of information about the key by having a block of both plaintext and ciphertext
2) Black MPEG-2 frames encode very predictably
3) Nearly all movies started with a few black frames

You never had the original cleartext file, BUT you could use statistical analysis to guess the cleartext for some blocks with high certainty and get you closer to knowing the key.

Re:Patterns?

[SuperJ]

This seems to be what everyone missed in this post. The article is just some marketroidish stuff. By saying that they detect hidden patterns, they're just saying they look for random data. All this product is an entropy scanner. Scan the disk, find areas with high entropy, mark it as random.

Re:Patterns?

[Fnord666]

I subscribe to a random number service which is connected to a quantum lab and space noise...

Space noise? Really? That's not good. You do realize that with the change in the rate of expansion of the universe, the amount of doppler shift is going to vary and possibly even change in sign. This could mean that in a very short time your experiment and conclusions might not be verifiable.

Umm...

[drakaan]

s-t-e-g-a-n-o-g-r-a-p-h-y...not stenography.

Re:Umm...

[Daimanta]

ssshhh, the "ga" is secretly embedded through steganography

Re:Umm...

[gnick]

No, stenography. Their software can detect files that are written in shorthand. I find that much more plausible than the idea that it can tell strongly encrypted data from noise.

Re:Umm...

[Zapotek]

Dunno, if the hidden data is 30 column wrapped that could be stenography[1].

Steno = narrow
graphy = writing
Greek /. readers I expect a funny mod up. xD

Re:Umm...

[mcrbids]

Uh, steganography is spelled like this:

I was understanding that we are generally aware of nothing where we are desperate to have fixed, at least not immediately.

Sans the bolds, that is.

Re:Umm...

[Kjella]

Ohh, and I thought it was just stenography shorthand. Not only secretly embedded, it has plausible deniability too.

Re:Umm...

[NoobixCube]

I never trusted that Stenographer Fred from Science Court... At least I think his name was Fred... Maybe Ted?

That's STEGANOGRAPHY!

[omnichad]

add a "ga" to every mention of stenography in the summary. Unless he means that encryption uses shorthand.

Re:That's STEGANOGRAPHY!

[wjh31]

compressed and encrypted?

Re:That's STEGANOGRAPHY!

[idontgno]

Our groundbreaking software can detect the presence of SHORTHAND* and allow law-enforcement decryption of this nefarious data-hiding technology!

*Currently can detect Gregg, Pitman, Teeline, and Speedwriting. Also detects the presence of steno pads and stenotype machines.

Re:That's STEGANOGRAPHY!

[mfnickster]

Easy, I'll just encrypt using a one-time steno pad!

Re:That's STEGANOGRAPHY!

[calmofthestorm]

They can see through that too, using the fluoride in our teeth.

Vindicated!

[AKAImBatman]

Or "How I learned that you can't fight information theory".

A few years ago, I recall arguing with someone here on Slashdot about this very issue. My take on it was that stenography could never be completely successful because there would always be a pattern sticking out from the file. The other poster argued that truly encrypted data should be indistinguishable from white noise. I pointed out that a) stenography disrupted the image coloring and therefore should be detectable by looking for irregularities and b) encrypted information (short of a one time pad, which is the only way to get true noise) has an underlying structure in the data operated on. Since the key repeats the transformation for the length of the data, the distribution of the codes cannot be guaranteed.

i.e. Encrypted information will stand out as structured data.

Which only makes sense when you think about it. Information theory doesn't mess around. You cannot destroy information. (The black hole experiments confirmed that.) Thus the structure of the information will remain, no matter how much you try to obscure its existence.

We never reached an agreement, so I guess we'll have to let this article finally settle it. l-)

Re:Vindicated!

[AKAImBatman]

"Steganography." Excuse me. I seem to have repeated an error.

Re:Vindicated!

[Burkin]

I wouldn't use a slashvertisement as the basis of my argument if I were you.

Re:Vindicated!

[gnick]

You may be able to convince me that steganographicly hidden could be detected by looking at a compressed image and detecting areas that would not normally be produced by whatever compression algorithm is employed (jpeg, gif, whatever). But that's an entirely different game than looking at a file that's completely white noise and deciding whether or not it's encrypted.

Also, maybe I'm just not deep enough, but I don't understand at all what this has to do with information being destroyed or black holes. This is a matter of information being detected, not destroyed.

Re:Vindicated!

[AKAImBatman]

We'll see how far they go with it. If their tool works as well as they claim, it wouldn't be the first time engineering has preceded the scientific theories behind them. Obviously testing the tool helps. If it finds TruCrypt data, that's not a good sign for the theoretical randomness of data. Next interesting question is: How far will TruCrypt be able to go to prevent the tool from detecting data? Will we see an arms race of encryption hiding vs. detection? If so, who will win?

My own feeling is that data smaller than the size of the key may be immune to detection. But larger quantities of data is going to show a pattern. The trick to detection is in understanding how, when, and why that pattern shows up. As I mentioned above, Information Theory already offers a pretty compelling answer for the "why".

Re:Vindicated!

[Burkin]

If it finds TruCrypt data, that's not a good sign for the theoretical randomness of data.

No, it just means there is a flaw in TruCrypt's implementation.

Re:Vindicated!

[Hatta]

encrypted information (short of a one time pad, which is the only way to get true noise) has an underlying structure in the data operated on.

The digits of pi have an underlying structure. If you have a way to distinguish an arbitrary stretch of pi from truly random data, I suspect you'll win a Fields Medal.

Re:Vindicated!

[pyite]

i.e. Encrypted information will stand out as structured data.

So you really believe that if I take a non-random stream and encrypt it with AES in CBC with a random key and random IV (both of which I can easily obtain as I can generate 2 * 128 bits of true random data with 256 flips of a fair coin) that you will be able to distinguish the resulting ciphertext from true random data?

I find that, and this company's claims, *highly* unlikely.

Re:Vindicated!

[AKAImBatman]

Also, maybe I'm just not deep enough, but I don't understand at all what this has to do with information being destroyed or black holes. This is a matter of information being detected, not destroyed.

The point I'm making is that in order for the original information to be undetectable, it must be destroyed. Destroying information is the process of making it non-retrievable. i.e. Completely random. Of course, there's no such thing as completely random in our Universe. There's a probabilistic nature to things, but even the "white noise" from, say cosmic radiation, carries information about its origins. It just happens to aggregate well enough to appear completely random. Then some space traveler interested in the origin of an encrypted packet reads the distribution and uses a database of celestial events to fingerprint Sol vs. Sirius.

The point is, the data is there and it is detectable. My opinion is that we're in an arms race to develop methods of making data appear undetectable in the short term, but unless we can destroy the original data (which we can't without making it unrecoverable) the data will always show structure. Or perhaps more to the point, "random" data on a hard drive is of a particular type of unstructured structure. (Whether it be from cosmic rays or a PRNG used to clear it.) We just need to detect the apparently unstructured data that's structured differently. :-P

Re:Vindicated!

[AKAImBatman]

I suppose one other point that's coming out of my rant that I should probably vocalize is that the assumption of "randomness" is an assumption made in a vacuum. If you have nothing else to compare the randomized data against, then the data will be invulnerable to an entropy check. The white elephant is that nothing is truly in a vacuum. One can ask the question, "What does typical 'random' data on a hard drive look like? What does this data look like?" If there is any difference in, say, the distribution of the random values or the entropy of particular values, it can be quite easy to detect the encrypted info.

That I guess is what I'm really getting at. The theory assumes a vacuum. There is no vacuum. (Or spoon for that matter. :-P)

Re:Vindicated!

[Cillian]

If you can have/use a larger key than your data, then you can XOR it with a one time pad, which *is* undetectable and unbreakable

Re:Vindicated!

[internerdj]

Could you not instead make even the white spaces look structured even structured in several possible different ways to hide the real data from the white space? I'm not a cryptography person, but it seems to hide the structure you are going to mix up the data throughout the disk, but a used disk is going to have random junk in the white space but the random junk will be related in some previously used block. So an encrypted disk is going to have its "junk" more randomly mixed than a non-encrypted disk. If your information "fit" the junk in its local area then it wouldn't look like it is encrypted, it would just look like data that hasn't been reallocated for something else yet.

Re:Vindicated!

[dgatwood]

I would think that you would merely need a random one-time pad that is as long as the data you are trying to encrypt. I predict the next evolution in encryption (at least for people who are really, really paranoid) will involve 256GB flash sticks containing random one-time pad data the size of your HD (possibly encrypted), and replaced with new random data every time you write a block. Both data sets in isolation should appear to have very little order to them, but when combined, would suddenly be data.

Besides, unless you are talking about something seriously bad in your encrypted data, even if you are doing something illegal, it probably just has to remain hidden past the statute of limitations. If you're encrypting information about where you hid the body... well, then, you're screwed, but it should be pretty obvious to anybody on Slashdot that today's crypto will probably not remain unbreakable for the rest of your life....

Re:Vindicated!

[phoenix321]

Any program that does this is not not only destined for a Fields Medal, but the Skynet Award For Outstanding Achievements In The Field Of AI.

Re:Vindicated!

[addie macgruer]

Pi is a transcendental, infinitely non-recurring number: every finite sequence of numbers is present within it, including all "truly random sequences", of any length. Although, of course, some of them start at a very large offset indeed.

Alas, I'm too old for a fields medal.

Re:Vindicated!

[daveime]

So this is your high-tech solution ?

Please enter the result of coin flip #1 ?
Please enter the result of coin flip #2 ? ...
Please enter the result of coin flip #256 ?

A far better method is to use my UNDETECTABLE ENCRYPTION ALGORITHM (patent pending)

A simple transform thus :-

If any bit = 0, then leave it alone
If any bit = 1, then transform it into a 0

Ha, that'll fool the bastards, they won't ever decrypt that !!!

Re:Vindicated!

[Stray7Xi]

The other poster argued that truly encrypted data should be indistinguishable from white noise. I pointed out that a) stenography disrupted the image coloring and therefore should be detectable...Encrypted information will stand out as structured data.

I'll disprove you with a trivial example. If I use white noise as a one-time pad then the encrypted data is indistinguishable from white noise.

81 90 0F 37 93 F6 D3 8D 5A 6F
81 90 0F 37 93 F6 D2 8D 5A 6F

One of those is a truly randomly generated string. One of those is the encrypted string for my cleartext:
00 00 00 00 00 00 01 00 00 00
What you're saying is that one is random, while one is not... Which is which? I even made my plain-text full of patterns for you, this would make it easier for you... if it were a problem that was possible to solve.

As far as stego, you actually have it backwards the encrypted data is too unstructured, it's too random. By adding encrypted data to structured data, you're making the image LESS structured.

By using a structured pad you make the OTP solvable... (which is why you don't do it)
01 02 03 04 05 06 07 08 09 0A
01 02 03 04 05 06 06 08 09 0A

Re:Vindicated!

[Old Wolf]

Information theory doesn't mess around. You cannot destroy information.

Yes you can. I had a file on my hard drive, and I deleted it. It's destroyed now.

(The black hole experiments confirmed that.)

There have not been any black hole experiments.

Re:Vindicated!

[againjj]

Pi is a transcendental, infinitely non-recurring number: every finite sequence of numbers is present within it, including all "truly random sequences", of any length.

That does not follow. The Liouville constant is an example of "a transcendental, infinitely non-recurring number" where it is not true that "every finite sequence of numbers is present within it". This is trivially shown, as the Liouville constant consists of only zeros and ones.

Re:Vindicated!

[treeves]

Don't *all* numbers consist of just ones and zeros ? C'mon this is Slashdot!

Re:Vindicated!

[russotto]

No, it just means there is a flaw in TruCrypt's implementation.

Or they've found a distinguisher for XTS-mode encrypted data.

Re:Vindicated!

[AKAImBatman]

Thanks for not reading what I wrote. :-/

encrypted information (short of a one time pad, which is the only way to get true noise)

You said:

By adding encrypted data to structured data, you're making the image LESS structured.

You are correct. I did describe that a bit backwards. The encrypted data would look like anomalies in an otherwise structured file. Detecting them is difficult, but should not be impossible.

Re:Vindicated!

[againjj]

That's not what addie macgruer or I meant, and you know it. Besides, if you think in binary, the same number still is a counter example.

Don't worry

[sakdoctor]

The company has "innovations" in it's name, so their product probably won't work.
If it did work against true crypt, which is a yard stick of well implemented encryption, I'm sure they'll come up with a counter measure by the next minor release.

Also: In before XKCD strip.

Re:Don't worry

[SerpentMage]

What I am guessing is that they are doing Gaussian analysis. It is actually quite simple, and not too hard to implement. If a data set is truly random then the statistics will have some basic indications that it is random.

Since encryption implements a lossless conversion then the data is not random. BECAUSE random data is just that random.

Though it would not be that hard to get around this because the statistics can be fooled. Actually would not be that hard to do that. Thinking about it, rather interesting problem...

BTW I do statistical and probabilistic analysis in a hedge fund...

Re:Don't worry

[inviolet]

The company has "innovations" in it's name, so their product probably won't work.
If it did work against true crypt, which is a yard stick of well implemented encryption, I'm sure they'll come up with a counter measure by the next minor release.

This will probably become an arms race, in order to use vs detect subtler and subtler patterns in the bytes.

In any case, this tool will probably end up being used by law-enforcement as a polygraph, or breathalyzer: not true, not quite false either, but exciting enough to get the suspect to confess.

Reminds me of a funny story about polygraphs. The cops were questioning a particularly stupid criminal, and they knew he did it (disclaimer, disclaimer). So they taped some stripped wire ends to his fingers, and ran the other ends of the wires into some random slot on a nearby xerox machine. They had secretly placed a paper onto the copier's glass with the words "HE'S LYING" written on it. When the guy answered a question and they knew he was lying, they'd fully press the copy button, rather than just pretending to press it. Out would come a copy of the paper -- HE'S LYING -- and the guy, whelmed, confessed. Ha ha, owned. :)

Re:Don't worry

[Kjella]

Since encryption implements a lossless conversion then the data is not random. BECAUSE random data is just that random.

Encryption in ECB mode leaves a very clear pattern, because identical input blocks leads to identical output blocks. Pretty much every other block chaining mode doesn't though because they mix it the preceding blocks, so i'm guessing an implementation flaw because the cryptographic primitives are pseudorandom, they have no distinguishable non-randomness unless you know the exact key.

Re:Don't worry

[SerpentMage]

What I think they are doing and I think it would indicate an encrypted drive is distribution analysis.

If you have truly random data then there is a specific pattern. If you have deleted or unused blocks there will be a specific pattern.

But if you have an encrypted block the distribution will not be like any of the other pieces of data. This is your indicator.

Think of it as follows. You are driving on the highway and somebody on the highway drives the speed limit exactly, stays in the center lane, and does not switch lanes at all. Even though that would seem to be right, it is actually quite wrong and it would make police suspicious.

Re:Don't worry

[FutureDomain]

The company has "innovations" in it's name, so their product probably won't work.

I actually tried it with a Truecrypt volume and a random file (/dev/urandom) and it seems to work. The Truecrypt is identified as "Encrypted Data (Headerless)" and the random file is identified as "Data File (Unknown)".

Re:Don't worry

[compro01]

What does it say about a Truecrypt hidden volume?

Re:Don't worry

[maxume]

I would guess that they are just looking for files that are overly random.

Plain text won't be particularly random, and unwritten space won't be particularly random either. Do a quick check if the file can be decompressed by 7-zip and you filter out a bunch of false positives.

Re:Don't worry

[EnglishTim]

I seem to remember that being a scene from The Wire.

Re:Don't worry

[MaskedSlacker]

You've never heard of cruise control on a 500 mile trip have you?

Re:Don't worry

[Burkin]

I would guess that they are just looking for files that are overly random.

How can something be overly random? It's either random or it's not.

Re:Don't worry

[maxume]

Yeah, that was a terrible way to phrase it. Files that have exactly no structure aren't something that are normally present, which is what I was trying to say. Most files will have some amount of structure, there is a good chance that the ones below some threshold are either compressed or trying to hide something.

Re:Don't worry

[e4m]

It's chi-square and modulo 512. It's nothing more complicated than that. TCHunt has been doing this for more than a year now.

Re:Don't worry

[Anonymous Coward]

You realize that you aren't saying anything at all, right? Your argument is that since encrypted data is different than random data (an assumption you make without stating), encrypted data will look different than random data.

In reality, one of the standards for encryption algorithms (and block chaining methods) is that they produce a pseudorandom output. In fact, block ciphers are often called upon to operate as PRNGs when given random input data. The idea is that they will produce a significantly larger amount of pseudorandom output data than the random seed data.

BTW I do mathematical cryptanalysis at a university...

Re:Don't worry

[1729]

I seem to remember that being a scene from The Wire.

It first appeared in the David Simon's Homicide: A Year on the Killing Streets. The anecdote had been passed down within the Baltimore Police homicide squad, and was presented as a true story in the book. Simon later adapted this and other events from his true crime books for use in The Wire.

Re:Don't worry

[Stray7Xi]

BECAUSE random data is just that random.

Any kind of analysis that answers the question of whether a piece of data is random or deterministic can't do so with certainty. You can't prove a string of a million 1's wasn't randomly generated. Every piece of random data long enough will have substrings that appear to be a pattern.

Give a voice recognition program a low enough certainty threshold and it'll pick out words from below the noise floor. But the lower you go, it'll make more and more mistakes and eventually it'll pick out words from plain white noise.

Re:Don't worry

[wdef]

Love your sig SerpentMage - where did it come from?

Re:Don't worry

[Atzanteol]

Sounds like the do something like the free ent utility. It calculates a "randomness" of files. It can be quite useful to tell "data" from "encryption."

Re:Don't worry

[Bigjeff5]

BTW I do statistical and probabilistic analysis in a hedge fund...

Oh, so according to Nassim Taleb, you're one of the guys we should be blaming for the housing crash? You know, for improperly applying financial algorithms in a situation that has inherent (and significant) statistical uncertainty?

Ok, good to know.

Re:Don't worry

[mgiuca]

So basically this doesn't tell the difference between an encrypted drive and a blank drive, it tells the difference between a pure random drive and a blank drive.

That is, out of the following three possibilities:

1. Default/blank drive, possibly non-random.
2. Drive written over with pure random bytes.
3. Full disk encryption.

This tool can tell the difference between 1 and {2,3}, but it can't tell the difference between 2 and 3. That should still give you plausible deniability then, because there's the possibility you've written over your drive with /dev/random, and there's no way (in theory) for any tool to distinguish between that, and TrueCrypt.

Re:Don't worry

[gringofrijolero]

The company has "innovations" in it's name, so their product probably won't work.

Yep, shoulda gone with Acme..

Re:Don't worry

[Amazing Quantum Man]

Did it on Law & Order, too.

Re:Don't worry

[PitaBred]

That's why I name my TrueCrypt volumes stuff like "moo.zip"

"Awww, jeez... the damn thing's gotten corrupted! My boss told me to keep my sensitive company files in an encrypted zip file, and it keeps screwing up"

Just because security through obscurity isn't good as the only defense doesn't mean that it's not quite handy in addition to others.

Re:Don't worry

[digitig]

Every file is random. It's as likely as any other sequence of bits.

Re:Don't worry

[shutdown -p now]

Your argument is that since encrypted data is different than random data (an assumption you make without stating), encrypted data will look different than random data.

He didn't say that. He said that, for TrueCrypt case, the "random" data on the disk in free sectors is not random at all - it's got bits of deleted files in it, and so on. So, it's rather low entropy. On the other hand, sectors used for TrueCrypt will actually contain truly random, high-entropy data. And statistical analysis will be able to tell the difference easily.

Re:Don't worry

[shutdown -p now]

This tool can tell the difference between 1 and {2,3}, but it can't tell the difference between 2 and 3. That should still give you plausible deniability then, because there's the possibility you've written over your drive with /dev/random, and there's no way (in theory) for any tool to distinguish between that, and TrueCrypt.

You still live in the logic land. It doesn't work like that in real life - what they'll do is ban tools that write over your data with /dev/random, because they can be used to "obstruct [child porn] investigation".

After all, everyone knows that, if you don't break the law, you have nothing to hide - and you don't happen to have any nekkid kid pics on your hard drive, do you, citizen? Would be a shame if any were suddenly found, too...

Re:Don't worry

[jhantin]

Think of it as follows. You are driving on the highway and somebody on the highway drives the speed limit exactly, stays in the center lane, and does not switch lanes at all. Even though that would seem to be right, it is actually quite wrong and it would make police suspicious.

Wonderful. Now using cruise control is suspicious?

Re:Don't worry

[AmberBlackCat]

That would not account for people like me, who actually drive like that regardless of criminal intent. What if a truly random file just happened to have that pattern? Does the person go to jail due to the unlikely nature of the file?

Re:Don't worry

[Jane Q. Public]

If that is the case, then the solution is trivial: initially fill up the encrypted file space with pseudorandom data.

Re:Don't worry

[Anonymous Coward]

"...and eventually it'll pick out words from plain white noise."

Oddly, there's a wonderful lady(Deutsch is her last name, I forget the first, Diane maybe) who proved that the human brain will do the same thing with utter gibberish. From the same repeated nonsensical sounds, I heard "copy", "on me" "meow" and a ton of others. So I agree with you 100% in that it couldn't be 100% certain there was anything encrypted at all.

Re:Don't worry

[Just Some Guy]

Since encryption implements a lossless conversion then the data is not random. BECAUSE random data is just that random.

Really? What algorithm will detect patterns in a OTP-encrypted block? Your assertion has some pretty major loopholes.

Re:Don't worry

[Jane Q. Public]

No, but you can be pretty damned certain that 500 copies of "10001101" in a row are not random: that is higher entropy than your million 1s.

Re:Don't worry

[MichaelSmith]

People don't typically have files full of true random numbers floating around anyway. I could generate a nice near random screen background. When the cops say that must be encryption because its not random I just say its an image and it wasn't meant to be random in the first place.

Re:Don't worry

[Nefarious Wheel]

You must encrypt

If you'll be free

The tool is Steganography

Guys, stenography is taking shorthand. Steganography is embedding coded messages in images. They're two different words, honest.

Re:Don't worry

[greg1104]

Urban legend, eventually featured in both "Homicide" and later "The Wire".

Re:Don't worry

[JamesP]

And that's why it's recommended to compress things before encryption.

Re:Don't worry

[postbigbang]

Not necessarily.

Elliptical encryption can produce waves, but if the seed is large enough, it's a bear to detect. Bigger waves, bigger cache to AND for rhythms.... hint hint.

What's needed is some sort of slam dunk header with Britney Spears in some sort of Japanese HD interlaced display. Hash it with bluefish, then salt it up with Atomic Rooster.

This also bodes badly for Layer 7 router problems-- the kind where ISPs 'deep dive' into packet streams to throttle them back, so that all important ISP-provided movies can go through unfettered.

Re:Don't worry

[MSG]

I don't think so... It's recommended that you compress things before you encrypt them if you plan to do both (usually for network transmission). If you encrypt and then compress, your compression will not be very effective. Good encryption produces very few patterns, and patterns are what compression applications need in order to function.

Re:Don't worry

[nospam007]

"How can something be overly random? It's either random or it's not."

Think of it as "a little bit pregnant".

Re:Don't worry

[FreeFull]

I regularly store dumps of /dev/urandom. They certainly are normally present.

Re:Don't worry

[maxume]

I hope your are joking. Not because I care whether you store those dumps or not, but because if you aren't, you think that using some oddball definition of normally is an interesting comment (for instance, /dev/urandom isn't normally present on a personal computer; this is less true now that OS X has a decent amount of market share, but it still isn't a ridiculous statement).

Re:Don't worry

[Garse Janacek]

You can't prove a string of a million 1's wasn't randomly generated.

Remind me not to hire you as a QA tester for my random number generator.....

Re:Don't worry

[nabsltd]

No, the entropy is identical...that's the point of truly random data, as opposed to pseudo-random.

For truly random data of a given length, then every bit pattern is equally likely to occur. Humans tend to view things with repeating patterns as not random, but that doesn't make it true.

Part of the problem with convincing people about true randomness is the way pseudo-random generators work, in that "good" ones will not have long blocks of repeated patterns, but that's only for the full bit-length of the generated number. So, if you use a 64-bit PRNG and generate 64-bit numbers, the sequence has no (or very few) "patterns" as far as a human is concerned. But, if you only want numbers from 0-7 (i.e., 3-bit numbers), then you will end up with a lot of "patterns".

Re:Don't worry

[gfim]

What if a truly random file just happened to have that pattern?

There are worse things that could happen. It might contain the latest pop track - then you'd have the RIAA after you. That's much worse. Or perhaps it could contain the sequence "09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0". Then you'd really be in trouble.

Re:Don't worry

[Jane Q. Public]

If you have 2 files, each of 1M bytes, and one is filled with all 1s, but the other is filled with 500 copies of "10001101" followed by all 1s, the entropy is NOT identical. Please show your calculations.

I assert, once again, that 500 patterns followed by 1s is not of the same entropy. If it were, Fourier analysis would not work.

Re:Don't worry

[Jane Q. Public]

If you were discussing random bit-patterns, rather than random bytes, you should have said so. As it is, you were basically talking about something different from what everyone else was.

Re:Don't worry

[MoxFulder]

I wish I had mod-points for you.

Finally we hear from someone who knows WTF he/she is talking about.

Just to expand a bit: encryption algorithms (except for one-time-pad) don't produce truly random output. But all good, modern ones seek to produce output that's as indistinguishable as possible from truly random output, as a necessary but not sufficient component of their security. There are a variety of techniques to produce pseudorandom data based on a variety of sophisticated mathematics.

It seems like the height of hubris to claim that one software program can reliably detect all these different kinds of extreme slight deviation from perfect randomness.

A more plausible approach (as others have pointed out), is to look for files that do appear to be totally random. Such files are likely to be either (a) the output of a random number generator, or (b) encrypted. All files that have some useful content in their present form have some structure or non-randomness.

Re:Don't worry

[MoxFulder]

He didn't say that. He said that, for TrueCrypt case, the "random" data on the disk in free sectors is not random at all - it's got bits of deleted files in it, and so on. So, it's rather low entropy. On the other hand, sectors used for TrueCrypt will actually contain truly random, high-entropy data. And statistical analysis will be able to tell the difference easily.

True. But TFA claims that the software works by looking for patterns in the encrypted data. Whereas you are arguing (in my mind correctly), that the way to find the encrypted data is to look for data without any patterns.

Re:Don't worry

[mrchaotica]

His point was that that's what his excuse would be for the presence of "too-random" data on his computer. I'm not convinced it would be all that successful though, because /dev/urandom isn't a real file: when you "read" from it the kernel just generates [pseudo-]random bits on-the-fly.

Sometimes I want to build myself a lava lamp random number generator just to have a plausible excuse for storing big "random" files.

Re:Don't worry

[mrchaotica]

TCHunt has been doing this for more than a year now.

Have the TrueCrypt people added the ability to create non-multiple-of-512-byte volumes yet?

Re:Don't worry

[jallen02]

Yeah, you can fool it but you would definitely end up inflating the data. Properly done crypto should leave the data completely indistinguishable from random noise. If I can determine your data is encrypted and somehow different from random noise then you have implemented your cryptography improperly. However, if you scan a disk and, using statistical frequency analysis, find bits of random noise floating around that is not a common event. Compressed files are pretty easy to test for, even though they do start approaching noise in their randomness compared to a regular file format like HTML. Basically in order to attain proper cryptographic properties you can't cheat and use some method of crypto that is not random. What you could do is properly spread out the random data enough and intersperse it with regular patterns to make it resemble something else. You would basically end up with a data hiding routine. Small messages are the easiest to hide. Ala steganography. A lot of work has been done in that arena.

I realize you may have known some or all of this (crypto and the breaking of it is just fancy statistics for the most part!) Just elaborating on the idea a bit for others. I work in infosec so I love crypto :)

Re:Don't worry

[plnix0]

They'll still detect the (pseudo-)randomness, and they will assume that because it appears random, it must be encrypted. After all, most people don't have random bytes written all over their hard drives, do they?

Re:Don't worry

[nabsltd]

It all comes down to random bit patterns.

A sequence of 4 bytes can be considered an 32-bit unsigned number (assuming 8-bit bytes). Any truly random way to create those 4 bytes will result in an equal chance for all 4,294,967,296 bit patterns. Any way that fills those 4 bytes that doesn't have an equal chance of each of those bit 4,294,967,296 patterns is not random...it is pseudo-random.

Likewise, a sequence of 100 bytes can be considered an 800-bit unsigned number, and any truly random way of creating those bytes will have an equal chance for all 2^800 bit patterns.

Although some of those bit patterns may seem more organized to a human observer, and might have less "entropy" by some definition, they do not require any more work to create than those with "more entropy".

Last, it all depends on how you look at those bit patterns whether there truly is more or less entropy. A sequence interpreted as individual bytes may have far more "organization" than the same sequence when treated as 32-bit little-endian values, for some definition of "organization".

Re:Don't worry

[Rockoon]

Such files are likely to be either (a) the output of a random number generator, or (b) encrypted.

..or (c) compressed with a decent headerless compression algorithm

Re:Don't worry

[andy_t_roo]

because, as one of the up-thread comments says, a large file which looks true random is either encrypted or the output of a (good) random number generator. This software wouldn't be able to tell the difference. Unfortunately very few people need very large amounts of true random data, as the people who need the most random numbers are probably computational scientists, and then a good PRNG will do that for you.
Alternative needs of true random data relate to communication, or cryptography. Either way you are either 1) an academic (easy to rule out/use as an excuse) 2)have a need for better than internet banking security on your communication (once again easy to prove or rule out, as almost noone needs this) or 3) have a large encrypted file.

One way around this would be to format the blank space on your hard drive true-random, rather than a specific pattern. that way all space which hasn't been used recently looks like a blob of "encrypted/random" text. If you then go and shred (overwrite with random data) all files as you delete them, then having a block of random text on your hard drive is only then evidence of paranoia, not criminal conspiracy.

Re:Don't worry

[Jane Q. Public]

Please save your lectures. I know what a pseudo-random generator is and how they work, and also how numbers are represented in processors. I have been doing this for some time, you know.

You have contradicted yourself. First you wrote that you do not use pseudo-random generators because they have byte-patterns, while you are looking for bitwise randomness. Then, you turn around and say that there is no difference between the two. You can't have it both ways.

I am aware that bits can be grouped arbitrarily, which in fact works against what you were saying. While a typical pseudo-random generator might return a number in a particular processor family's format (resulting in patterns), it is trivial to map that formatted number to a flat binary value that is guaranteed to be unique for each number represented, which in turn gives you your bitwise randomness, and knocks your argument right out of the water.

I retract my statement that a pattern has higher entropy than all "1"s. While that is possible, is not necessarily so. For example, the 1M file of 1s could be conveying the information that 1 million pixels should all be turned "on", while the pattern cannot really convey more (bits of) information, so the entropy is the same. Rather, I was referring to the ability to detect information (Fourier analysis), which is not the same thing. That was my error.

Nevertheless, detection does not "all depend" on how you look at bit patterns. Patterns are patterns. They are mathematical entities, whether you look at them as bytes or individual bits or groups of 2 or 3 or 5 or 8 or 32 or even 37 (though I admit that certain choices can complicate the math for finding some patterns).

Re:Don't worry

[Nogami_Saeko]

Just FYI, an encrypted ZIP file looks nothing like a truecrypt volume internally... And an investigator who sees this and is smart enough may be curious enough to look into it more...

Re:Don't worry

[Maguscrowley]

Exactly. Boasting that you're a quant or a financial mathematician these day's is NOT exactly a smart move. *whispers* someone prepare the oil ...

Re:Don't worry

[TechyImmigrant]

My job currently requires me to hold and use very large amounts of random data. If it can happen to me, it can happen to anyone.

Many people who don't strictly need large amounts of random data, would probably benefit from it if in fact they did have it. Particularly when the anti encryption gestapo come knocking.

Re:Don't worry

[TechyImmigrant]

People don't typically have files full of true random numbers floating around anyway.

I do.

Re:Don't worry

[TechyImmigrant]

This will probably become an arms race, in order to use vs detect subtler and subtler patterns in the bytes.

I don't think so. A good cryptographic PRNG or encryption algorithm+mode yields data that is indistinguishable from random. In that respect, it is already game over.

Re:Don't worry

[Chrisq]

Though it would not be that hard to get around this because the statistics can be fooled. Actually would not be that hard to do that. Thinking about it, rather interesting problem...

BTW I do statistical and probabilistic analysis in a hedge fund...

XOR it with known random files.

Re:Don't worry

[RockDoctor]

Just to expand a bit: encryption algorithms (except for one-time-pad) don't produce truly random output. But all good, modern ones seek to produce output that's as indistinguishable as possible from truly random output, as a necessary but not sufficient component of their security. There are a variety of techniques to produce pseudorandom data based on a variety of sophisticated mathematics.

Now, that makes me ask a clearly-begged question : if a particular encryption package produces an output that is distinguishable from truly random data (which I'll take as given ; I'm not a mathematician, but I have read up on the subject sufficiently to understand this point), then likely a different encryption package would produce a different output which would also (in theory) be distinguishable from random data. BUT, would it be possible to distinguish the two different encryption packages?

AIUI, most of these systems have a number of steps : combining multiple input files into one "archive" file ; compression of the archive into a high-entropy compressed file ; encryption of the compressed file to produce the output file. Each one of these steps can be done in different ways : on the archiving step, do you chain files in date order, alphabetical order or in folder order? ; compression could be done with the free zip libraries, or using the gZip libraries, or bZip, or something you cook up yourself? ; in the encryption stage you have choices like Blowfish, AES ... yadda yadda. Given that, one would expect different choices to have consequences for what exact differences there are from randomness for each set of choices.

So, the potential may exist to distinguish between these different sets of choices if you can find sufficient differences between the outputs from the different stages. Which is one thing. But equally one could incorporate this into the design of your encryption package so that the compression method used is switched regularly. Or irregularly. Or possibly, in keeping with the contents of the files and doing the compression step before the archiving step (which becomes null if you're writing a single file).

Given three reasonable choices for each of those stages, you've got the thick part of 30 different reasonable choices that would lead to different "fingerprints" of non-randomness in the output. Which the "Forensics Innovations" being touted here would have to be able to distinguish.

On the assumption that the encryption libraries do a good job, those fingerprints are going to be relatively slight. So the amount of data needed to confidently distinguish between the dozens of reasonable possibilities (and between them and "random") starts to get pretty high. My guess would be that this software only distinguishes between the default settings for WinZip's encryption, the defaults of TrueCrypt (does TrueCrypt have defaults? It's that long since I set it up on my memory sticks that I can't remember if it has defaults, or if it asks you questions in random order.) and "true random" data.

Reading the discussion on ForensicInnovations' web site they focus on rather different aspects. But the implication is that they (FI) only report a small number of different file types, and that any files composed of plain random data they report as "Encrypted Data (Headerless)) (type #3174)". Which doesn't surprise me.

So, time to start producing lots of nice, big, empty, formatted TC containers to fill up the empty bits of the hard drive. Then delete them. Just to confuse people, of course.

Doesn't someone produce a tool for doing this automatically? My memory tells me there is.

Re:Don't worry

[digitig]

It was sort of an allusion to the famous incident reported by Press et al in the "Numerical Recipes in [language]" series: "One of us recalls producing a 'random' plot with only 11 planes, and being told by his computer center's programming consultant that he had misused the random number generator. 'We guarantee that each number is random individually, but we don't guarantee that more than one of them is random.' Figure that out."

Re:Don't worry

[es0vyr4fVY9LD8ub]

Actually it's much easier than that: all they need is to "sell" to the judge/jury the notion that they
"can" find the patterns. Because you can't supply the keys if you don't have them, you will always be guilty.

It's called "reversing the burden of proof" with a bit of "witch-hunt" in the mix.

Re:Don't worry

[Bender0x7D1]

It's either random or it's not.

Randomness isn't an all-or-nothing concept. Here's a really bad analogy to demonstrate that.

Imagine you have three 20-sided dice (d20). The first d20 is normal with each face numbered from 1-20. The second d20 is almost normal and is numbered from 1-19 with a duplicate #1. The third has a #1 on all of it's faces. Now, how random are these dice?

Well, the first dice would be totally random, and the third would be totally deterministic (non-random). However, the second die would be "mostly random". It isn't totally random since it has a duplicate #1, giving that a higher probability of showing up. But it definitely isn't deterministic since we don't know what the next roll is going to be. So, different levels of randomness are possible.

My apologies to any mathematicians who hated that analogy.

Re:Don't worry

[euri.ca]

That's an interesting point. The test files for our software include 100% random files of 1k, 2k...2^n k bytes, which I imagine is exactly what random noise could look like on the hard drive.

Re:Don't worry

[Fnord666]

You realize that you aren't saying anything at all, right? Your argument is that since encrypted data is different than random data (an assumption you make without stating), encrypted data will look different than random data.

I never thought I would see a real life case of "begging the question", but here it is. I want to thank the GP for making the statement and some AC for pointing it out.

Re:Don't worry

[PitaBred]

Oh, I know. But again, more and more layers of misdirection that aren't hard for me to deal with, but would be increasingly hard to investigate. Besides... would a corrupted encrypted zip look terribly different from a truecrypt volume? Would an investigator know that?

Re:Don't worry

[hoggoth]

I am a computer forensic investigator, and I know what the structure of a zip file looks like internally. It's NOT a blob of random bits. Even a corrupted zip file has a well defined header, indexes, etc.

It's extremely difficult, if not impossible, to hide data from a good investigator who has the time and motivation to investigate thoroughly. If I find a large file containing only random bytes, it is NOT a normal thing and I will look into it further, especially if the file size is an even multiple of 512 bytes. If I can find traces of TrueCrypt ever having been used on that drive I will have a pretty good idea what I'm looking at. I can try to decrypt the file using every possible string found on the hard drive, including bits of memory saved to the paging file and hibernation file. If I manage to decrypt and open the file and find it is formatted with the FAT32 filesystem instead of NTFS I will be very suspicious that this was done because there is a hidden "plausibly deniable" inner volume. I will then work on cracking that open like I did the outer volume. I will also report to the authorities I am working for that there is a significant possibility of a hidden volume. They will use their social skills to get the key from the owner.

The real limitation is that cases usually DON'T give me enough time or resources to investigate that deeply, or the lawyers manage to bury the issue of an encrypted file and it doesn't get addressed. The best bet for a person with something to hide is to make it very difficult and time consuming for an investigator to get to the bad stuff, and hope his case isn't that important to warrant the time to dig deeply. In practice that means if you cheated your partner in a small business and hide it very very well I probably won't find it. If you killed someone I will find it.

Re:Don't worry

[BitterOak]

If I manage to decrypt and open the file and find it is formatted with the FAT32 filesystem instead of NTFS I will be very suspicious that this was done because there is a hidden "plausibly deniable" inner volume.

Actually, all of my Truecrypt volumes are formatted as FAT32, and I don't use hidden volumes at all. The reason I use FAT32 is that it makes it easier to mount and read/write the volumes in Linux as well as Windows.

Re:Don't worry

[MoxFulder]

..or (c) compressed with a decent headerless compression algorithm

"Headerless compression"??? Why would anyone want such a thing...?

Normally you want your compressed files to have a very well-defined header structure, in order to make it easy to identify them and recover data in case of corruption. The only reason you'd want to *not* have a header is for data hiding, and at that point we're back to encryption (where we started), and compression or lack thereof is irrelevant.

Furthermore, even without a header, most compression algorithms work by splitting compressed files up into repetitive chunks of some kind and building tables that index these. That pattern itself is detectable.

Re:Don't worry

[Rockoon]

Furthermore, even without a header, most compression algorithms work by splitting compressed files up into repetitive chunks of some kind and building tables that index these. That pattern itself is detectable.

umm... no, unless you mean non-entropy encoders.

Perhaps you should take up your "theory" with Claude Shannon or any modern day information theorist, since you seem to think that you can find coherence in the output of an algorithm designed to reach the limit of entropy.

If you can find a pattern in the output of a decent entropy compressor using a particular model, then someone else can take your model and make a better compressor. Why havent they? Do you think that you are smarter than the hundreds of people who research this stuff?

Re:Don't worry

[MoxFulder]

Furthermore, even without a header, most compression algorithms work by splitting compressed files up into repetitive chunks of some kind and building tables that index these. That pattern itself is detectable.

umm... no, unless you mean non-entropy encoders.

Fair enough. I was thinking more along the lines of things like headerless DEFLATE output, which has some noticeable structure (which is, as you point out, redundant and therefore slightly inefficient).

I agree that a straight entropy-encoder like Huffman will not show such a redundancy.

Windows Only

[rts008]

Won't someone please think of Linus and RMS? *ducks*

No info in the article. It is an advert for some Windows only software.

Repeat, this is an advert, not an article.

Re:Windows Only

[BitterOak]

The advertisement says the software runs on Windows. It doesn't say that it is incapable of scanning non-Windows disks or filesystems.

Re:Windows Only

[RiotingPacifist]

AFAIK windows can only read fat,ntfs,cds and at a push ext

Re:Windows Only

[chmod a x mojo]

There is also a plugin for Reiser3 that is really sloppy. The easiest way to READ Reiser3 is user the plugins built in mini webserver in a web browser ( a serious PITA ).

They also had a really buggy Java reader as well ( was buggy as hell last I tried to use it anyways) maybe they have had better luck lately though.

Re:Windows Only

[rts008]

Yes, that's what I said.

Did you miss the 'No info in the article.' bit?

Benford's law

[tyrr]

This is probably another application of the Benford's law.

Re:Benford's law

[owlstead]

Well, not if it is encrypted data. I'm not so certain that you cannot detect AES (CBC) encrypted files, but the leading digit will certainly look like it is randomly distributed - the numbers are certainly not part of a logarithmic scale (which is what seems to drive this "law").

And mods, questions and sentences that have "probable" in them should - in most cases - be modded "interesting", IMHO.

Re:Benford's law

[noidentity]

This is probably another application of the Benford's law.

Doesn't that law hinge on the fact that you're looking at the first (non-zero) digit of each of a series of numbers? Without that, you don't get the logarithmic aspect, and thus no distinct pattern.

Who Cares?

[DomNF15]

The Wikipedia page on TrueCrypt already indicates that the volumes can pretty much be detected since they are always divisible by 512, it's just impossible to PROVE they are TrueCrypt volumes...

Be enlightened: http://en.wikipedia.org/wiki/TrueCrypt

Re:Who Cares?

[dgatwood]

That's just sloppiness, then, if being undetectable is a goal. There's nothing preventing them from adding N pad bytes to the end of the file where N is some random number from 0-511....

Re:Who Cares?

[HappyEngineer]

If that's true, why don't they just append a random number of 0 to 511 bytes to the end of all files?

Re:Who Cares?

[Spit]

Just use raw devices.

Re:Who Cares?

[TechyImmigrant]

There exist cryptosystems that do explicitly obfuscate the length of the underlaying data. It is not a new thing.

Yet another scam

[trifish]

Wow, the quality of Slashdot has really been going down lately. Now any random fraud can submit his misleading material and it gets accepted to front page just because it sounds interesting? Is this actually tabloid or serious news for nerds who understand what the talk about?

In short, this is yet another lame attempt to make money by posting bogus claims about a popular product.

First, hidden volumes are the only kind of steganography that TrueCrypt offers. Second, if you read the TrueCrypt documentation, you'll learn the following about hidden volumes vs. dynamic:

On Linux or Mac OS X, if you intend to create a hidden volume within a file-hosted TrueCrypt volume, make sure that the volume is not sparse-file-hosted (the Windows version of TrueCrypt verifies this and disallows creation of hidden volumes within sparse files).

Furthermore, when I try to create a dynamic TrueCrypt volume, TrueCrypt displays a big warning saying that dynamic volumes are insecure. That's right. Insecure.

So again, I demote this story as total and utter bogus motivated by the vision commercial gain.

Re:Yet another scam

[trifish]

Sorry, I forgot to add a reference for the quote in bold. Here it is:

http://www.truecrypt.org/docs/?s=hidden-volume-precautions

Re:Yet another scam

[trifish]

And yet another omission on my side:

Dynamic volumes = Sparse-file-hosted volumes

Re:Yet another scam

[gurps_npc]

I am the poster. I have ZERO connection to the company mentioned I read about because I do computer programming for a law firm.

The article may in fact just be an advertisement, created for commercial gain.

But it was posted because I personally read it and was interested in it.

Re:Yet another scam

[ion.simon.c]

:/

Please... do a little bit of research into the claims behind a story before you post it. Keep slashdot beautiful. Don't give timothy or kdawson more shit-tastic stories to post on the front page. :(

TCHunt for Truecrypt containers

[Anonymous Coward]

TCHunt (free tool, you can find it with google) has worked for some time doing this exact thing on Truecrypt encrypted containers.

Actually tried it out last night. It does get false positives, but on my system it did indeed manage to find what it was looking for (total: 4 false positives, 1 true positive).

Re:TCHunt for Truecrypt containers

[Beve Jates]

Maybe for old versions of TrueCrypt but it doesn't seem to work on anything else. For example, loop-aes, dm-crypt, and the latest TrueCrypt. I tested it on all kinds of encrypted sample data and it didn't detect any of it. I imagine this proggy ain't much better.

The easiest way to defeat TCHunt is to use a volume size that isn't a multiple of 512 (easy to do when not using TrueCrypt). Not that it matters much because like I said, it can't accurately detect anything based on the data either.

This is just a bunch of advertisement bullshit. Nobody that takes security seriously uses TrueCrypt anyway because the author(s) are unknown and the whole code base has not been publicly analyzed. I won't even install it outside of a throwaway vm because of that.

If there are patterns visible in your encrypted data then that means your encryption is weak. If that's the case then you're screwed because they can probably break the encryption too. Good encryption is indistinguishable from random data, period.

OK, this is just ... um, scam

[trifish]

It works by detecting hidden patterns that don't exist in a random file.

That would be equal to breaking AES or the mode of operation (XTS).

If they could distinguish the AES-XTS ciphertext from random data, they would be famous in the cryptographic community instantly. However, these fraudsters obviously cannot do anything like that. They are just posting a bunch of lies hoping to earn big money on it.

This makes perfect sense...

[offrdbandit]

This makes perfect sense. There's always been a way to distinguish between pseudorandom and random value sets. There's also no reason to believe an encryption algorithm could somehow start producing genuinely random values, so it follows that encrypted value sets should be distinguishable form truly random value sets.

I'm calling BS

[zindorsky]

I'm pretty familiar with TrueCrypt, but I don't know what a TrueCrypt "Dynamic" file is. Are they just talking about an encrypted virtual volume?

Anyway, I'm pretty sure this is BS. I think they're just doing regular entropy tests on files. That will tell you when you have random data. A good test might be able to distinguish a large amount of compressed data from encrypted data since compressed data does have a little redundancy (emphasis on "might" and "little").

But I guarantee that they are not detecting any redundancy in ciphertext. Detecting even a small amount of redundancy in the output of any modern cipher algorithm (like AES or Twofish) would be a HUGE cryptanalytic result. It would be front page news (in cryptographic circles).

In summary, I'm positive that they can't distinguish between a TrueCrypt volume and true random data.

Put up or shut up.

Re:I'm calling BS

[e4m]

All TC volumes are modulo 512 (very rare) and pass chi-square test (even rarer). Check out TCHunt. It's amazing. http://16systems.com/TCHunt/index.php It will find *all* of your TrueCrypt volumes. They also disclose how they do it.

Re:I'm calling BS

[zindorsky]

OK, I checked it out. Here's how they "do" it:

1. No File Header.

2. (File size % 512) = 0

3. Successful X2 and Arithmetic Mean tests on certain bytes.

4. File size greater than 15 MB.

Step 2 == entropy tests.

In other words, they detect random looking files (which implicitly implies "no header") whose size is 0 mod 512 and is greater than 15MB.

Big fucking deal. It might be true that on your system, the only files that meet these characteristics are TrueCrypt volumes, but again it's trivial to create non-TrueCrypt files that meet these criteria. Simply, any true random file (whose size meets the above requirements) will be detected as a TrueCrypt file.

I stand by my assessment: BS.

Re:I'm calling BS

[TheCoop1984]

There is a problem with this - it forces a hole in the 'plausible deniability' defence. Really, how many people keep bits of /dev/random data lying around on the filesystem? Having this tool 'officially' point out that it is random data means the police can ask you, however impolitely, for the encryption keys (UK encryption key laws...). And you don't really have a defence for that, as what are the chances of it _actually_ being completely random data?

TCHunt Does this very well

[e4m]

TCHunt found all of my TrueCrypt volumes. It's free too. http://16systems.com/TCHunt/index.php

Re:How about...

[s0litaire]

Drat! above post is mine! It got marked as Anon by mistake!!

Sure they do! :)

[PaulBu]

[pb@localhost ~]$ tail ~/.bash_history
less GnosLoadPDFForms.pdf
file GnosLoadPDFForms.
mv GnosLoadPDFForms.pdf GnosLoadPDFForms.fdf
file GnosLoadPDFForms.fdf
evince GnosLoadPDFForms.fdf
less GnosLoadPDFForms.fdf
su
acroread GnosLoadPDFForms.fdf
top

Re:Sure they do! :)

[nathan.fulton]

history -c;shred ~/.bash_history;rm ~/.bash_history; This is in a cron job that ones once a minute on my machine. It's so that all those pesky things I did yesterday don't come up in my tab-completion... or something like that.

Re:Sure they do! :)

[pigeon768]

or add HISTSIZE=0 to /etc/bash/bashrc or ~/.bashrc

more secure, uses less resources.

Re:Sure they do! :)

[Randle_Revar]

# ignores commands preceded by a space
HISTCONTROL=ignorespace

of course then you have to remember to put a space in front of any commands you don't want recorded

Re:Sure they do! :)

[Talchas]

Or among the other things people mentioned,

kill -9 $$

, as bash doesn't write history until it exits.

Re:Sure they do! :)

[Spit]

unset HISTFILE
shred -uv .bash_history

Re:Sure they do! :)

[PReDiToR]

As root: 'ln -s /dev/null .bash_history' for all users. I do it in the first bash session I log into after installing. No need to shred as the file never had any content.

Stops anyone from turning the function on, or creating spurious histories.

Re:Sure they do! :)

[Bill, Shooter of Bul]

That's why you use SELinux with auditing turned on to track those that do not want to be tracked. Mu ha ha ha

One-time pad

[davidwr]

Disk space is cheap.

Have a large one-time pad on one drive. For good measure, encrypt it.

Have an encrypted drive, and inside it have a one-time-pad-encrypted container file.

To keep from being toast if someone grabs both disks, have the one-time-pad-decoder program grab the bits from the one-time pad in some non-obvious order.

For example, when mounting the one-time-pad-encrypted container file, the mounter could ask for:
1) starting offset
2) byte-interleave
3) additional XOR string

The starting offset says start reading the one-time-pad file at byte OFFSET instead of byte 0.
The byte-interleave is how many bytes to skip between each read.
The additional XOR strong is an additional string that is applied as if it was a one-time pad, but it's used repeatedly. It won't offer a whole lot of protection on its own but it is just one more roadblock to decryption.

If the one-time pad is
01010111 11110000 00001111
and the data is
11111111 11111111 11111111
and the one-time XOR string is
01010101
and the starting offset is 0 and the byte-interleave is 0,
then the encrypted data is the xor of
01010111 11110000 00001111
11111111 11111111 11111111
01010101 01010101 01010101 <-- note the repetition

which is
11111101 01011010 10100101

Headerless encrypted files

[SnarfQuest]

Ummm, brains...

Are they just checking for the existance of the evil bit?

Is unreadable data really encrypted data?

[stokessd]

I do a lot of data acquisition for work and in grad school. I've got lots of my data on my drive. They are written in binary formats of my design. There is lots of repetition, there are no headers, who knows what my data looks like to someone who doesn't know the "decoder ring" to unpack it.

That doesn't mean that my files are kiddie pron or directions to make a dirty bomb.

Sheldon

Re:Is unreadable data really encrypted data?

[e4m]

There is a reason that high-quality encryption was once classified as a "munition" by the US government. You cannot accidentally create it. You need a very good PRNG or an algo such as AES. Don't worry, your formats will (and cannot) be confused with encrypted data.

Re:Is unreadable data really encrypted data?

[Rockoon]

That doesn't mean that my files are kiddie pron or directions to make a dirty bomb.

of course not.. you are a terrorist..

You have dirty porn and kiddie bombs, not kiddie porn and dirty bombs.

Re:Is unreadable data really encrypted data?

[TechyImmigrant]

AES applied correctly makes an excellent PRNG. SP800-90 gives details.

A random file?

[noidentity]

It does not decrypt the file, just tells you that it is in fact an encrypted file. It works by detecting hidden patterns that don't exist in a random file.

I guess anyone who has a drive full of files made of random bits, with a few encrypted files mixed in, is going to be in trouble!

obligatory question..

[miknix]

..will it crash scanning /dev/urandom?

Because it starts generating pseudo-random bits when the buffer with "true" random bits is exhausted.

Disclaimer: No, I didn't read TFA.

Re:Stenography, NOT Steganography

[daveime]

The UK Police can simply put you in jail until you hand over your father.

Re:.bash_history

[psyclone]

Um, instead of running that as a constant cron job, which means history data is still being written to disk, try changing the mode of the file.

example:
  echo > ~/.bash_history
  chmod ugo-w ~/.bash_history

Indistinguishability

[mrstrano]

Actually crypto systems are evaluated by a concept called indistinguishability. The key idea, is that an attacker cannot distinguish the output of a cipher algorithm from a uniform and random distribution.

For example if you use AES in CBC mode, there is no way of distinguishing its output in polynomial time from one truly random. It's specified in polynomial time because, for example AES in CBC will cycle after 2^128 blocks generated.

My guess is that there must be an implementation or a design error in TrueCrypt that should be fixable.
Or maybe this company is full of BS..

wait for truecrypt to patch

[DragonTHC]

then we'll have a cold war of encryption.

This is complete BS, and is easy to test

[anom]

This is complete sensationalist crap. Truecrypt isn't broken, (probably) nor are any of the other programs they possibly claim to have broken.

This is easy to test for yourselves folks, I just did it in 5 minutes.

dd if=/dev/urandom of=/home/me/somefile.jpg bs=512 count=10000

Performing this command and then scanning the resulting file with "File Investigator" results in the file being detected as a headerless encrypted data file.

Whoever pointed out that they simply identify any randomly filled binary file of a size of a multiple of 512bytes is correct.

TrueCrypt doesn't use ECB mode, hasn't for some time, etc etc etc. Stop freaking out every time someone claims to have broken it.

Re:This is complete BS, and is easy to test

[onsager]

But /home/me/somefile.jpg IS a headerless encrypted data file. It is the number 0 stored as a 40960000 bit integer, encrypted against a pseudo-random one time pad.

Re:This is complete BS, and is easy to test

[anom]

hahaha you got me, well played ;)

Re:This is complete BS, and is easy to test

[Cruicky]

I just tried with a 2GB file from /dev/urandom and it was of course detected as headless encryption.

I then however tried with a file that was 2GB + 1 byte, and it was still seen as headless encryption. Based on the time taken for the program to think it's headless encryption probably means it's only scanning the first X bytes, so it's possible it's just an entropy test on the header area.

This is a bit like TCHunt, except TCHunt actually does do the mod 512 test.

Use ephemeral keys stored elsewhere.

[EWAdams]

Establish a system whereby all your data is re-encrypted daily using a random keyfile that is generated every day and stored on the Internet somewhere, not on your own machine (someplace where you know they don't make backups). The keyfile is deleted at midnight. If you have not re-encrypted your data with a new keyfile by the time the old keyfile is deleted, the data is irrecoverable.

When they seize your computer, invoke your right to silence and stall until midnight.

You risk losing all your data that way if you don't get it re-encrypted before the keyfile goes away, but for some it may be worth the risk.

Overkill.

[Jane Q. Public]

While quantum data may be "truly" random, as far as we know, there are perfectly good pseudo-random generators that are guaranteed not to demonstrate a pattern for over 2^70 iterations. That should be good enough for anybody who doesn't need to do anything heavier than randomly distribute every particle in a galaxy.

Proof?

[nurb432]

Does it *prove* the file/partition is encrypted or does it just strongly suggest that it is suspicious?

i think you would still have plausible deniability.

Hidden stenography files!?!?

[frank_adrian314159]

I guess you better send in my secretary, then. She knows how to take my dictation.

Oh, you meant steganography... Never mind.

This seems like BS

[MoxFulder]

Properly encrypted files don't contain significant "patterns that don't exist in random files."

If the cryptography is actually secure, then its result will be indistinguishable from random noise. See the wikipedia article on crypto random number generators, for starters. Any departure from this represents a security vulnerability. That's not to say that every modern cryptosystem is totally secure, but it seems highly unlikely that they all contain similar types of non-randomness.

I would guess that a better way to detect encrypted files is to look for total randomness. Or at least something that's nearly so. There's hardly any other reason to have a file of any size with random contents. Any file that's useful in its current form almost by definition contains some significant structure.

TrueCrypt uses XTS

[Ux64]

That's just why XTS is being used instead of ECB. Even if every block would contain same plaintext data, encrypted output is always different. And for speed and random access, any chaning modes aren't being used like CBC.

Re:TrueCrypt uses XTS

[TechyImmigrant]

XTS mode is deeply depressing.

Its use represents the inability of people who don't really understand cryptography to accept a single byte of overhead for security.

A significant class of cryptographic attacks is enabled if you do not use any redundancy in your cryptographic encoding. That is why so many good and fine protocols have things like nonces, PNs or IVs.

But it simply wouldn't do to have to call your 1.2 TB disk a 1.1 TB disk. So the market droids win over the cryptographers and the customers who would prefer their data to be properly protected, lose.

Fail.

[Wanon]

Fear my one time pad!

Thumbnail cache

[Ux64]

It seems that at least Ubuntu and Windows Vista are using tricky thumbnail cache, which is saved to system path. So it's nice to find thumbnails of all those so secret photos from it. ;) Oops, I didn't have those, I don't know what those are. Err...

Random or Encrypted? It Can't Tell

[Orzeszek]

This tool can't tell the difference between a TrueCrypt container and a random file generated using a basic subtractive random number generator algorithm. Results here.

A simple solution

[TheCoop1984]

There's a simple solution to this - cat your encrypted file with a certain length of /dev/random data, both front and back, and remember the offsets. Then, when you want to access it, dd the encrypted part into a ramdisk, then dd it back when you're done. Problem solved.

Off topic for a sec

[Joe U]

Just going off topic for a second:

If you're running Windows 32 and you have 4GB of RAM in the system, you don't really need a swap file. You can't go over 4GB of memory on a 32 bit system (with some exceptions that you most likely won't ever see at home), a swap file would just be for debugging. So, if all you're doing is playing games & browsing, why bother with a swap file?

False positives?

[jonnat]

Naturally, the success rate and, most importantly, the fraction of false positives are not provided. Without these numbers, it's not only useless, but misleading and dangerous.

Re:.bash_history

[psyclone]

I suppose bash can forcibly remove it, but it's never happened to me. I just tried it.