Slashdot Mirror
Forensics Tool Finds Headerless Encrypted Files
gurps_npc writes "Forensics Innovations claims to have for sale a product that detects headerless encrypted files, such as TrueCrypt Dynamic files.
It does not decrypt the file, just tells you that it is in fact an encrypted file. It works by detecting hidden patterns that don't exist in a random file. It does not mention steganography, but if their claim is true, it seems that it should be capable of detecting stenographic information as well."
I'm am a citizen of the United Kingdom. Amongst many odd laws we have here, there's one that basically means that you can go to jail if you refuse to hand the police your encryption keys if they ask for them. The one saviour was Truecrypt's plausible denial. If they don't know you have encryption they can't ask for keys!
Now they do know I have encryption... ...and I've forgotten my password.
Can someone please give me tips on how to avoid dropping soap in the shower?
I should first say that I'm rather ignorant about encryption but I hope someone will be able to explain this. I was under the impression that any sort of good-quality encrypted data is indistinguishable from completely random data. That seems to directly contradict the ability to determine whether a volume contains encrypted data by means of locating patterns. Is this really a contradiction?
It is a miracle that curiosity survives formal education. - Einstein
s-t-e-g-a-n-o-g-r-a-p-h-y...not stenography.
"Murphy was an optimist" - O'Toole's commentary on Murphy's Law
add a "ga" to every mention of stenography in the summary. Unless he means that encryption uses shorthand.
Or "How I learned that you can't fight information theory".
A few years ago, I recall arguing with someone here on Slashdot about this very issue. My take on it was that stenography could never be completely successful because there would always be a pattern sticking out from the file. The other poster argued that truly encrypted data should be indistinguishable from white noise. I pointed out that a) stenography disrupted the image coloring and therefore should be detectable by looking for irregularities and b) encrypted information (short of a one time pad, which is the only way to get true noise) has an underlying structure in the data operated on. Since the key repeats the transformation for the length of the data, the distribution of the codes cannot be guaranteed.
i.e. Encrypted information will stand out as structured data.
Which only makes sense when you think about it. Information theory doesn't mess around. You cannot destroy information. (The black hole experiments confirmed that.) Thus the structure of the information will remain, no matter how much you try to obscure its existence.
We never reached an agreement, so I guess we'll have to let this article finally settle it. l-)
Javascript + Nintendo DSi = DSiCade
The company has "innovations" in it's name, so their product probably won't work.
If it did work against true crypt, which is a yard stick of well implemented encryption, I'm sure they'll come up with a counter measure by the next minor release.
Also: In before XKCD strip.
Won't someone please think of Linus and RMS? *ducks*
No info in the article. It is an advert for some Windows only software.
Repeat, this is an advert, not an article.
Down With Slashdot BETA!!! I've been around the corner and seen the oliphant; you can only abuse me from your perspecti
Usually. Consequently, there isn't much to be found.
This is probably another application of the Benford's law.
The Wikipedia page on TrueCrypt already indicates that the volumes can pretty much be detected since they are always divisible by 512, it's just impossible to PROVE they are TrueCrypt volumes...
Be enlightened: http://en.wikipedia.org/wiki/TrueCrypt
Wow, the quality of Slashdot has really been going down lately. Now any random fraud can submit his misleading material and it gets accepted to front page just because it sounds interesting? Is this actually tabloid or serious news for nerds who understand what the talk about?
In short, this is yet another lame attempt to make money by posting bogus claims about a popular product.
First, hidden volumes are the only kind of steganography that TrueCrypt offers. Second, if you read the TrueCrypt documentation, you'll learn the following about hidden volumes vs. dynamic:
On Linux or Mac OS X, if you intend to create a hidden volume within a file-hosted TrueCrypt volume, make sure that the volume is not sparse-file-hosted (the Windows version of TrueCrypt verifies this and disallows creation of hidden volumes within sparse files).
Furthermore, when I try to create a dynamic TrueCrypt volume, TrueCrypt displays a big warning saying that dynamic volumes are insecure. That's right. Insecure.
So again, I demote this story as total and utter bogus motivated by the vision commercial gain.
Think "For your Eyes Only" where the Bond girl says, in reference to the ships log, "{My father} used a special form of short-hand. Only I can read it."
TCHunt (free tool, you can find it with google) has worked for some time doing this exact thing on Truecrypt encrypted containers.
Actually tried it out last night. It does get false positives, but on my system it did indeed manage to find what it was looking for (total: 4 false positives, 1 true positive).
It works by detecting hidden patterns that don't exist in a random file.
That would be equal to breaking AES or the mode of operation (XTS).
If they could distinguish the AES-XTS ciphertext from random data, they would be famous in the cryptographic community instantly. However, these fraudsters obviously cannot do anything like that. They are just posting a bunch of lies hoping to earn big money on it.
This makes perfect sense. There's always been a way to distinguish between pseudorandom and random value sets. There's also no reason to believe an encryption algorithm could somehow start producing genuinely random values, so it follows that encrypted value sets should be distinguishable form truly random value sets.
I'm pretty familiar with TrueCrypt, but I don't know what a TrueCrypt "Dynamic" file is. Are they just talking about an encrypted virtual volume?
Anyway, I'm pretty sure this is BS. I think they're just doing regular entropy tests on files. That will tell you when you have random data. A good test might be able to distinguish a large amount of compressed data from encrypted data since compressed data does have a little redundancy (emphasis on "might" and "little").
But I guarantee that they are not detecting any redundancy in ciphertext. Detecting even a small amount of redundancy in the output of any modern cipher algorithm (like AES or Twofish) would be a HUGE cryptanalytic result. It would be front page news (in cryptographic circles).
In summary, I'm positive that they can't distinguish between a TrueCrypt volume and true random data.
Put up or shut up.
If the geiger counter does not click, the coffee, she is not thick.
...If the "hidden encrypted file" is stored on an encrypted drive? (say a drive that's mounted and decrypted automatically in say an OS like Linux)
Would this software detect the pattern in the "encrypted hidden file" of the underlying encryption of the drive it's stored on?
Could they prove it's an encrypted file rather than a junk file on an encrypted drive?
Reasonable doubt?
TCHunt found all of my TrueCrypt volumes. It's free too. http://16systems.com/TCHunt/index.php
hidden stenographic information can really put a damper on employment prospects for secretaries
who wants to hire stenographers when the stenographic information is already hidden therein?
must be some cutting edge dictation software that actually hides the text in the audio
next you'll be telling me we can do away with typewriters!
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
[pb@localhost ~]$ tail ~/.bash_history
less GnosLoadPDFForms.pdf
file GnosLoadPDFForms.
mv GnosLoadPDFForms.pdf GnosLoadPDFForms.fdf
file GnosLoadPDFForms.fdf
evince GnosLoadPDFForms.fdf
less GnosLoadPDFForms.fdf
su
acroread GnosLoadPDFForms.fdf
top
Disk space is cheap.
Have a large one-time pad on one drive. For good measure, encrypt it.
Have an encrypted drive, and inside it have a one-time-pad-encrypted container file.
To keep from being toast if someone grabs both disks, have the one-time-pad-decoder program grab the bits from the one-time pad in some non-obvious order.
For example, when mounting the one-time-pad-encrypted container file, the mounter could ask for:
1) starting offset
2) byte-interleave
3) additional XOR string
The starting offset says start reading the one-time-pad file at byte OFFSET instead of byte 0.
The byte-interleave is how many bytes to skip between each read.
The additional XOR strong is an additional string that is applied as if it was a one-time pad, but it's used repeatedly. It won't offer a whole lot of protection on its own but it is just one more roadblock to decryption.
If the one-time pad is
01010111 11110000 00001111
and the data is
11111111 11111111 11111111
and the one-time XOR string is
01010101
and the starting offset is 0 and the byte-interleave is 0,
then the encrypted data is the xor of
01010111 11110000 00001111
11111111 11111111 11111111
01010101 01010101 01010101 <-- note the repetition
which is
11111101 01011010 10100101
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Ummm, brains...
Are they just checking for the existance of the evil bit?
Who would win this election: Andrew Weiner vs Andrew Weiner's weiner.
I do a lot of data acquisition for work and in grad school. I've got lots of my data on my drive. They are written in binary formats of my design. There is lots of repetition, there are no headers, who knows what my data looks like to someone who doesn't know the "decoder ring" to unpack it.
That doesn't mean that my files are kiddie pron or directions to make a dirty bomb.
Sheldon
This sort of reminds me of that movie with Russel Crowe... "A Beautiful Mind", where Crowe's character thinks he is seeing patterns in random newspaper and magazine articles.
I guess anyone who has a drive full of files made of random bits, with a few encrypted files mixed in, is going to be in trouble!
..will it crash scanning /dev/urandom?
Because it starts generating pseudo-random bits when the buffer with "true" random bits is exhausted.
Disclaimer: No, I didn't read TFA.
Um, instead of running that as a constant cron job, which means history data is still being written to disk, try changing the mode of the file.
example:
echo > ~/.bash_history
chmod ugo-w ~/.bash_history
Actually, the XKCD strip is wrong. What will actually happen is that there are no hidden or encrypted files, after all who really encrypts his own drive (physical access = bye bye security anyway), and that therefore you cannot even give your torturers the information they're looking for even if you want to, because it doesn't exist.
Then they've also discovered a miracle compression algorithm! Useless if somebody trademarks "FIzip" first.
Any bets it will trigger false positives on random chunks of pseudorandom data?
Wonder how many individuals will be subjected to rubber hose cryptanalysis based on a false positive from a tool like this.
Actually crypto systems are evaluated by a concept called indistinguishability. The key idea, is that an attacker cannot distinguish the output of a cipher algorithm from a uniform and random distribution.
For example if you use AES in CBC mode, there is no way of distinguishing its output in polynomial time from one truly random. It's specified in polynomial time because, for example AES in CBC will cycle after 2^128 blocks generated.
My guess is that there must be an implementation or a design error in TrueCrypt that should be fixable.
Or maybe this company is full of BS..
then we'll have a cold war of encryption.
They're using their grammar skills there.
If you bother to look at the list of supported file types they recognize, they DO list Truecrypt dynamic containers, but DO NOT list Truecrypt fixed containers. Probably there is some start-of-piece sign in the dynamic, growable containers but not in the others. Given the variety of ways TC can encrypt a volume, this is perhaps plausible. Also they have not been using pure ECB mode on disks for some time (though that is hardly an innovation; my early 1980s VMS cryptodisk managed that, done a bit differently). If you don't want your disk to have the same pattern of encrypted data at the start of any volume (and there tend to be lots of predictable plaintexts in the start of filestructures), you must find some way to encrypt. Block number is a convenient additional number that can be used which does not interfere with the ability to extract data in random order. Normal chaining algorithms assume sequential access, which is very very very poor for a disk. Any of you blokes who commented ever implemented either???
This is complete sensationalist crap. Truecrypt isn't broken, (probably) nor are any of the other programs they possibly claim to have broken.
This is easy to test for yourselves folks, I just did it in 5 minutes.
dd if=/dev/urandom of=/home/me/somefile.jpg bs=512 count=10000
Performing this command and then scanning the resulting file with "File Investigator" results in the file being detected as a headerless encrypted data file.
Whoever pointed out that they simply identify any randomly filled binary file of a size of a multiple of 512bytes is correct.
TrueCrypt doesn't use ECB mode, hasn't for some time, etc etc etc. Stop freaking out every time someone claims to have broken it.
Nice try, but unless your home directory isn't writable by you, bash will unlink the unwritable .bash_history and create one that it can write to.
Establish a system whereby all your data is re-encrypted daily using a random keyfile that is generated every day and stored on the Internet somewhere, not on your own machine (someplace where you know they don't make backups). The keyfile is deleted at midnight. If you have not re-encrypted your data with a new keyfile by the time the old keyfile is deleted, the data is irrecoverable.
When they seize your computer, invoke your right to silence and stall until midnight.
You risk losing all your data that way if you don't get it re-encrypted before the keyfile goes away, but for some it may be worth the risk.
I piss off bigots.
While quantum data may be "truly" random, as far as we know, there are perfectly good pseudo-random generators that are guaranteed not to demonstrate a pattern for over 2^70 iterations. That should be good enough for anybody who doesn't need to do anything heavier than randomly distribute every particle in a galaxy.
Does it *prove* the file/partition is encrypted or does it just strongly suggest that it is suspicious?
i think you would still have plausible deniability.
---- Booth was a patriot ----
Haha, really? Genius coders from the GNU camp at work again, lol.
I guess you better send in my secretary, then. She knows how to take my dictation.
Oh, you meant steganography... Never mind.
That is all.
Properly encrypted files don't contain significant "patterns that don't exist in random files."
If the cryptography is actually secure, then its result will be indistinguishable from random noise. See the wikipedia article on crypto random number generators, for starters. Any departure from this represents a security vulnerability. That's not to say that every modern cryptosystem is totally secure, but it seems highly unlikely that they all contain similar types of non-randomness.
I would guess that a better way to detect encrypted files is to look for total randomness. Or at least something that's nearly so. There's hardly any other reason to have a file of any size with random contents. Any file that's useful in its current form almost by definition contains some significant structure.
My bicyles
That's just why XTS is being used instead of ECB. Even if every block would contain same plaintext data, encrypted output is always different. And for speed and random access, any chaning modes aren't being used like CBC.
Put this way:
The number of file types is huge, but you can probably write enough filters to spot all the media, document and executable type of files just by looking at their headers. Any of those files that don't parse correctly are items of interest. Any files that don't match one of the filters are likewise items of interest. Any files that are of a type that you wouldn't expect, such as SunOS binaries on a Windows box, are also interesting.
Process the items of interest for randomness, those that are random are likely encrypted files with some certainty.
The problem is identifying and cataloging all the various file types.
Fear my one time pad!
It seems that at least Ubuntu and Windows Vista are using tricky thumbnail cache, which is saved to system path. So it's nice to find thumbnails of all those so secret photos from it. ;)
Oops, I didn't have those, I don't know what those are. Err...
This tool can't tell the difference between a TrueCrypt container and a random file generated using a basic subtractive random number generator algorithm. Results here.
There's a simple solution to this - cat your encrypted file with a certain length of /dev/random data, both front and back, and remember the offsets. Then, when you want to access it, dd the encrypted part into a ramdisk, then dd it back when you're done. Problem solved.
95% of all computer errors occur between chair and keyboard (TM)
Nothing is ever certain. Maybe FTL is possible. Maybe telepathy exists. Maybe P=NP.
But to think that a small startup from nowhere did an incredible cryptographic breakthrough (distinguishing good ciphertext from good noise is a world achievement), and then what they do with it is put it in a friggin' shareware, rather than claim the world fame they should deserve, or go to some agency and get bajillions kadjillions or killed... *sigh*. That's just bad reporting and crowd hype and crap.
And yes, I bothered to download it and test it: dd of 1024k from /dev/urandom vs. an 1024k losetup aes encrypted ext3 filesystem with a subset of my /etc directory. It said urandom was headerless crypto and that the encrypted losetup was some TrueCrypt file.
Crypto isn't something you do in your back yard any more than time machines are, numnuts!
I have a 2MB Truecrypt volume, that I keep for all my passwords and other data I need while traveling.
It detected that 2MB file as a headerless encrypted file.
So your alleged 15MB test in step 4 is bunk.
Just going off topic for a second:
If you're running Windows 32 and you have 4GB of RAM in the system, you don't really need a swap file. You can't go over 4GB of memory on a 32 bit system (with some exceptions that you most likely won't ever see at home), a swap file would just be for debugging. So, if all you're doing is playing games & browsing, why bother with a swap file?
Naturally, the success rate and, most importantly, the fraction of false positives are not provided. Without these numbers, it's not only useless, but misleading and dangerous.
I'm don't think the bar of soap thing is a greatest idea if you think it all the way through.
So you put it in the way of someone big a violent...
So how are you supposed to get it out after they pound it in?
This problem is actually very simple to solve. Create a large crypto volume. Use only a portion of it for data, and fill the rest with random bits. If your ratio of data to random bits is good enough, it should fool the statistics. You may also weave the random data into your crypto or alternate bytes, which would have the a similar effect.
I suppose bash can forcibly remove it, but it's never happened to me. I just tried it.
I once read about one technique some protesters looking to interfere with bomb detection units. They simply all went to the range a shot a couple hundred rounds. (Total cost: about two or three bucks if you use a .22).
All these people then attend the event. The alarm gets set off so many times, it is useless. The protesters do nothing else, no signs, no slogans. To the untrained eye, the problem would seem to lie with the machine - there's just too many false positives.
Maybe a similar technique could be used with truecrypt?
Let's say I have a relatively small encrypted volume I use to keep personal data (financial records, documents under attorney client privilege, HIPPA protected information... lots of important things don't actually take a lot of space.)
Let's say there's one 1GB file that is an encrypted volume. There's also nineteen other files which are really just a large collection of random data. Then it becomes much harder to perform any sort of cryptographic analysis... too many false positives.