Slashdot Mirror
Fast File Encryption for Windows?
cryptoz wonders: "I've used numerous encryption applications
for both Windows and Linux over the past few years and have always been satisfied. Until I realized I needed to start encrypting large files (say 10 to 30 GiB), or at least a large number of small(er) files. I found that everything I use seems to take hours and hours to compress, encrypt and shred. Not to mention decompressing, decrypting and deleting on the other end.
Every web search I do on the topic seems to turn up mostly closed-source applications or snake oil, neither of which is acceptable. Does Slashdot have any suggestions for fast file encryption? I should make it clear that in my particular case, I do not need to have a perfect key or incredibly secure encryption, since it is not the weakest link (as I am susceptible to hardware key-loggers, CRT eavesdropping and the like). The encryption needs to be just strong enough, but most importantly, *fast*." This is a worthwhile question, but when dealing with files in the 10s of GB, can anything really be considered to be "fast"?
I'd say your best bet'd be TrueCrypt.
You linked to it yourself, so you should be aware of the strengths of the application. It does on-the-fly disk encryption with either whole partitions or disk image files, has absolutely no problem with massive disks (I have a 40GB image on a USB drive), and is pretty fast. My benchmarks come up with 50MB/s average throughput (around 56MB/s encrypting, 47MB/s decrypting) for 256bit AES encryption on my machine. TrueCrypt seems to cope well with files of any size, and while I can't say I've tried 30GB, 4.7GB DVD images work very well indeed.
One thing that really makes it stand out in your scenario is the ability to use keyfiles. This allows you to select one or more files that will be used (hashed?) with your password to secure your data against those hardware keyloggers. (Although, I would question whether encryption is really required if you aren't that bothered about security.)
The best part of TrueCrypt is that it is completely open-source. No closed/proprietary systems and no snake oil. For encryption on Windows, when the built in stuff doesn't cut it, TrueCrypt is the only way to go, IMHO.
I don't know if this is in the ballpark, but we Truecrypt on hard drive backups we take off site. It is open source which is nice. It allows you to mount a virtual hard drive that is either a file on an existing partition or as a sort of phantom partition that only TrueCrypt will see. It encrypts on the fly, hence it's usefulness to us. We just have a few usb hard drives. When we plug them in, we can mount them using a password or more elaborate means. It may be worth a peek.
http://www.truecrypt.org/
Security is the antithesis of comfort/ease of use.
Also, security can be increased to downright unusability, too.
It takes a man to suffer ignorance and smile
Be yourself no matter what they say
It's what I use and it's quite fast. I get about 52 MB/sec encryption speed and I'm loving it.
But you should look at TrueCrypt
The Statue of Liberty is America's lawn jockey.
Yes, a station wagon filled with tapes of 10GB+ files doing 80mph on a highway is going at a pretty fast clip in my opinion. YMMW.
With apologies to AS Tanenbaum.
An Indian-American Hindu committed to non-violent thought/speech/action alarmed by the global explosion of radical Islam
Ever tried this? SureCrypt
SureCrypt is an ultra small encryption program designed for fast processing of extremely large files. It can encrypt or decrypt files as fast as Windows Explorer can copy them. SureCrypt presents a flexible user interface with detailed record of all operations.
Just ROT2 the bits. Or is it ROT1?
Ok ... is it just me, or is TrueCrypt linked IN THE FREAKIN' POSTING ABOVE? Why is everybody's answer of "TrueCrypt" getting modded as informative?
GiB? Dude, just say GB, we all get it. It's a buttload of data.
I want to delete my account but Slashdot doesn't allow it.
The submitter's question linked to truecrypt as one of two programs he's tried and found not fast enough. I hear it's real nice, but he's already found it too slow for his needs.
I've found Apple's FileVault too slow for video on a 1.2GHz G4, but maybe on a G5 or Core Duo it's fast enough. That's AES-128 or 256, so maybe the TrueCrypt AES implementation just needs tuning for his hardware.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
TrueCrypt works like a v-twin on a golf cart.
Dear cryptoz, We'd like to discuss you encrytption concerns. With our vast experience in encryption and decryption, we believe through our highly effective questioning we can find the right product for you. Please arrive at our facility at Fort Meade at any time. Ask for the Best Interest of National Security Special at the front desk when you arrive. Sincerely, The NSA
Well, there's spam egg sausage and spam, that's not got much spam in it.
As far as shredding files goes, that isn't really connected with the encryption process, but more to your hard disk speed. Writing random bits to a 10-30 GiB file is going to take a while no matter what program you use.
Qualitas edurus commercium, nullus penitus net rimor, nullus deus beneficium
One thing that really makes it stand out in your scenario is the ability to use keyfiles. This allows you to select one or more files that will be used (hashed?) with your password to secure your data against those hardware keyloggers. (Although, I would question whether encryption is really required if you aren't that bothered about security.)
It all depends on the threat model. I could see desiring encryption without being bothered by keyloggers if you're worried about someone breaking into your car and stealing your laptop full of sensitive information. Most people won't break into your car to install a keylogger.
XOR against a repeated key would be ultra-fast but woefully insecure. When will people learn that it takes CPU cycles to encrypt that much plain-text? In just about every other field you don't get something for nothing; why should Cryptography be any different?
Simon
I suggest getting some hardware acceleration: the VIA EPIA boards use electrical interference in their traces to suppy entropy to a hardware encrypt/decrypt enginge that can achieve 25 Gb/s encryption. This is a 1.0GHz passively-cooled board with SATA ports, hardware MPEG2 decoding and all on a 17x17 cm^2 board.
but if you really want speed and want to pay for it:
http://www.tarari.com/
and you can accelerate your gneomics research while you're at it. Or whatever else you do, assuming its a CPU intensive process. (If you have a really slow I/O system, then this is a lot of wasted money; not for a friend of mine who works there though....)
Everyone who uses encryption is a terrorist and/or a child molester. If you're not doing anything wrong, what do you have to hide?
Personally, I videotape all my daily activities and archive them in case a law enforcement agency wants to know what I was up to on a particular date. I suggest you all do the same. Think of the children and 9/11!!!
Check BestCrypt. I've been using it for years, and like it (haven't tried 10Gb though). TrueCrypt looks like the same concept, so use the trial to speed test for comparison. It's not free, but it is available for windows and linux.
Also, be aware that your encryption choice will affect speed greatly. 448-bit is slower than 224 bit, etc. Also some algos are optimized - twofish is a pentium-optimized version of blowfish.
Is it specifically the encryption, or are the compression and shred (both of which often do not scale well) taking the most of the time?
Seagate recently released a self-encrypting hard-drive... does hardware level encryption at S-ATA link speed, or so they claim. More info: http://www.apcstart.com/site/dwarne/2006/06/263/se agates-self-encrypting-hard-drive
A fast and reliable way to encrypt a file is to sweep a strong magnet across your hard disc. Decryption of the files is more difficult and time comsuing, scientist are still working hard to find the final solution.
If you aren't that worried about security, you probably don't need 256-bit, 14 round AES. Consider finding software that uses Twofish or Blowfish, both of which are quite common in commercial software, and very fast. If you're really concerned with speed, you can use the XTEA algorithm. (AFAIK, no commercial software packages use this at the moment, so it might not be very helpful...)
http://en.wikipedia.org/wiki/XTEA
NTFS encryption seems to be pretty fast -- we use it for doing encrypted backups onto portable USB hard drives. The bottleneck seems to be the hard drive speed rather than the encryption. I think we put about 40GB on the USB drives in under an hour or so. Mind you, the drives aren't that fast, they're the little laptop drives, plus there's the USB overhead. Perhaps, rather than looking for a faster encryption program you need faster/more hard drives. Lots of data is slow to push around and if the source and destination are on the same disk, that's a lot of disk seeks to deal with.
Biham and Seberry's "Py" can clip along at approaching 2 cycles/byte. That means a high-spec machine could be decrypting that 30GB file in around ten seconds - far faster than it can read it from disk, in fact.
Xenu loves you!
Assume a sustained transfer rate of 30MB/s, which is quite good for a single hard disk. You won't get that much when transferring lots of small files. Reading 30GB takes 1000s or about 18 minutes, writing it back another 18 minutes, doing both takes longer, because interleaving both processes will lower the transfer rate. If you're shredding the old data, you can add in another 20 minutes per pass. So encrypting 30GB takes 60 minutes, probably a lot more, and there's nothing you could do about it in software.
Encryption itself... I seem to remember that TwoFish needs 26 clocks to encrypt 8 bytes on a Pentium. So your 2.6GHz CPU can encrypt 8GB/s (but the bus cannot deliver that much, I suspect). Add in some fudge factors for OS overhead and other tasks, and you're still two orders of magnitude below the IO time.
You need faster disks.
Hi,
I would recommend "KRYPTO", or more precise "KRYPTO 2.0/2006 Professional Multi User Professional Data Fullbit Coding Program". The program uses the best encryption possible (called 256 bit fullbit encryption). Read up on it here:
Kryptochef
The application even sports a friendly GUI that is easy to use and allows even novice users to encrypt files.
Cheers, Fogger
Um... I didn't do it!
PKZIP has built-in encryption, both their old-style proprietary algorithm as well as AES. It works, it's fast, and it has all sorts of other benefits. I use it all the time and I'm very happy with it.
I don't know if it's of any use in this situation but you might want to look into AxCrypt
http://axcrypt.axantum.com/
Software Freedom Day!.
The submitter's question linked to truecrypt as one of two programs he's tried and found not fast enough. I hear it's real nice, but he's already found it too slow for his needs.
I'm also amused by the submitter's "too slow" comment for TrueCrypt. I use it on my 4-year old laptop (a 1.7Ghz Pentium 4 mobile) and find that it's the hard drive that is the bottleneck rather then the CPU. I'm using the stock TrueCrypt settings for encryption algorithm (256bit AES, LRW mode) and hash (RIPEMD-160). I have two volumes on the laptop, one is a ~700MB TrueCrypt file volume used for extra sensitive data and the second is a full-disk encrypted FireWire drive attached to the unit (160GB).
Copying from the laptop's hard drive to the encrypted external FireWire drive gives me transfer rates of around 10-12MB/sec and uses up around 30% of my CPU. Which is not too shabby for a 4 year old laptop. I would hardly call it "too slow".
I just did the benchmarks for a 100MB buffer, the left number is speeds on my 1.7Ghz Pentium 4 mobile laptop CPU, on the right is performance of a 2Ghz Opteron 246 chip (TrueCrypt 4.2 is not multi-threaded so it only used one of the two chips installed in that system):
Blowfish 35.1MB/s 46.8MB/s
Twofish 21.3MB/s 40.6MB/s
AES 28.5MB/s 32.6MB/s
Serpent 11.7MB/s 34.3MB/s
CAST5 10.5MB/s 34.7MB/s
Triple-DES 6.2MB/s 12.0MB/s
Those are not scientificially rigorous tests, but the built-in benchmark tool shows that the laptop's P4 is capable of very high encrypt/decrypt rates. It also looks like Serpent/CAST5 algorithms possibly don't fit inside the CPU cache very well (the Opteron chip has a larger L2 cache) or Serpent/CAST5 use operations that are more efficient on the Opteron chip. I don't know enough about the individual characteristics to make more educated guessed then that.
It's a pity that TrueCrypt isn't multi-threaded, or the dual-CPU Opteron system would've scored even higher on the TrueCrypt benchmark. I've run the benchmarks for a few different sizes (10MB / 50MB / 100MB / 500MB) and the numbers all tend to add up the same way (within a few percentage points) across the board.
Wolde you bothe eate your cake, and have your cake?
Check out Steve's SecurityNow! podcast 41
to hear why & more about it:
http://media.grc.com/sn/SN-041.mp3
For slow modem users, here's the transcript:
http://www.grc.com/sn/SN-041.pdf
A list of his other podcasts:
http://securitynow.info/
http://www.interhack.net/people/cmcurtin/snake-oil -faq.html
We're all used to seeing just "GB", so when you see "GiB", it throws you off because it doesn't look right. Plus it looks like a word, in fact is a word (slang) in online FPS games. Plus it's a way of "showing off" for the technically "elite", aka the snobs who think they are so fucking smart and they just can't believe other people would prefer "gigabyte" over "gibibyte" :P
Joseph?
Good data, thanks! Sounds like the submitter is looking for algorithmic magic then, or his machine is just too old - blowfish is a nice fast algorithm - I use it for all my SSH tunneling work and it gives me about 3x over stock (3des?).
IIRC TrueCrypt is going to multithreadded-IO in a near-future release so that should help even more with heavy disk access.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
and that's good or bad????
The real solution is don't encrypt individual file. Encrypt the whole disk including free space. Takes awhile to initially encrypt but not a big performance hit on use.
Wow...if ever there was a site that screamed to be included in Schneier's Snake Oil Crypto page, this is it. Total rubbish.
Bear with me for a moment while I try to get to the point as directly as I can.
There's a program called "Tiny IDEA" which implements the IDEA cipher. It's written in assembler for DOS, and comes with source code; the executable is about 500 bytes. It was originally written by Fauzan Mirza (who has credibility in that he also won Bruce Schneier's $10,000 award for best attack on Twofish during the AES competition). It was later further optimized and improved by someone named Mark Andreas, who I've never heard of in any other context. I'm not qualified to judge the quality of these programs directly, but when you read the manual it's pretty clear that Andreas knew what he was doing.
Tiny IDEA inspired a slew of other "Tiny" encryption programs by other authors, some anonymous, which can be found (free, and mostly with source code) at http://www.afn.org/~afn21533/rgdprogs.htm
(This is an interesting little site largely devoted to privacy and encryption, run by some random cool old guy. I think he wrote some of the "Tiny" spinoff programs.) Of particular note are a couple of Blowfish implementations, because as far as I know Blowfish is the fastest of the strong algorithms. Again, I can't vouch for the quality of these programs, but at least they seem to have a good genealogy.
Now obviously Blowfish implementations are a dime a dozen, but the reason I mention these "Tiny" ones in particular is because they encrypt your original file in-place, right there on the same disk sectors where it already resides, instead of creating a separate output file. This means you probably don't need to shred the original as a separate step, which might save you a great deal of time.
But have you seriously considered using Windows built in Encrypting File System (EFS)? WHEN CONFIGURED PROPERLY, it can be both very secure and speedy.
Not sure what you are doing with the files (i.e. staying on your machine or being distributed, etc.) but the EFS might be a very simple and effective option. Microsoft's website actually has some fairly good articles about it's usage beyond the stupid-user stuff.
What's important to remember is that you MUST use Window's SYSKEY program in mode 2 or 3 in order for EFS to be secure... otherwise it can be cracked in minutes. Good luck!
First, read the forums and learn about the people who have lost all their EFS data because of the sloppiness of Microsoft.
In some cases EFS is tied to the computer on which it is installed. You cannot restore it to another computer, even if you have all the keys.
Were you thinking, oh this time Microsoft won't be sloppy?
An open source program called "Primary Cryption" seeks high security over speed.
But since you get the source code and it is well-commented, you could probably modify it yourself to be less secure ( you decide how much) and a lot faster.
It works under Linux/Wine, and It can also handle multiple files. (Confession: I wrote it, and need to make myself write a helper program to keep track of keys and make it easier to handle multiple files, but I haven't had the time.)
It crimps the submitters style to have skipping porno. What else are people doing to generate 30GB of data the needs to be encrypted.
Conformity is the jailer of freedom and enemy of growth. -JFK
get a mac turn on file vault problem solved
I found that everything I use seems to take hours and hours to compress, encrypt and shred. Not to mention decompressing, decrypting and deleting on the other end.
It sounds you don't know what TrueCrypt really does. Real-time transparent encryption does not "compress" nor "shred" anything.
Nice troll you got going there. Real nice.
Anyone else notice that the submitter is called 'cryptoz', or that his linked website, http://www.sheehy.ca/crypto/, is called "The Cryptography Center"?
Also the little matter of his website's description saying "This website is designed as a location for as many cryptography resources as possible. The intent is to collect a large number of articles for those who are interested in learning more, practical computer applications to download, lists of other resources, and an open forum for discussion on cryptographic techniques. As well, I hope this website will be a new home for those interested in the science of keeping secrets."
Am I the only one who thinks that this is someone looking to pick the brains of millions of nerds in order to populate their website? You really need to encrypt "30GiB"? Yeah, my arse you do matey.
Agreed. One point worth noting is that I can't think of many ways of producing a dataset that large where the data is produced faster than TrueCrypt can encrypt it. Don't store it on an unencrypted partition and then encrypt it for processing, produce it directly on the encrypted partition and then move the resulting data. Similarly, don't decrypt to local storage at the other end, use the file directly from the encrypted partition; chances are your consuming application (presumably some kind of data analysis or data mining tool) will be slower than the decryption too. You'll want two encrypted partitions so you can alternate which one you're writing to and which one you're moving, but that doesn't need any more space (20GB) than doing the encryption offline would.