Slashdot Mirror
Exhaustive Data Compressor Comparison
crazyeyes writes "This is easily the best article I've seen comparing data compression software. The author tests 11 compressors: 7-zip, ARJ32, bzip2, gzip, SBC Archiver, Squeez, StuffIt, WinAce, WinRAR, WinRK, and WinZip. All are tested using 8 filesets: audio (WAV and MP3), documents, e-books, movies (DivX and MPEG), and pictures (PSD and JPEG). He tests them at different settings and includes the aggregated results. Spoilers: WinRK gives the best compression but operates slowest; AJR32 is fastest but compresses least."
Nothing to see. High compression = slow and low compression = fast. umm duh?
I never would have guessed that there was a tradeoff between the quality and speed of compression! No way! Next they'll be saying things like 1080p HD offers quality at the expense of computational power required!
Screw speed and size reduction. All I want it compatibility with other OSs (i.e., fewest things that have to be installed on a base OS to use it). For that, I'd have to say Zip and/or gzip wins.
So that's why smaller computers are slower, right?
I fill an old station wagon with backup tapes, and then put it in the crusher.
Not every software achieves maximum efficiency. It is perfectly imaginable that a compressor could be slow and bad. It is nice to see that these compressors did not suffer that fate.
as its slashdotted
this site
http://www.maximumcompression.com/
has been up for years and performs tests on all the compressors with various input sources, much more comprehensive
These two formats are still widely used out there, and why are we compressing MP3's?
yep, looks like they're using WinRK on the fly to decompress the website from storage
I read this earlier today through the firehose. It was interesting, but the graphs are what struck me. It seems to me all the graphs should have been XY plots instead of pairs of histograms. That way you could easily see the relationship between compression ratio and time taken. Their "metric" for showing this, basically multiplying the two numbers, is pretty bogus and isn't nearly as easy to compare. With the XY plot the four corners are all very meaningful. One is slow with no compression, one each good compression/time, and the sweet spot of good compression and good time. It's easy to tell those on two opposing corners apart (good compression vs good time), where as with the article's metric they could look very similar.
Still, interesting to see. The popular formats are VERY well established at this point (ZIP in Windows and Mac (stuffit seems to be fading fast), and GZIP and BZIP2 on Linux). They are so common (especially with ZIP support built into Windows since XP and also built into OS X) I don't think we'll see them replaced any time soon. Of course, with CPU power getting cheaper and cheaper we are seeing formats that are more and compressed (MP3, H264, Divx, JPEG, etc) so these utilities are becoming less and less necessary. I no longer need to stuff files on floppies (I've got the net, DVD-Rs, and flash drives). Heck, if you look at some of the formats they "compressed" (at like 4% max) you almost might as well use TAR.
Comment forecast: Bits of genius surrounded by a sea of mediocrity.
Nice comparison, but there's really only two that matter (at least on PCs):
ZIP for cross-platform compatibility (and for simplicity for less technically-minded users).
RAR for everything else (at 3rd in their "efficiency" list, it's easy to see why it's so popular, not to mention ease of use for splitting archives, etc).
"Nothing strengthens authority so much as silence." - Charles de Gaulle
This is a poor article on several points. First, the entropy of the data in the files isn't quantified. Second, the strategy used for compression isn't described at all. If WinRK compresses so well on very high entropy data, there must be some filetype specific strategies used.
Versions of the programs aren't given, nor the compile-time options (for the open source ones).
Finally, Windows Vista isn't a suitable platform for conducting the tests. Most of these tools target WinXP in their current versions and changes to Vista introduced systematic differences in very basic things like memory usage, file I/O properties, etc.
The idea of the article is fine, it's just that the analysis is half-baked.
What's the point of compressing JPEG,MP3,DivX etc since they already do the compression? The streams are close to random (with max information) and all you could compress would be the headers between blocks in movies or the ID3 tag in MP3.
the most interesting thing about text compression is that there is only about 20% information in the english language (or less). yes, that means that 4/5ths of it is meaningless filler. filled up with repetitive patterns. as you can see, i really didn't need four sentences to tell you that, either.
i wonder how other languages compare, and if there is a way to communicate much more efficiently.
7-zip cribsheet:
weak on retarded things to zip like WAV files (use FLAC) mp3's, jpegs and divx movies.
7zip does quite well in documents (2nd) and ebooks (2nd) 3rd on MPEG video, 2nd in PSD
also i expect 7zip will improve in higher end compressions settings, when possible i give it hundreds of megs and unlike commercial apps 7zip can be configured well into the "insane" range
Snowden and Manning are heroes.
These days, file compression is pretty much only used for large downloads. In those instances, you really have to use either gzip, pkzip, or bzip2 format, so that your users can extract the file.
Yes, having a good compression algorithm is nice, but unless you can get it to partially supplant zip, you'll never make much money off it. Also, most things these days don't need to be compressed. Video and audio are already encoded with lossy compression, web pages are so full of crap that compressing them is pointless, and hard drives are big enough. Although, I haven't seen any research lately about whether compression is useful for entire filesystems to reduce the bottleneck from hard drives. Still, I suspect that it is not worth the effort.
I have to admit I switched over/back to ZIP about a year ago for everything for exactly this reason. yeah, it meant a lot of my old archives increased in size (sometimes by quite a bit), but knowing that anything anywhere can read the archive makes up for it. ZIP creation and decoding is supported natively by Mac and Windows and most Linux distros right from the GUI, so it makes it brain-dead simple to deal with.
Recursive: Adj. See Recursive.
The L-Zip project at http://lzip.sourceforge.net/ seems to be down right now but it should be included in any file compression comparison. It could reduce files to 0% of their original size and it was quick too.
It was so good at what it did that I bet Microsoft bought them out and are going to incorperate the technology into Windows.
See also: the Archive Comparison Test. Covers 162 different archivers over a bunch of different file types.
It hasn't been updated in a while (5 years), but have the algorithms in popular use changed much? I remember caring about compression algorithms when I was downloading stuff from BBSs at 2400 baud, or trading software with friends on 3.5" floppies. But in these days of broadband, cheap writable CDs, and USB storage, does anyone care about squeezing the last few bytes out of an archive? zip/gzip/bzip2 are good enough for most people for most uses.
Repton.
They say that only an experienced wizard can do the tengu shuffle.
It seems odd that they didn't include executables/dlls in the comparison (where maxmumcompression.com does). I also find it odd that they are compressing items that normally don't compress very well with most data compression programs (divx/mpegs/jpegs/etc). I'm guessing this is why 7-zip ranked a bit lower than most.
I did some comparison last year, and found 7-zip to do the best job for what I needed (great compression ratio without requiring days to complete). It also doesn't take into account the network speed at which the file is going to be transmitted. I use 7-zipfor pushing application updates and such to remote offices (most over 384k/768k WAN links). Compressing w/ 7-zip has saved users quite a bit of time compared to winrar or winzip.
I would definitely recommend checking out maximumcompression.com (As others have, as well) over this article. It goes into a lot greater detail.
> UM, yeah, the dataset includes WAV files. Try flac [sourceforge.net].
...
> Then you will have exhausted a little more of the compression programs available.
You are aware that all the tools tested are general purpose compressors, and FLAC is not, aren't you?
Otherwise, you would also have to talk about Wavepack, Monkey Audio, Shorten and others.
And those are only the loseless audio codecs. What about lossy codecs?
What about all those different formats for pictures? They compress data as well.
And what about the different video codecs?
I have been thinking about creating a new language with about 60 or so words. The idea is that you don't need a lot of words when you can figure out the meaning by context. Strong points are that the language would be very easy to pick up, and you would get that invigorating feeling of talking like a primitive cave man.
As an example of the concept, we have the words walk and run. They are a bit too similar to be worth wasting one of our precious few 60 words. Effectively, one could be dropped with have the other taking on a broader meaning without any real repercussions. The words sit and shit are also fairly similar. When you have a guest over, you can say something like, "Please, shit down." Because of context, it would be all okay. Just remember, there is a difference between shitting on the toilet and shitting in the toilet.
Once you start despising the jerks, you become one.
Er... did ya check out the comparisons? As you can see here here jpeg at least can be compressed considerably with Stuffit. According to this the program can "(partially) decode the image back to the DCT coefficients and recompress them with a much better algorithm then default Huffman coding." I've no idea what that means, but it does seem to be more thorough and complex than what you wrote.
So they've compressed it to 11. I'd say that's a step forward.
It's closed sourced and proprietary though. Someone needs to make an open-source RAR compressor - the problem is you can't use the official code to do that (as it's specifically in the licence), but you could use unrarlib as a basis...
Even it the amount of additional compression is insignificant, ZIP, RAR, etc. are still very useful as container formats for MP3, JPG, etc. files since it's easier to distribute 1 or 2 .ZIP files than it is 1000 individual .JPG files. And if you're going to package up a bunch of files into a single file for distribution, why not use the opportunity to save a few kilobytes here and there if it doesn't require much more time to do that?
"People that quote themselves in their signatures bother me" - athakur999
By default, Stuffit won't even bother to compress MP3 files. That's what it shows an increase in file size (for the archive headers) and why it is the fastest throughput (it's not trying to compress). If you change the option, the results will be different.
I imagine some other codecs also have similar options for specific file types.
Still, in both cases, it works; who can argue with that.
Back in the early/mid 90s I was pretty obsessed with data compression because I was always short on hard drive space (and short on money to buy new hard drives with); as a result I tended to compress things using whatever the format du jour was if it could get me an extra percentage point or two. Man, was that a mistake.
Getting stuff out of some of those formats now is a real irritation. I haven't run into a case yet that's been totally impossible, but sometimes it's taken a while, or turned out to be a total waste of time once I've gotten the archive open.
Now, I try to always put a copy of the decompressor for whatever format I use (generally just tar + gzip) onto the archive media, in source form. The entire source for gzip is under 1MB, trivial by today's standards, and if you really wanted to cut size and only put the source for deflate on there, it's only 32KB.
It may sound tinfoil-hat, but you can't guarantee what the computer field is going to look like in a few decades. I had self-expanding archives, made using Compact Pro on a 68k Mac, thinking they'd make the files easy to recover later, which didn't help me at all now -- a modern (Intel) Mac won't touch it (although to be fair a PPC Mac will run OS 9 which will, and allegedly there's a Linux utility that will unpack CPP archives, although maybe not self-expanding ones).
Given the rate at which bandwidth and storage space are expanding, I think the market for closed-source, proprietary data compression schemes should be very limited; there's really no good reason to use them for anything that you're storing for an unknown amount of time. You don't have to be a believer in the "infocalypse" to realize that operating systems and entire computing-machine architectures change over time, and what's ubiquitous today may be unheard of in a decade or more.
"Ladies and gentlemen, my killbot features Lotus Notes and a machine gun. It is the finest available."
Theres also another, rather uncommon format that wasn't tested that is somewhat important. UHARC- File extension UHA. It is dog slow, but offers better compression than probably any of the others. It is still used by software pirates with their custom install scripts, and I have seen it in official software install routines as well.
You can keep Rar and zip and toss out the others, but the UHA extension (or a dummy extension) will probably exist on your computer at some point in time.
Even those who arrange and design shrubberies are under considerable economic stress at this period in history.
RAR irritates me though. It's rare enough that I usually have to dig up a decompresser for it and install it special for just one file and then I never use it again. I just don't like having to deal with files that require me to install new software just so I can use that one file. In that vein, I really don't think the article is relevant. I certainly won't use novelty file formats unless it looks like it has "legs". It's not like I want to make a file that becomes useless when the maintainer of the decompression utility loses interest and it goes away.
Andrew Tridgell's rzip wasn't on there either.
http://samba.org/junkcode/
Tridge is one of the smart guys behind samba. And rzip is pretty clever for certain things. Just ask google.
I take it you come from a planet where very few people use Windows. Please, I'm curious to know, what are things like there?
The article conveniently forgets to mention whether the conpression tools are cross-platform (OSX, Linux, BSD) and/or open source or not.
That makes a lot of them utterly useless for lots of people. Yet another windows-focussed review, bah.
Give it am MD5 hash and a file length and it will compute all the possible files that could have produced the hash. Automatically filter our the invalid files and the set you're left with can't be that large.
There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter
for general purpose lossless compression. Most modern compression utilities out there mix and match the same algorithms which do the same thing.
With the exception of compressors that use arithmetic coding (which has patents out the wazoo covering just about every form of it), virtually all compressors use some form of Huffman compression. In addition, many use some form of LZW compression before executing the Huffman compression. That is pretty much it for general purpose compression.
Of course, if you know the nature of the data you are compressing you can come up with a much better compression scheme.
For instance, with XML, if you have a schema handy, you can do some really heavy optimization since the receiving side of the data probably already has the schema handy which means you don't need to bother sending some sort of compression table for the tags, attributes, element names, etc.
Likewise, with FAX machines, run length encoding is used heavily because of all the sequential white space that is indicative of most fax documents. Run length encoding of white space can also be useful in XML documents that are pretty printed.
Most compression algorithms that are very expensive to compress are usually pretty cheap to decompress. If you are providing a file for millions of people to download, it doesn't matter if it takes 5 days to compress the file if it still only takes 30 seconds for a user to decompress it. However, when doing peer to peer communication with rapidly generated data, you need the compression to be fast if you use any at all.
Nevertheless, most generaly purpose lossless compression formats are more or less clones of each other once you get down to analyzing what algorithms they use and how they are used.
It has poor compression and slows down the filesystem viciously, mostly due to fragmentation; I've see 200000 fragments in a single file!
I think the compression algoritim it uses is ZLW, you're lucky to get 1.5:1 in the best cases.
There are other issues, like a 20Gb compressed file giving fake disk errors (on a drive with 40Gb of free space) but generally the poor compression and performance is enough to ensure that you don't want to use it.
This reminds me of... pkunzip.zip
The saddest poem
*imagines parent comment spoken in the voice of comicbook store guy off the simpsons* . . .
heh
I only skimmed the article but what with all the hullabaloo about dual/quad core chips, why didn't they use "exhaustive" as an excuse to check out the parallelisability (if that's a word) of each compression algorithm? IIRC they didn't list the hardware they used or any of the switches they used, which is a glaring omission in my book.
;)
Of all the main compression utils I use, 7-zip, RAR and bzip2 (in the form of pbzip2) all have modes that will utilise multiple chips, often giving a pretty huge speedup in compression times. I'm not aware of any SMP branches for gzip/zlib but seeing as it appears to be the most efficient compressor by miles it might not even need it
It's mainly academic for me now though anyway, since almost all of the compression I use is inline anyway, either through rsync or SSH (or both). Not sure if any inline compressors are using LZMA yet, but the only time I find myself making an archive is for emailing someone with file size limits on their mail server. All of the stuff I have at home is stored uncompressed because a) 90% of it is already highly compressed and b) I'd rather buy slightly bigger hard drives that attempt to recover a corrupted archive a year or so down the line. Mostly I'm just concerned about decompression time these days.
Moderation Total: -1 Troll, +3 Goat
While the main thrust of JPEG is to do "lossy" compression, the final stage of creating a JPEG is to do lossless compression on the data. There are two different official methods you can use: Huffman Coding and Arithmetic Coding.
Both methods do the same thing: they statistically analyse all the data, then re-encode it so the most common values are encoded in a smaller way than the least common values.
Huffman's main limitation is that each value compressed needs to consume at least one bit. Arithmetic coding can fit several values into a single bit. Thus, arithmetic coding is always better than Huffman, as it goes beyond Huffman's self-imposed barrier.
However, Huffman is NOT patented, while most forms of arithmetic coding, including the one used in the JPEG standard, ARE patented. The authors of Stuffit did nothing special - they just paid the patent fee. Now they just unpack the Huffman-encoded JPEG data and re-encode it with arithmetic coding. If you take some JPEGs that are already compressed with arithmetic coding, Stuffit can do nothing to make them better. But 99.9% of JPEGs are Huffman coded, because it would be extortionately expensive for, say, a digital camera manufacturer, to get a JPEG arithmetic coding patent license.
So Stuffit doesn't have remarkable code, they just paid money to get better compression that 99.9% of people specifically avoid because they don't think it's worth the money.
Does my bum look big in this?
In it's efficiency graphs they order the negative scoring ratios wrong! Afterall, they considering something that adds 1MB in 2 seconds to be worse than one that increases the size by 1MB in 2 minutes. So doing the same thing *slower* actually ranks it ABOVE the other one. Plus, what matters, even for large files, is NOT the time for compression. What you REALLY want to compare is the ratio and the time for EXTRACTION on those settings. Any file will be compressed once, decompressed thousands of times. A minute longer to produce means little. A minute longer to extract for everyone extracting it matters a lot.
The article seems to be measuring the compression speed of each program with its native algorithm, it would have been better to do a set of programs with each algorithm first. As the article is comparing two variables at once, how good the algorithm is and how good the implementation in that program is, the results are slightly meaningless.
.zip and .tar.gz because you know people will be able to open them. Proprietary algorithm X may be really efficient but if no one can open it, who cares?
Having said that, do I really care in practice that much about if algorithm A is 5% faster than algorithm B? I personally do not, I care if the person receiving them can open them. So the second problem with the article is that it is one computer user on his own, in the real world you would just distribute
My little Linux and tech blog
Things are less blue.
(And I'm not speaking of the sky)
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
RAR has recovery records (settable percentage of each archive dedicated to ECC, default off) and recovery volumes (dedicated files with PAR-like recovery capabilities). "Keep broken files" can be used to extract from broken or truncated archives.
With Arithmic encoding however you'd encode each character according to the exact probability it has of occuring and write it as a fractional number between 0 and 1. For example, if you want to encode an "A", you'd pick a number between 0.0 and 0.2 (the lower 20% of our number); if you want to encode a "B", you'd use a number between 0.2 and 1.0 (the upper 80% of our number).
What you keep track of during encoding is the upper and lower bound of this number. So, when I want to encode the first "A", my lower bound is 0.0 and the upper bound is 0.2. The next character to encode is "B". We already now the range we can pick from is 0.0 - 0.2, but to encode a "B" we need to pick a number in the upper 80% of this range, so 0.04 to 0.20 (picking a number between 0.0 and 0.04 would encode another "A").
The next letter, another "B", would use a range 0.072 - 0.200. The 3rd "B" would narrow the range to 0.0976-0.2000. The 4th "B" narrows it to 0.11808 - 0.2000.
At some point, the upper and lower bound will have a few most significant digits in common that cannot change anymore. When this occurs, you can start writing these out as part of your compressed stream. For example, when we encode the 6th character (the 2nd "A"), the range becomes 0.118080 - 0.134464. The first two digits (0.1) can't change anymore now, so we can write them out, and just continue narrowing the range further for subsequent data to be compressed.
At some point, there'll be no more data to be compressed, and you then just pick a number (as convenient as possible) between the upper and lower bound you have established, write it out and end the stream. The process is the same when doing this with binary floating point numbers.