Google Publishes Zopfli, an Open-Source Compression Library

alphadogg writes "Google is open-sourcing a new general purpose data compression library called Zopfli that can be used to speed up Web downloads. The Zopfli Compression Algorithm, which got its name from a Swiss bread recipe, is an implementation of the Deflate compression algorithm that creates a smaller output size (PDF) compared to previous techniques, wrote Lode Vandevenne, a software engineer with Google's Compression Team, on the Google Open Source Blog on Thursday. 'The smaller compressed size allows for better space utilization, faster data transmission, and lower Web page load latencies. Furthermore, the smaller compressed size has additional benefits in mobile use, such as lower data transfer fees and reduced battery use,' Vandevenne wrote. The more exhaustive compression techniques achieve higher data density, but also make the compression a lot slower. This does not affect the decompression speed though, Vandenne wrote."

Comments

Overhyped

[Anonymous Coward]

This team is clearly just trying to make a name for themselves. It improves over gzip by a mere 3% or so, but takes an order of magnitude longer to compress.

Their underlying implemented might be cool research. But it's practical merit is virtually nil.

Now, cue the people who are going to do some basic arithmetic to "prove" me wrong, yet who probably don't even bother using gzip content-encoding on their website right now, anyhow.

Re:Overhyped

[TeknoHog]

If I understand this correctly, the point is to be compatible with zlib decompression. Obviously, you can bet much better compression with xz/lzma, for example, but that would be out of range for most browsers.

Re:Overhyped

[Baloroth]

Actually, they state that the 3-8% better maximum compression than zlib is 2-3 orders of magnitude longer to compress.

I can't imagine what kind of content you're hosting that'd justify 3 orders of magnitude compression time to gain 3% compression.

Static content that only has to be compressed once, yet is downloaded hundreds of thousands or millions of times. 3-8% is a pretty significant savings in that case.

Re:Overhyped

[sideslash]

It improves over gzip by a mere 3% or so, but takes an order of magnitude longer to compress [...] it's practical merit is virtually nil.

Maybe it's useless to you as a developer(?), and to most people. However, you benefit from this kind of technology all the time. Compare this to video encoding, where powerful machines spend a heck of a lot of time and CPU power to gain extra 3%'s of compression to save bandwidth and give you a smooth viewing experience.

This tool could have many useful applications for any kind of static content that is frequently served, including web services, as well as embedded content in mobile games and other apps. Every little bit of space savings helps (as long as it isn't proportionally slower to expand, which the article says it stays comparable).

Re:Overhyped

[Trepidity]

One example that comes to mind: Android APKs use the zip format.

Re:Overhyped

[K. S. Kyosuke]

But it's practical merit is virtually nil.

...unless you're a large web-based company serving terabytes of identical textual files to end users using deflated HTTP streams.

Re:Overhyped

[K. S. Kyosuke]

In addition to all the other explanations of how you missed the point, Deflate is also used in PNG. This will allow you to make smaller PNG files, too, which can be quite a significant part of your bandwidth.

Well, If you're Google and you detect Chrome on the client side, it might be even better for you to serve a WebP version instead. Out of a random sample of 1,000 PNG files, a lossless WebP version was at least 20% smaller in more than 50% of the cases (link).

Re:Overhyped

[AliasMarlowe]

Android APK

The hosts troll is a robot? Somehow, I'm not surprised.

Re:Overhyped

[nabsltd]

For example, assuming browsers incorporate the capability to decompress it, lowering the bandwidth of Youtube by ~3% is an achievement.

I don't know why people keep mentioning Youtube, since all videos are already compressed in such a way that pretty much no external compression is going to gain anything.

Although when compressing a video Zopfli might result in a smaller file compared to gzip, that doesn't mean either will be smaller than the original. All H.264 files should be using CABAC after the motion, macroblock, psychovisual, DCT, etc. stages, and that pretty much means that the resulting files have as much entropy per bit as possible. At that point, nothing can compress them further.

Re:Overhyped

[citizenr]

Word, when I'm downloading the latest pirated release of a 1080p movie

"word", and intend to download zipped h.264 files leads me to believe you are retarded.

Wow, gzip -9 is very competitive for most usages

[Antony T Curtis]

Looking at the data presented in the pdf, it seems to me that gzip does a fantastic job for the amount of time it takes to do it.

The interesting bit is this:

[mrjb]

"Zopfli is a compression-only library, meaning that existing software can decompress the data." (source: http://littlegreenfootballs.com/page/294495_Google_Compression_Algorithm_Z). As long as the compression can be done on cached pages, hey- that's another 3-8% more people served with the same amount of bandwidth, without any additional requirements on the client side.

Re:Wow, gzip -9 is very competitive for most usage

[Anonymous Coward]

Looking at the data presented in the pdf, it seems to me that gzip does a fantastic job for the amount of time it takes to do it.

Pfft. Another blatant corporate shill for gzip early in the Slashdot comments. You can't trust anybody on the internet these...

Oh, wait, the data actually does say that. Huh. That's... a really weird feeling, having someone on the internet legitimately say something's good and have data to back it up.

JavaScript libraries, for one thing

[DragonWriter]

I can't imagine what kind of content you're hosting that'd justify 3 orders of magnitude compression time to gain 3% compression.

In anything that is static enough that it will be downloaded many times in its lifetime, and not time sensitive enough that it needs to be instantly available when generated, very small gains in compression efficiency are worth paying very large prices in compression.

If you, for just one of many Google-relevant examples, host a fair number of popular JavaScript libraries (used on both your own sites -- among the most popular in the world -- and vast numbers of third party sites that use your hosted versions) and commit, once you have accept a particular stable version of a library, to hosting it indefinitely, you've got a bunch of assets that are going to be static for a very long time, and accessed very large numbers of times. One time cost to compress is going to be dwarfed by even a miniscule savings in transfer costs for those.

Re:Wow, gzip -9 is very competitive for most usage

[DragonWriter]

Yes, and gzip isn't so slow that it can only be used on static content. Even if you always generate into a cached version, do you really want to spend 81x the CPU time to gain a few percent in compression, and delay the content load on the client each time that happens?

Why would you recompress static content every time it is accessed? For frequently-accessed, static content (like, for one example, the widely-used JavaScript libraries that Google hosts permanently), you compress it once, and then gain the benefit on every transfer.

For dynamic content, you probably don't want to do this, but if you're Google, you can afford to spend money getting people to research the best tool for very specific jobs.

Re:Wow, gzip -9 is very competitive for most usage

[n7ytd]

Looking at the data presented in the pdf, it seems to me that gzip does a fantastic job for the amount of time it takes to do it.

So the obvious conclusion is that what we need is a gzip -11 option.