Slashdot Mirror

← Back to Stories (view on slashdot.org)

Gzip on a PCI card

Posted by michael on from the what-will-they-think-of-next dept.

steve writes "The German tech news site heise.de is reporting here (in German, of course) about a PCI card developed by the Universiy of Wuppertal and Vigos AG being shown at CeBIT, which does Gzip compression in hardware, thus freeing the CPU to do other tasks. The PCI card can compress 32MB/sec, which is more than enough to compress a 100Mbit LAN in realtime. A future version will do 64MB/sec. The article mentions that this will be of particular interest for web servers. The card should be on sale by the end of the year."

8 of 141 comments (clear)

  1. Useful for netbackups too by walt-sjc · · Score: 5, Insightful

    Seems this would be a great help to those doing backups over a LAN. Shouldn't take too much to alter a version of tar , rsync, etc. to use this card.

    1. Re:Useful for netbackups too by walt-sjc · · Score: 3, Insightful

      Interesting, didn't know that. I just assumed it used the same code. Note that one of the cool things about open source is that you could swap out the compression code which is exactly what I was suggeting, so it wouldn't really matter what algorithm the code originally used. (of course it would no longer be compatible, but I'm also assuming that this wouldn't be an issue in this case for this application.) I normally don't use the built-in compression with rsync, instead I use the compression in ssh which I believe IS gzip.

      It would be Very cool if the card supported multiple compression algorithms. Considering that GNU tar supports bzip as well., this would definately be useful.

  2. bandwidth saving by buro9 · · Score: 5, Insightful

    the key to using gzip is really not to compress at too high a ratio... a low rate of compression offers a pretty sizeable saving in bandwidth for an acceptable CPU usage... once you edge up to the higher compression levels then you pay for it in the CPU and your app slows.

    i love the idea of a hardware based gzip... but i'd start by educating the software users on the cost vs benefit ratio of their existing configuration... i always seem to find that those who don't know what they're doing are the ones that have it set to maximum compression

  3. A bzip2 version would be nice ... by geirt · · Score: 4, Insightful

    I try to avoid bzip2 because it is so slow, even on modern hardware. bzip2 compresses very well, much better than gzip. A bzip2 version of this card makes sense ....

    --

    RFC1925
  4. Re:How cute but useless. by sporty · · Score: 2, Insightful

    Not really. Can you cheaply create a cluster of say.. 50 web servers, all that use mod_gzip for line compression?

    Xeon's arent' THAT cheap, but hey, 1ghz machines (or even 500mhz machines) with this card would easily match your Xeon once the 64MB/s cards come out. Or was that 64mb/s. Well, you get the point.

    As for the bus latency, well.. you are right, it'd be better in the network card, but remember, that's layer 1 and 2 stuff you'd be meddling with, where gzip would end up in layer 4. Layer 3 is tcp/udp, 4 is app data, right?

    --

    -
    ping -f 255.255.255.255 # if only

  5. Sun machines use PCI busses, too. by Vengeful+weenie · · Score: 3, Insightful
    A little late posting, but I did want to point out that modern Sun machines use PCI buses, and the Enterprise class [4000+] machines have a crap load of bandwidth through their backplanes.

    I think it's a little naive to say "Oh, my 1000 hit a day web box, running on a cheap 686 wouldn't benfit from this, so it must suck." Hey, dont get mad! You said it! :P

  6. Here is a thoguht! by f00zbll · · Score: 2, Insightful

    What if you run a website that gets say 5million+ page views a day and you generate around 2gigs of logs per day per machine across 8 machines. At night you setup an automated batch to zip the logs and ftp them to a log reporting server. Then a cron jobs kicks off log analysis of all 16gigs of logs. Wouldn't this hardware acceleration help? Now let's try to scale that up to 20million+ page views a day. Or what if you're Yahoo who gets 1billion page views a day. How many gigs of logs do you have to process now. Not everyone needs hardware acceleration, but I would hardley call it useless.

  7. Re:Why use Gzip? by NtG · · Score: 2, Insightful

    There are many many issues with this test, which has proved absolutely nothing:

    a. It appears (as someone mentioned elsewhere) that you are compressing an already compressed file

    b. You have not specified the options used when compressing, which can seriously alter the result

    c. You have thrown in TAR, which can be overlooked, however taring a single file before gzip compressing it is simply a waste of time unless there is some particularly pertinent permissions/directory structure data you want to preserve. Basically, you have inflated the gzip output by doing this

    d. Each of these compression methods has its own benifits and shortfalls. Good compression ratio is not the be-all and end-all. Certainly many people have explained the whole block-compression theory and why gzip is so versatile.

    e. You seem to be trying to prove here that RAR is a superior compression method. It is also not free. It certainly can't be used without licensing fees as gzip can.

    f. Where is output such as time taken, i/o and cpu demands, etc?

    You may want to rethink your research.