Webaccelerator with mod_gzip ?

Christian Bjerre writes: "Currently I'm using Squid as an webaccelerator to sit infront of apache and serve static files (.html .js .gif .png .jpg). Squid also serves my dynamic content which is .php files. My goal is to let Squid or another opensource program gzip the html, since I have Squid installed on another machine, where cputime is not an issue. I've been reading page up&down and Squid dosen't seem to support gzip encoding in Webaccelerator mode. I haven't tried feeding it gzipped html, which could be a solution, if there was a way of having Squid handle all content as binary. What do you guys & girls do to minimize bandwidth usage? // chris"

Comments

From Tanenbaum...

[cperciva]

Network bandwidth is increasing much faster than computational power. As time tends towards infinity, trying to "save bandwidth" at the expense of more complicated processing is dumb.

Re:From Tanenbaum...

[Account 10]

However, time isn't tending towards infinity very quickly and presently there are still good reasons to save bandwidth.

Re:From Tanenbaum...

[bartjan]

Does this formula hold when you compare the price of bandwidth with the price of computational power ??

I can get all the bandwidth I want (and a bit more) but I have to pay for it.
All the CPU's I use are overpowered, but I already have paid for the full capacity. This is not the case for my bandwidth, because I have to pay for every bit of data my webserver sends to the internet.

Re:From Tanenbaum...

[WasterDave]

Yeah, whatever but right now network bandwidth is fscking expensive. OTOH I'm not sure that an appreciable quantity of the bandwidth from a website is actually text.

FWIW these guys use php's ob_gzhander function and appear to get quite good results.

Dave

Re:From Tanenbaum...

[dietz]

Slashdot uses mod_gzip (watch your connections sometime if you don't believe me), so apparently they decided CPU time was cheaper than bandwidth, at least on very large scales.

That should be endorsment enough, right?

Re:From Tanenbaum...

[shario]

This might be true, but we're not near infinity yet - in this sense the increasing abundance of mobile web access devices has taken us back in time... :-)

Re:From Tanenbaum...

[Monkeyman334]

Well, I don't think you realize how much bandwidth is saved. My stats before gzip say 329,391 pages for 3.2 gigs. January (after gzip) was 285,289 pages, and 1.2 gigs. So I can serve out 4 times as many pages a month and still make our 5 gig quota.

Re:From Tanenbaum...

[Sloppy]

This is utterly contrary to everyday experience. I just can't believe it; it's so absurd.

Re:From Tanenbaum...

[Stinking Pig]

When is waste of resources ever a good idea? Someone pays for them eventually, so if a little forethought could conserve resources, it should be applied.

Use HTTP Properly

[Fweeky]

HTTP is designed to be cached; for example, the Vary: header lets the cache know what to cache based on; in this case it'd be Content-Encoding.

Squid may well be smart enough to know to cache documents with two different Content-Encoding's as different entities, but if not you just add Vary: Content-Encoding so it does.

(that, at least, is the theory, I'm afraid I can't claim to have read the entire HTTP 1.1 spec, tsk :)

Big picture

[Anonymous Coward]

Such reasoning, while technically valid, lacks consideration to real world demands for low cost (hence, less bandwidth).

I made some experiences with RabBit recently. Maybe you can't use it, but check it for ideas, as it aims your very needs, IMO.

Have you looked at apache?

[nathanh]

Apache+mod_proxy+mod_cache can also be an httpd accelerator (aka reverse proxy) and I've had great success with it. My original configuration used Squid but several limitations with Squid forced me to investigate the alternatives. I finally settled on Apache and I've been very happy with it. Apache also has better support for 301/302 rewriting (the PassProxyReverse directive) than Squid, and this was a deciding factor for me in dumping Squid and going with Apache.

Internet Explorer

[Molina the Bofh]

I run mod_gzip on my servers.

I was using tcpflow to monitor the connections, and I noticed something strange. It seems most IE versions (like 5, for instance) append the following string on the request connection:

Accept-Encoding: gzip, deflate

that tells the server it can send compressed data.
Strangely, I notice that if I access the very same site using IE 6 + patch Q312461 on XP, it DOES NOT send such string, thus all my requests return unencrypted.

I checked, and I am not using a proxy, and on advanced options, Use HTTP 1.1 is checked allright.

How can I enable compression ? It *SHOULD* be default. Has somebody else noticed this strange behavior ?

Re:Internet Explorer

[Anonymous Coward]

Strangely, I notice that if I access the very same site using IE 6 + patch Q312461 on XP, it DOES NOT send such string, thus all my requests return unencrypted.

I sure hope you're not relying on gzip for encryption.

Re:Internet Explorer

[gilgongo]

He must have meant "compression" - if not he's insane. No wonder nobody's replied to his post. Mind you, if he does mean compression, then it's an interesting observation he's making, if it can be proved.

Personally though, I think the whole HTTP compression thing is overblown - causes more problems than it solves sometimes, and if you're compressing to please modem users, I'd prefer to simply put a banner on my site saying "Burn your modem, you fewl".

G

Re:Internet Explorer

[Molina the Bofh]

Yes, I am not insane, I meant compression instead of encryption, and I am aware it'd be no better than ROT 13. I wrote that when I was sleepy and in a hurry.

And, mind you, modem users are not so minority. Not everyone lives in a place where 1 Mbps DSL is available for $50. I wonder I did. Here where I live a 256 Kbps DSL goes for about $50/mo. And don't forget overseas connections, wich are tipically slower, RTT times higher, and sometimes packet loss can happen, what, added to TCP low-start, can be a pain. So a file size reduction is a big plus.

Anyway, the fact remains. APPARENTLY IE 6.0 on XP does not request compressed connections. At least mine doesn't. And I don't see any reason for such.

Re:Internet Explorer

[JediTrainer]

Personally though, I think the whole HTTP compression thing is overblown - causes more problems than it solves sometimes, and if you're compressing to please modem users, I'd prefer to simply put a banner on my site saying "Burn your modem, you fewl".

I work as the lead programmer for a (somewhat) large e-Business application, and I can honestly say that gzip compression is definitely worth it.

In our case, the application is Java-based. It was a piece of cake to add dynamic gzip compression to the 'main' servlet which every other servlet in the application extends. For us, it was important to see how quickly a page can be sent to our customers. Even a small gain would improve things a lot.

The results? Well - after some testing, we figured out that for any pages greater than 8k in size, it's worth the extra time to compress with gzip. We've saved a bundle in bandwidth costs, and more importantly, customers are happier that they can download their (large - a couple hundred k) reports in a decent timeframe, for example.

I've had no trouble with it either. The only case where it had issues was when the content wasn't textual, such as dynamically-generated PDF documents. The browers didn't know what to do with this, so some care had to be taken to only use gzip on the html and text content. Naturally we check the accept-encoding in the http request headers to make sure it's ok.

Many companies who are our customers don't have T1s, after all. Quite a few have ISDN. The thing is, you can't get them to listen and spend more money on a pipe just so they can use the advanced (bandwidth-intensive) features of your site. Instead, you have to do what you can to accomodate them as much as possible.

The gzip compression actually made the site bearable even on a 28.8. Thus, customers who have faster lines benefit even more, since our own pipe is less clogged as a result too.

We can always throw an extra machine or two into our application server pool (relatively cheap for a simple 1U server). Adding extra bandwidth, though, is expensive and management has already forbidden it for the time being.

Re:Internet Explorer

[Anonymous Coward]

APPARENTLY IE 6.0 on XP does not request compressed connections.

Mine does, IE 6.0.2600, on Win2k, either it is XP or you are doing something wrong...

GET / HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/vnd.ms-powerpoint, application/vnd.ms-excel, application/msword, */*
Accept-Language: en-ca
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Q312461)
Connection: Keep-Alive

Re:Internet Explorer

[gilgongo]

OK you talked me into it. I'm compiling mod_gzip right now.

By the morning all 88 of my web servers will be running it. If I get problems, I'll send 'em to you ;-)

G

Re:Internet Explorer

[JediTrainer]

OK you talked me into it. I'm compiling mod_gzip right now.

I never said I'm running mod_gzip. I did say I'm using gzip to compress our output on the site. But it's using Java's build-in gzip packages (java.util.zip.GZIPOutputStream I think).

You shouldn't have any problems though - I'm using mod_gzip on another one of my servers and it hasn't given me any issues.

Re:Internet Explorer

[Joe U]

Hate to say this, but I think it's your system.

I just ran a test using IE 6.0.2600 Q313675; Q316059; on XP Pro 2600 against an IIS5 server.

It's requesting compression.

Edge Serving

[spike666]

anyone looked at edge server technology? its something ibm and various others came up with to basically move the content out towards the clients.

what i'm wondering is if that technology conecpt has any other compression novel ideas.

RabbIT proxy

[bisscuitt]

I have been using the RabbIT proxy for a while now..

It is really good for low bandwidth connections like Mobile Phone connections (such as 9600 and 14400bps)

It GZIP's both Dynamic and Static content from any source.

I've been using it with Mobile Internet Explorer on my iPAQ via my Mobile phone

It's worth looking into.

You could compress with php.

[nkg]

There is an article on php's output buffering (which includes compression)here
There are also a collection of third party class's which you can browse here.