Slashdot Mirror

Since the cache servers in between the client and the server can't cache the content for multiple users.

Oh, you thought only browser caches mattered.

Consider the still excellent though ancient http://www.ircache.net/

Feel free to have a look at some of the research yourself. There are many other examples.

For the record, the file is set to expire in about 12 weeks from the first time you fetch it but as it supplies both Last-Modified and an ETag, there's no reason to fetch a new copy (it hasn't changed in 424 weeks).

Data available via cacheability checker.

The Cacheability Engine which runs this is open source (Freely Distributable, according to freshmeat), and can be found at http://www.mnot.net/cacheability/. There is another online demo available at http://www.ircache.net/cgi-bin/cacheability.py

Another approach to try and prevent this might be to get the browser not to send conditional GET requests *at all* and to just reload silently from cache.

This however would of course mean that everyone has to make sure their webpages are properly cache able with reasonable (perhaps dynamically generated) expiry dates.

The nature of HTTP and the web make it very difficult to remain totally untrackable all you can really do is prevent the worst of it.

1. Get behind a firewall/router

2. Configure your IE to use a public Squid proxy to download MS updates (I have my own Squid setup at home - and also a portable vmware Squid server - which also blocks ads and other nasty content.)

--Note: You will want to set this up in advance, since registration can take up to (4) hours; but in my case it was pretty quick.

http://www.ircache.net/FAQ/
( See section 2.2, and especially 2.5 )
http://www.ircache.net/ == Full frames version

--You have to send them your email, then it auto-registers you; it will be necessary to enter a username and password to authenticate yourself to the cache initially before you can browse. BTW, it doesn't handle secure connections; this is mentioned in the FAQ section 3.4. But it should be a good additional buffer against outside attacks.

1. Get behind a firewall/router

2. Configure your IE to use a public Squid proxy to download MS updates (I have my own Squid setup at home - and also a portable vmware Squid server - which also blocks ads and other nasty content.)

--Note: You will want to set this up in advance, since registration can take up to (4) hours; but in my case it was pretty quick.

http://www.ircache.net/FAQ/
( See section 2.2, and especially 2.5 )
http://www.ircache.net/ == Full frames version

--You have to send them your email, then it auto-registers you; it will be necessary to enter a username and password to authenticate yourself to the cache initially before you can browse. BTW, it doesn't handle secure connections; this is mentioned in the FAQ section 3.4. But it should be a good additional buffer against outside attacks.

I'd be interested in seeing what the average webpage size is from 1994-2004

According to this graph your guess that it levels of is correct.

Bittorrent reduces the load on the central server by having everyone who downloads content upload content to other users. Couldn't a similar system be designed for HTTP connections?

What a great idea! Something like a distributed network of caching web proxies. Quick! Patent this immediately!

Young whippersnappers just don't know any better. And you idiot moderators who rated it +5 interesting. squid faq or NLANR.

I don't buy that. If you have machines going down for whatever reason, maybe you should try to fix the problem one way or another. Setup throttling at the webserver, give the database server more RAM, get rid of the second server and use it as a cache for the first server, tell your users to use IRCache.net to take some of the load off the servers, just do something. Granted, new machines are nice and whatnot, it's just not cost effective.

While these are insightful points, these are not a problem because the OCN obeys HTTP caching semantics. Thus, just like your browser cache, it can deliver stale content within a certain amount of time before the OCN refreshes its cached copy.

To get a better picture of how HTTP caching semantics work, I recommend trying out Cacheability Engine and enter in a couple of sites to see how cacheable their content is.

Also, the OCN uses the SHA-1 hashes for all content addressing as soon as it translates the URL to the SHA-1 URN. There-after, the content is only referred to by its SHA-1 URN, so there is no concern about version conflicts between mirrors/peers, because a single SHA-1 URN can only ever point to a single version.

Your point re: dynamic data is a good one. The OCN really isn't designed for delivering dynamic content because it changes too frequently and the cached copies on the peers would quickly become stale. However, we are doing some work with caching RSS feeds, which provides a nice trade-off between dynamic and static content.

I looked around at sites like ircache.net, vancouver-webpages.com, and elsewhere looking for a way to get pages from caches besides of course hitting them from the side of the served network (i.e. with a browser or a spider like wget or wwwwoffle).

There is a hierarchical cache at U. of Melbourne for students there, so if anyone is reading this from a dorm there you might be able to spider the cache of the site to preserve it on your hard drive.

If anyone is familiar with caching protocol and how to query other caches on the net, why not share them here. Much of the data may be on the net. Likewise if anyone knows how much is replicated on other sites it will save people the trouble. I'm just worried that the contents of these caches may expire one day soon..

All you geek households need to go set up Squid and set it up to distribute the load. The Slashdot Effect can be eliminated within our lifetimes.

All you geek households need to go set up Squid and set it up to distribute the load. The Slashdot Effect can be eliminated within our lifetimes.

I'm the only one on the design team that works almost exclusively in CSS. The current design on the site wasn't made by me, the old one wasn't either.

Then I would suggest that it was a pretty poor example to use, or rely upon, to demonstrate a real-world challenge to Zeldman's case.

You can get a better idea of my CSS style by going to a css layout version of our current design or my not close to being done SVG site. Which has a very clean body.

It's unfortunate that the layout version isn't close to valid HTML or that the CSS isn't much better. The markup for the SVG site is a bit better, though the validator doesn't catch the odd inclusion of XHTML style BR elements in a document clearly marked as HTML 4.01 Transitional. I'll leave a check of its CSS to you, I'm getting timeouts on multiple requests to your server. You may want to investigate its cache status, too. But I assume both are incomplete projects, and everyone makes mistakes (though why they would post them to /. is beyond me).

The change was to pretty basic CSS and worked fine in all browsers. Although it still wasn't *exactly* what the old version looked like (text a couple points off here and there).

I suspect that the search for pixel- and proportion-perfect design is a root issue. Thinking in those terms when implementing on the Web is troubling and could even be labeled by some as naive or impossible beyond the narrowest of visitor communities. I begin to understand the excessive use of DIV elements in your markup.

[A]s I pointed out 3-4% is only 40megs. We pay $10 for 10 gigs of transfer and the account on the machine. So 40 megs is less than 5 cents. Hardly anything compared to the 400% difference of having gzip turned off.

Bandwidth is a resource, and like any resouce, it ought to be conserved where possible. Read my original post where I anticipate (and do not contest) a return to the issue of transfer compression. You don't have sufficient traffic to worry about cutting even small chunks from your costs, fine. Don't indict the practice for everyone else then ("I can assure you [bandwidth savings] are negligible.") or just send me the five cents you will sav every day, I'll put it to good use. :p

The point is this: you still seem to be glossing over my earlier observation that your ~4% savings could have been even greater had you actually understood the holistic approach Zeldman offers. It troubles me to think that you still don't "get it", even though you offer a couple new example sites that would seem to suggest that you do, in part.

If I were on dial-up (which thankfully, I'm not at the moment), I would certainly be appreciative as a user for the additional savings in time and bytes. Yes, there are still people on metered plans out there in the world. How much bandwidth a document or a site needs is not only a cost for the supplier, but also for the consumer, in money and time.

No one is contesting the utility of transfer compression. Stop complaining about the lack of gzip encoding, switch to a better host if it bothers you so much. [It's odd, your servers identify as "Apache/1.3.26 (Unix) PHP/4.2.2 mod_gzip/1.3.19.1a" -- mod_gzip is there but your host isn't permitting usage?]

Feel free to email me at monkeyman at oswd.org if you want to discuss it more.

Look, it may seem like I'm coming down on you like a ton of bricks. That's because I am.

It frustrates me to encounter Slashbots who insist on posting with the +1 bonus (even deep in a now off-topic thread), forcing an escalation to my own bonus. It frustrates me to read about how everyone should be beholden to one person's experience, when that person argues in the same thread that individual experiences don't matter much in the face of aggregate counts. And when that person needlessly crows about search engine placement. And especially when he attempts to push a public discussion off to email when someone takes the time to respond in kind.

Best of luck in your quest to regain mod_gzip.

Unfortunately there doesn't seem to be a copy of the Cease and Desist letter online, but you can see the patent here.

Funny, things are already changing though, if you look at this bake-off/cache-off page

Ot's over 10x faster than a Linux based caching proxy. In some cases its almost 100x faster. So I
don't think Novell is out of the picture yet.

Bullshit.

http://polygraph.ircache. net /Results/bakeoff-2/#Sect:7.3

Squid beat out the Novell box (the Dell)in most tests.

"IRCache Polygraph Homepage"

For web cache benchmarking they are the gold standard and everyone in the industry (even the much maligned IBM) shows up to the IRcache events and use Polygraph results in their ad copy.

If you do it right, open source benchmarks can become the standard in an industry...in fact, I think if you do it right, they will become the standard in an industry. Good companies want a level playing field. Bad companies will be weeded out eventually...

And that's all I have to say about that.

Where the hell is Novell's BorderManger in this little test?

Novell's BorderManager? Dunno.

Novell's Internet Caching System? See the Vendor Comments page on the bakeoff site, where somebody from Dell says:

The Novell Internet Caching System - Powered By Dell (Dell ICS 130B) used in the Second IRCache Web Cache Bake-off tests, is currently available from Dell Computer.

There were other boxes running it as well, e.g. at least some, perhaps all, of the IBM boxes.

(The "Vendor Comments" section seems to be filled primarily with "Vendor Advertisements"; yes, my employer, NetApp, proudly participated in the marketoonery in question.)

I was always wondering why slashdot is so heavily depending on dynamically generated pages. Even if just the main page had proper caching information, you could already hand off all the anonymous users (people not logged in) to the worldwide caching hierarchy out there.

See what the cacheability engine on ircache.net has to say about slashdot.org. At least the gifs are cacheable, but why is that "Expires"-Tag set? Do you expect the gifs to change every week?

Some comments in this thread mentioned that it is
difficult for Linux volunteers to find the expensive hardware required to conduct benchmarking tests. It might be a good idea to let
a few of these consulting/analysis agencies sponsor a "bake-off": a benchmarking contest between interested groups. They provide the hardware, the participants bring software and perhaps some of their own hardware and we have a head-to-head contest. Media attendance might keep matters above board.

This has been done recently (last month) for Web proxy servers. NLANR (the makers of the Squid web proxy) sponsored a bake-off. See this link for more information.

It would have to implement some restrictions (e.g., total budget) for fair comparisons, since companies can easily afford to bring their expensive hardware while academic/non-profit groups will have to skimp.

Yumpee