Slashdot Mirror

I read that when Google switched Gmail over to HTTPS that their server load increased by 1%. Today's CPUs are blazingly fast. Why would you think that the server load would be an issue with encryption and decrypting all communication? A web site is largely about having a large enough Internet connection and a large and fast enough database to keep up with the Internet traffic. Those CPUs are mostly just sitting around twiddling their thumbs waiting for I/O.

And what about their bandwidth usage?
There is a well-thought out caching system in HTTP -- implemented throughout the internet -- that saves a lot of bandwidth. You can kiss all this goodbye, and have every request come to your server. This also means akamai doesn't work, which many large sites rely on.

The real solution for facebook and others would be to make large, static content, such as images, stylesheets available on another, cache-able domain, and not send cookies on this domain. The dynamic content -- and javascript -- should stay on the main domain and be encrypted.

"Let's just encrypt everything all the time" is just a narrow-minded comment of an end-user.

Yeah, but using Mediawiki for read-only content is not the smartest thing to do, it prohibits caching.
If Mediawiki would allow proxies to cache their content (maybe there is a plugin, but Wikipedia doesn't allow it either), a lot of trouble would go away. And does Wikileaks need uncached requests? No.

I am not talking about (web) server-side improvements, I am talking about the problematic 'Cache-Control: private, s-maxage=0, max-age=0, must-revalidate' HTTP header. HTTP caching is so misunderstood. http://www.mnot.net/cache_docs/#WHY

The Cacheability Engine which runs this is open source (Freely Distributable, according to freshmeat), and can be found at http://www.mnot.net/cacheability/. There is another online demo available at http://www.ircache.net/cgi-bin/cacheability.py

Either install squid as a reverse cache, or use Apache's mod_cache. Unless Mambo is so broken as to make caching infeasible (some CMSes aren't exactly cache-friendly), this should take a load off the database.

Also, make sure you aren't screwing up public caches anyway. Things like serving CSS through PHP for compression etc aren't brilliant ideas, and you can get decent load reduction by setting expiry times in your HTTP headers. Mark Nottingham has written a good caching tutorial.

Er, just a thought, but you have actually profiled and made sure that it's the database, right? A lot of people will say stuff like that based on nothing more than guesswork. Measure before optimising, or you're just wasting your time.

I already pointed out several falsehoods in your claims about HTTP and FTP. I'm not going to bother going through your site to find more. Either you haven't done your homework (ignorance) or you are dishonestly representing competing protocols (malice).

Well, you pointed some out and I refuted some of the ones I thought you pointed out. Some of the ISP running ww stuff is what I think is a misdirection of where whitewater's strengths are. Yes, http caching works well at the corp or ISP level but I don't know if http caching is tuned towards keeping large iso's, tarred up software files, etc around for a number of days.

There are the components of the http and ftp protocols and there are the features of most common http and ftp servers and clients and then there are separately, the more exotic and less common software features and protocols for http caching. http caching is great
None of the features of the protocols or software cover the territory of bt, whitewater or konsire2b. In trying to describe his project, the author of ww may have made what you consider grevious injuries towards http and/or ftp by somewhat oversimplifying but the fact still is that neither the http protocol nor most http servers have a capability to "get the file from a closer source". In constrast, all whitewater software does auto-discovered-multi-source-simultaneous-distribu ted-crypto-integrity-checked file retrieval. Mis-statements can be made without ignorance or malice. Apparently, these "uber" capabilities of ftp and http will have to remain a mystery to me and presumably whitewater's author.

Blogdex and Daypop already are close to this, by keeping track of what the current popular links are. Every time a weblog links to something, it's a vote for that URL. That's the closest to democratization of the media if I've seen yet.

The only thing that needs to happen to match your view is personalization through a web of trust. Perhaps a person's FOAF defines who's opinions they value, and their RSS Aggregregators will rate stories accordingly. I think NewsMonster is working on something like this, but I don't know if it's implemented yet.

Regarding:

This document specifies HTTP extensions that bridge the current location-based Web with the Content-Addressable Web. -- HTTP Extensions for a Content-Addressable Web

The World Wide Web is "the universe of network-accessible information", i.e. anything with a URI, including URIs that are not tied to a particular hostname.

The Web already includes non-location-based URIs like mid: (for referring to message-ids), and urn:sha1: for referring to a specific set of bits by their checksum.

This proposal seems like a decent way of bridging HTTP-space with URN-space, but please remember that the Web is more than just HTTP. (see also: URIs, URLs, and URNs)

Anyway, it seems to me that sites that tend to suffer from slashdotting are:

1. those that use dynamically-generated pages for what is basically static content: this problem can be fixed by sites making sure their content is cacheable, and further deployment of HTTP caches. (I'm not convinced a p2p-style solution is the solution here.)

2. those with large bandwidth needs (kernel images, linux distribution .iso's, multimedia): as p2p software becomes more mature and widely deployed, everyone will have a urn:sha1: resolver on their desktop (pointing to their p2p software of choice), then whenever a new kernel is announced, the announcement can say:

   Linux kernel version 2.4.20 has been released. It is available from:
   
   Patch: ftp://ftp.kernel.org/pub/linux/kernel/v2.4/patch-2 .4.20.gz
   a.k.a. urn:sha1:OWXEOVAK2YJW3G6XSULXDWFCNWTX7B2K
   Full source: ftp://ftp.kernel.org/pub/linux/kernel/v2.4/linux-2 .4.20.tar.gz
   a.k.a. urn:sha1:PPWXYMA32YNDNO35UD3IQTCWBVBYK5DC

   and people can just fetch the files using urn:sha1 URIs instead of everyone hitting the same set of mirrors. (gtk-gnutella already supports searching on urn:sha1: URIs)

We need web caches... It's stupid to have files crossing the ocean thousands of times. Besides not using web caches causes that those who cannot afford bandwidth costs cannot put content in the web... Caches now!.

Web developers must not be afraid of web caches, since the HTTP/1.1 protocol allows them to precisely define how and when their content will be cached.

The mpeg file is gone... See? We need web caches... It's stupid to have this file crossing the ocean thousands of times. Besides not using web caches causes that those who cannot afford bandwidth costs cannot put content in the web... Caches now!.