Researching the Slashdot Effect?

leonia asks: "At Columbia University, we are investigating methods and software to automatically deal with sudden, unpredictable load spikes ala the 'Slashdot effect'. We are looking for web sites that have been the 'victim' of this effect who would be willing to share anonymized web logs with us. If you know of a site that went under after it was mentioned on Slashdot or similar site or are the proud owner of such a site, we'd love to talk to you."

Comments

What? No link?

[RQuinn]

I guess we should all try and slashdot the mail server then.

Hah!

[orangesquid]

Smart author... he didn't include any links to Columbia University's pages! ;)

Well,

[MattCohn.com]

isn't there a way to just block refers from high-trafic sites? Who cares if someone gets a 403 when clicking a link from slashdot/cnn/google news/neowin because they are only getting the message if your site would have been hit. Seems simple, and a list could be maintained. While this wouldn't protect you from anything not on the list, neither does antivirus protection.

Agenda

[tedDancin]

Slashdot Research Dept. Agenda:

• Investigate Slashdot Effect (in progress)
• Interrogate the 40 Slashdotters we have captured in the 1st lab for continually posting "First Post".
• Cover the 40 "Beowulf Cluster" posting Slashdotters we have in the 2nd lab with mollases and feathers.

naive Question?

[pamri]

How come Slashdot itself is never a victim of the 'slashdot effect', atleast on a regular basis as compared to the sites, it links to, considering the typical slashdotter does not read the article linked to.

Re:naive Question?

[MattCohn.com]

Same reason as CNN and such sites arn't when linked to. Multiple, dedicated servers and more bandwidth then they know what to do with. (So they made slashdot)

Re:naive Question?

[rcs2]

when's the last time you saw Slashdot personally serve a page full of casemod pics or a 12mb MPEG of sodium skipping across a lake?

Re:naive Question?

[SuiteSisterMary]

Slashdot gets slashdotted regularly. They just know how to handle it.

Ever notice how sometimes, the front page doesn't have your login information? Or the post count doesn't get updated? You're being served a static page for one reason or another.

The simple fact of the matter is that most of the 'database driven' sites, which is where you get the 'mysql socket error' or 'too many sql server connections' or whatever problems, don't need to be database driven. You don't write a static article, put it in a database, then hit that database every bloody time somebody wants to read it. Put a copy in a variable, display that, refresh the variable every five minutes. Or even every minute. Or even ever ten seconds. You'll still never get more than six connections a minute, at that rate, and the information is just as current and fresh.

We can help

[ajuda]

Everyone, CLICK THIS LINK. Let's give those ivy researchers some first hand data.

A simple way to research the Slashdot effect

[darkov]

(1) Setup one or more servers on the end of a fat pipe

(2) Submit an 'Ask Slashdot' asking people to visit a link to the above servers

(3) Wait

(4) Study logs

Looking for people who have been slashdotted?

[gnovos]

Allow me to introduce you to the Archives.

Drupal's Throttle

[inerte]

There's a PHP script called drupal that has a "throttle" module. Jeremy, the owner of Kernel Trap, developed it after many /. stories with links to his pages.

It generates static files (similar to caches) when access is too high. You can check drupal's cvs (drupal -> modules -> throttle) or go straight to it.

Ps: Some links may contain whitespaces, cut, paste and edit...

Re:Drupal's Throttle

[Anonymous Coward]

The "throttle" module you refer to is a mechanism to detect surges in traffic, and to allow other modules to thus intelligently and automatically react. For example, the "statistics" module will automatically stop some of its logging and statistical analysis during heavy loads. In any case, this is new to Drupal, to be released with 4.1 (otherwise currently available in the development version from CVS). I hope in time other modules will utilize this logic. (I.E., themes could automatically become simpler under heavy loads, fancy features requiring lots of CPU could be temporarily disabled or reduced, etc) For more information read the help page.

I have written another module, called "filecache" which does what you have described. That is, it simply caches pages to the filesystem. If Slashdot results in MySQL choking, the site will continue to function thanks to the static cache files, automatically resuming dynamic generation after the rush is past and MySQL is able to respond to queries again. The module and patches to Drupal can be viewed here. I'm adding some more features to this module, but the core is done and seems to work quite nicely. (Additionally, with pages cached to the file-system I'm seeing a significant performance boost)

Drupal is an excellent engine, very modular in design and easy to work with. More, it's backed by a team of friendly and helpful developers. I made the switch after performance issues with PHP-Nuke, and I've not looked back. And it just keeps getting better...

-Jeremy

Sept 11, 2001

[erpbridge]

I would think that a lot of the major news sites like CNN and MSNBC, as well as Slashdot itself, would have some info as to how they held up on Sept 11, 2001. If anything was a Slashdot effect, that day was. (BTW, Slashdot seems to have been the site that best withstood the increased traffic that day... the other major ones crumbled.)

That said, people have written about the same thing, researching the Slashdot effect on their own sites, and posted logs here on Slashdot. I'm too lazy right now, but you may want to search the archives for some info.

Google is your friend

[Cy Guy]

Try some Google searches similar to this one that will locate websites that have archived their hit statistics including the referring page information. I'm sure if you go through enough of them you will find a bunch that were seriously slashdotted during the time period those statisitics cover. Plus when you find them you will be able to put them in context of how many hits they are getting from sites other than slashdot, and if they have the stats for other months/weeks so you will be able to really put them in context of what level of traffic they normally get.

Note: make sure you check Google's archive of the pages for those that aren't static.

Can you see it?

[Otter]

Here's a traffic plot from denbeste.nu's recent clobering...