Researching the Slashdot Effect?

leonia asks: "At Columbia University, we are investigating methods and software to automatically deal with sudden, unpredictable load spikes ala the 'Slashdot effect'. We are looking for web sites that have been the 'victim' of this effect who would be willing to share anonymized web logs with us. If you know of a site that went under after it was mentioned on Slashdot or similar site or are the proud owner of such a site, we'd love to talk to you."

Comments

What? No link?

[RQuinn]

I guess we should all try and slashdot the mail server then.

Hah!

[orangesquid]

Smart author... he didn't include any links to Columbia University's pages! ;)

Re:Hah!

[rask22]

she said get in touch.. and the first word in the body of the post is an email address. /gasp!

Well,

[MattCohn.com]

isn't there a way to just block refers from high-trafic sites? Who cares if someone gets a 403 when clicking a link from slashdot/cnn/google news/neowin because they are only getting the message if your site would have been hit. Seems simple, and a list could be maintained. While this wouldn't protect you from anything not on the list, neither does antivirus protection.

Re:Well,

[jonknee]

Personally as a webmaster I try to GET traffic... not turn it away. I think I'd talk to my host about getting some backup servers before turning away tens of thousands of visitors. That being said, my site has never been slashdotted or been featured on a major non mac site.... In the event that it ever is, I'll be on the phone in a heart beat :-D (and then send the logs in for the study)

Agenda

[tedDancin]

Slashdot Research Dept. Agenda:

• Investigate Slashdot Effect (in progress)
• Interrogate the 40 Slashdotters we have captured in the 1st lab for continually posting "First Post".
• Cover the 40 "Beowulf Cluster" posting Slashdotters we have in the 2nd lab with mollases and feathers.

just post a link to Columbia U next time!

[lburdet]

title says it all?

naive Question?

[pamri]

How come Slashdot itself is never a victim of the 'slashdot effect', atleast on a regular basis as compared to the sites, it links to, considering the typical slashdotter does not read the article linked to.

Re:naive Question?

[MattCohn.com]

Same reason as CNN and such sites arn't when linked to. Multiple, dedicated servers and more bandwidth then they know what to do with. (So they made slashdot)

Re:naive Question?

[rcs2]

when's the last time you saw Slashdot personally serve a page full of casemod pics or a 12mb MPEG of sodium skipping across a lake?

Re:naive Question?

[SuiteSisterMary]

Slashdot gets slashdotted regularly. They just know how to handle it.

Ever notice how sometimes, the front page doesn't have your login information? Or the post count doesn't get updated? You're being served a static page for one reason or another.

The simple fact of the matter is that most of the 'database driven' sites, which is where you get the 'mysql socket error' or 'too many sql server connections' or whatever problems, don't need to be database driven. You don't write a static article, put it in a database, then hit that database every bloody time somebody wants to read it. Put a copy in a variable, display that, refresh the variable every five minutes. Or even every minute. Or even ever ten seconds. You'll still never get more than six connections a minute, at that rate, and the information is just as current and fresh.

Re:naive Question?

[Sparks23]

Slashdot, I believe, also has multiple servers and load balancing...something that not all servers do have. And a much faster link. And unfortunately, a lot of the article-servers are done that way for ease-of-management reasons. Have a single interface thing that generates your sidebar menus and all your 'current top stories' links and whatnot, and then just pull the article from the database. Also allows quick keyword searching through SQL.

Doesn't mean they couldn't cache those things ahead of time, but most sites don't think of that.

Luke Codehacker with a page on his home server about his nifty new software that ends up linked from Slashdot...ouch! Even if he isn't pulling from a database, watch the poor guy's DSL get obliterated!

That said, there are solutions to fix the Slashdotting problem. My own server hosts a site which was linked to from Slashdot about a month ago, and we saw the corresponding traffic spike. But it didn't kill the site or the link. Why?

I use all sorts of bandwidth-shaping tools and rules at both the kernel and Apache levels. Normally, the site in question is bandwidth-limited to 128kbit/sec of the overall link. When a whole huge spike of hits from the same referrer domain came in, the rules adjusted; it freed up another 128kbit/sec of the link for that site, and only put people from that referrer into that pool. As a result, normal site access wasn't impeded.

Now, if everything on the site was generated on-the-fly from a database, I probably would've had a lot more problems. :)

Forget Ellen Feiss

[0x0d0a]

It seems (see the first result) that getting married to CmdrTaco is a stunningly effective way to beat out every model in the world in terms of popularity.

We can help

[ajuda]

Everyone, CLICK THIS LINK. Let's give those ivy researchers some first hand data.

A simple way to research the Slashdot effect

[darkov]

(1) Setup one or more servers on the end of a fat pipe

(2) Submit an 'Ask Slashdot' asking people to visit a link to the above servers

(3) Wait

(4) Study logs

Re:A simple way to research the Slashdot effect

[cyberkreiger]

(5) ???

(6) PROFIT!!!

Looking for people who have been slashdotted?

[gnovos]

Allow me to introduce you to the Archives.

sign up to be slashdotted!

[tristan-b]

I've been wondering what would happen first if a page (preferrably with large movies) hosted from my computer happened to get slashdotted:

1- the server is reduced to a blob of melted plasic and silicon, or
2- CIS revokes my network connection for life.

Now all I need is a suitable article to submit.

Drupal's Throttle

[inerte]

There's a PHP script called drupal that has a "throttle" module. Jeremy, the owner of Kernel Trap, developed it after many /. stories with links to his pages.

It generates static files (similar to caches) when access is too high. You can check drupal's cvs (drupal -> modules -> throttle) or go straight to it.

Ps: Some links may contain whitespaces, cut, paste and edit...

Re:Drupal's Throttle

[Anonymous Coward]

The "throttle" module you refer to is a mechanism to detect surges in traffic, and to allow other modules to thus intelligently and automatically react. For example, the "statistics" module will automatically stop some of its logging and statistical analysis during heavy loads. In any case, this is new to Drupal, to be released with 4.1 (otherwise currently available in the development version from CVS). I hope in time other modules will utilize this logic. (I.E., themes could automatically become simpler under heavy loads, fancy features requiring lots of CPU could be temporarily disabled or reduced, etc) For more information read the help page.

I have written another module, called "filecache" which does what you have described. That is, it simply caches pages to the filesystem. If Slashdot results in MySQL choking, the site will continue to function thanks to the static cache files, automatically resuming dynamic generation after the rush is past and MySQL is able to respond to queries again. The module and patches to Drupal can be viewed here. I'm adding some more features to this module, but the core is done and seems to work quite nicely. (Additionally, with pages cached to the file-system I'm seeing a significant performance boost)

Drupal is an excellent engine, very modular in design and easy to work with. More, it's backed by a team of friendly and helpful developers. I made the switch after performance issues with PHP-Nuke, and I've not looked back. And it just keeps getting better...

-Jeremy

Step 4

[Hektor_Troy]

profit!

Re:Step 4

[Hard_Code]

Ahh Classic!

Sept 11, 2001

[erpbridge]

I would think that a lot of the major news sites like CNN and MSNBC, as well as Slashdot itself, would have some info as to how they held up on Sept 11, 2001. If anything was a Slashdot effect, that day was. (BTW, Slashdot seems to have been the site that best withstood the increased traffic that day... the other major ones crumbled.)

That said, people have written about the same thing, researching the Slashdot effect on their own sites, and posted logs here on Slashdot. I'm too lazy right now, but you may want to search the archives for some info.

Re:Sept 11, 2001

[greenhide]

To be fair, although slashdot has a huge readership, it still is known to only a segment of the Internet using population. CNN.com and MSNBC.com, on the other hand, are widely known brand names and advertised all the time by their respective networks. I'm guessing that slashdot would have gotten at most 1/10-1/20th of the traffic that CNN.com and MSNBC.com received.

Re:Sept 11, 2001

[Anonymous Coward]

And Slashdot did go down... (though it was impressively quick to recover)

Re:Sept 11, 2001

[winse]

CNN used to use akamai to handle spike loads. I'm not sure what they were using that day, but I'm told that their bill (it's an outsourced thing to them) was very large and that started investigating how to build their own content distrobution network. I don't work for them...I work in the caching industry though.

Petswarehouse.com

[rm -f DMCA]

http://www.petswarehouse.com got hit for a few days for sueing several customers:
Google sued as PetsWarehouse Lawsuit Continues.

Remember, it is a bad idea to type the following:
while true; do wget -O /dev/null http://www.petswarehouse.com/; done

and do not email pet@Petswarehouse.com

Two Stories down

[RedWolves2]

Right now the story Quiet Desk(not desktop)... is suffering from the slashdot effect.

I got one for ya

[oli_freyr]

As I write this the top story is this one.

The site receiving the poinding (www.thefreeworld.net) is down so you could check with them after the dust settles and find out what it was like... ;)

It can be hard....

[Anonymous Coward]

The first computer to access the web page to be slashdotted is logged. The next computer is logged. Then when the effect starts, the server says to the next computer, 'I can't, go see the computer that downloaded this page previously'.

Then a connection is established 'peer to peer' as it were.

Just make sure there are no sub-slashdot effects by spreading out the requests to various computers.

Simple procedure

[f97tosc]

1 Open any old Slashdot topic

2 Search for the string (Score:5, Informative)

3 If the post below contains a link, record where it leads

4 GOTO 1 for mor data points


Tor

Thank You, Thank You

[greenhide]

At Columbia University, we are investigating methods and software to automatically deal with sudden, unpredictable load spikes ala the 'Slashdot effect'.

Have you tried a...

Beowulf cluster?

*rimshot*

Two?

[anthony_dipierro]

Oh, that's right, I'm blocking timothy...

Google is your friend

[Cy Guy]

Try some Google searches similar to this one that will locate websites that have archived their hit statistics including the referring page information. I'm sure if you go through enough of them you will find a bunch that were seriously slashdotted during the time period those statisitics cover. Plus when you find them you will be able to put them in context of how many hits they are getting from sites other than slashdot, and if they have the stats for other months/weeks so you will be able to really put them in context of what level of traffic they normally get.

Note: make sure you check Google's archive of the pages for those that aren't static.

Can you see it?

[Otter]

Here's a traffic plot from denbeste.nu's recent clobering...

Comment removed

[account_deleted]

Comment removed based on user account deletion

some hardware/and or gameing sites sites

[VS1]

www.bluesnews.com
neowin.net
icrontic.com
overc lockersonline.com

hope this helps.

Easy data collection...

[jaybird144]

All that Columbia really needs to do is set up a site that talks about using Beowulf Clusters for profit...maybe add some kind of messageboard that lets everyone think that they've posted before anyone else...and wait for their own personal /. effect. They may want to warn their network technicians first...unless their study is actually on the effect /. has on the people who have to deal with the aftermath...

Slashdot effect Analyzed

[jukal]

cut-n-paste from the analysis:

This analysis is based on the data gathered after the story 'Donating Time To Goodwill Projects?' was published at the Slashdot.org site on 24th October 2002 around 13:30 GMT -05:00 (US East coast time). What I wanted to find out is: what is the slashdot effect really like, what are visitors like.