Slashdot Mirror
Surviving Slashdotting with a Small Server
S.BartFarst writes "Our little departmental server has been slashdotted twice in the last year and survived! Implementation of a two-headed redundant hardware scheme using linux virtual server and backup and failover capabilities enhanced by the linux high-availability tools has produced a nifty low-cost solution. Gotta love those little white boxes!
(also having a university-supplied BIG PIPE doesn't hurt). More interesting is the documentation of the apparent exponentially decaying attention span of slashdotters. Anybody else observed similar phenomena?"
Our little departmental server has been slashdotted twice in the last year and survived!
Wait... is this a challenge?
Mike
They are asking for another test.
What's in a sig?
Third time's a char... DOH!
Heh, well they're actually still kicking, oh well, it would've been so apropos.
Thou shall not survive thrice. You're insolence will not be tolerated. You'll servers will suffer a slashdotting not hence seen....
Notice this comment was posted on a slow Sunday afternoon (EST). Very clever, because they know that /.'ers can't resist a challenge like that. Feel sorry for them on Monday morning though...
I was under the impression that a 20k fiber or 100mbs one that can dynamically shift traffic would be needed.
http://saveie6.com/
well there you go... having a massive amount of bandwidth will allow you to survive a slashdotting. In most cases of slashdotting, I dont think the server was the bottleneck... its no problem for a server to dish out static pages... its the bandwidth, especially for serving pictures or videos....
Or is it where the article is at any given time? Top of front page gives lots of hits. As it drifts down, the hits slow as fewer read; to the sidebar, fewer but still substantial hits; then off to the specialty pages such as Science or Games, then only a few will read.
Of course, the only test would be to repost the article and see if there's the same number of hits... Nah, slashdot would never go for duplicate stories.
Interesting how it peaks, drops off slowly then rises again a little before dropping off again. Maybe some 'behind the curve' slashdot readers?
I've got some exponentially decaying pieces of chicken on my table...
and some exponentially growing forms of life in some beer cans...
does that count?
PC moderators can suck my White pierced, tattooed dick. If you think pride == hate, s/dick/Aryan meat mallet/g.
More interesting is the documentation of the apparent exponentially decaying attention span of slashdotters.
Well, I was gonna reply, but I forgot what the post was about.
I hope the dupe jokes that get posted are better than what I had here before I decided to just post this in hopes of drawing better jokes later in the thread. They weren't good! But when are they?
This is almost as stupid as wearing a mask of Donald Rumsfeld and visiting 3ID troops to tell them "Are we planning on letting you go home anytime soon? Gosh, no!"
These SMU guys are obviously begging for it, I say we give them hell!
[o]_O
Just under 40,000 hits in the busiest day... this is a slashdotting? Come back when you get into the millions. :)
Anybody else observed similar phenomena?
Nope. In our jobs they make us do work.
There is a total of 80K of information on the entire front page! Dig a little deeper guys and perhaps we can find a few gigantic image downloads. If you find any do share =)
From excellent karma to terible karma with a single +5 funny post...
While your setup may make you real safe from machine outages, the effects of a slashdotting are to flood your resources rather than break them. So your configuration gives you at best the performance of two machines instead of one - which you could also have achieved by just ramping up the CPU or memory.
[x] auto-moderate all posts by this user as insightful
Its incredible, this person has actually proven that LOAD BALANCING MULTIPLE SERVERS INCREASES YOUR LOAD CAPACITY! This is incredible news! Wow, I am sure glad it made it as an article, stunning.
Every medium to large website out there will be pleased to know that what they have been doing for the last 8 years is actually VALID, thanks guys!
I think the only reason this made it to the front page is the slashdot self-reference.
I'll bet if you chart the data hour-by-hour, you'll see a sudden dropoff at the very moment the story scrolls off.
It's also interesting that there was a second little bump about a week later. Anyone have any ideas why?
You didn't get Slashdotted if the server was still operating normally. You just had some people from Slashdot visit.
May we never see th
What I want to know is, how fat a pipe do you need to survive a slashdotting, given that your server structure is viable? Will a 10mbps pipe keep the barbarians from trampling the gate?
Lets help them out.
:; do wget http://www.geology.smu.edu/~dpa-www/venus/mpeg/atl a1.mpg -O /dev/null -o /dev/null ; done
while
Don't forget to fix the space in the URL.
Get your own free personal location tracker
My server has been slashdotted a few times and I can tell you it's pretty simple to not get overloaded.
The first time I learned my lesson. The server was on a T1 line that was 2/3 full already, and slashdot linked to a page full of large photos. That'll kill your link pretty quickly. Low-budget solution: sign up for a burstable web hosting account somewhere and just put all your large images there.
Later when we got some actual office space for the business, I moved the main server up to a colo facility in fremont. All slahdottable content is hosted there on a fast server with a 100mbps ethernet link. Other oddball services that need their own machine are hosted from the other end of a point-to-point T1 line going directly back to the office from the colo.
So depending on your budget it's really not hard to set up your site to survive a slashdotting. If you don't have a lot of dough to spend but you want to run your own server for configurability/security reasons, just host the static stuff somewhere else. Or if you're serving enough to make it economical, get a colo account with a burstable link.
There's a widespread misconception here that slashdotting is caused by server overload. In reality this is almost never the case. It's caused by insufficient bandwith. This in turn may cause server overload because of too many slow clients being connected, but that is purely a secondary effect.
BTW, using static pages also helps too. What is more, the "how to not survive" includes "generate content dynamically every time".
Also, there was an article on a hardware review site, if I remember correctly, where their approach to handling extreme load was discussed after their site was linked on Slashdot. Unfortunately, I can't find the article right now. Anyone around here who remembers?
They're just begging for a 'real' test... ... such as everyone downloading this:
:p
;)
ar405eng.exe (5.41 MB)
from their webserver
5.41MB per slashdot reader should provide a test worth of such a fat pipe
Are you serious? Because... this is what I get:
...blah,blah,blah...
/. effect time, too. I'm really proud of them, and their little beige boxes. =)
Pinging geology.heroy.smu.edu [129.119.223.84] with 32 bytes of data:
Reply from 129.119.223.84: bytes=32 time=58ms TTL=232
Reply from 129.119.223.84: bytes=32 time=61ms TTL=232
Reply from 129.119.223.84: bytes=32 time=60ms TTL=232
Reply from 129.119.223.84: bytes=32 time=56ms TTL=232
Reply from 129.119.223.84: bytes=32 time=58ms TTL=232
Reply from 129.119.223.84: bytes=32 time=74ms TTL=232
Reply from 129.119.223.84: bytes=32 time=67ms TTL=232
Ping statistics for 129.119.223.84:
Packets: Sent = 16, Received = 16, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 56ms, Maximum = 74ms, Average = 60ms
I mean, that's pretty damn smooth for a 30 minute old story. That's probably peak
Huh? What?
Hate me!
Er... no. Logarithms increase.
Exponential decay is of the form:
y = e^-a x
Where a is positive. It's exponential in that a fixed increase in x multiplies y by a fixed number; it's just that this number is less than one.
I think that's part of it. But I bet most of the effect goes to Karma whoring. Notice the second minor blip later on in their data.
When the article is new, the rush is on for insightful comments that deal with commenting on elements of the referenced links. (Might as well, there aren't any comments beyond ascii pictures, and troll expiditionary forces.) They have their responses, which then triggers the volley of RTFA's, and now there are a number of posts, people don't have to RTFA so much and the thread contains so much information anyway. But certain conversations develop, some tangential, but others still tightly following the information referenced in the links, and may even provide deeper links, which cause people to go back and reference the original works which provoked the server beating.
So one might look at there data and then form the yet to be tested hypothesis that the second blip is accurately representative of the slashdotters who are genuinly interested in any random subject at hand, and the difference between that and the peak could be correlated to the number of whores.
But that one isn't me, because A) Sunday B) I forgot to take my adderall.
Yeah, I noticed that too.
Brackets contain world's first nanosig, highly magnified:[.]
I really don't think the Slashdotter attention span is any different (or if different, it is longer) than the average Internet user.
When articles appear on the first page, they get attention, as they scroll to the bottom they get less, as they move to background pages they get significant;y less.
While I often look beyond the front page, I am less likely to delve into the articles or discussions there, since almost everything that needs to be said HAS been said by then.
I've carried on conversations with people regarding Slashdot articles long after the article appears. This can take place in journal entries or via e-mail where the discussion material can be easily kept as opposed to Slashdot comments which ultimately disappear anyway.
The fact that people don't continue to click on the original source URLs doesn't mean anything.
The ideal situation would be if you got a warning from slashdot and then then made some mirrors of the pages on distributer mirror.
Mouse powered Chips, Open source Processors and Lego
...a glutton for punishment!!! Either that, or they want to test it some more. Do your worst, slashdotters!
--
Vote for your hopes, not for your fears - Vote Third Party
Here are some mpeg files from their server: 3.8mb , 3.6mb and 320kb
Congratulations on surviving /.ing. I have a few questions.
How were LVS and HA configured? With two systems, I can only guess that each was a real server (using the LVS terminology). Also both would be load balancers, with one being selected as active using HA.
How did using HA or LVS help surivive a /.ing? Were there failovers? How many? When? Why? If surviving /.ing consisted of a high rate of failovers then the hardware wasn't up to the job.
What is the "automated backup system?" Are you rsyncing the contents? From each other? From another system? Or does it refer to regular "tar" backups to tape?
Having separate UPSs is overkill, unless the one UPS could not handle the load of both systems.
Is there any dynamic content on the servers? Databases? How was keeping these synchronized handled?
What I'd really like to see would be a graph of a BIG site when we Slashdot them now. It would be very interesting to see the subscribers and what they do before the /.ing public sees it. I couldn't seem to see one on the graph that they posted. Is it just that small? Just wondering.
Comment forecast: Bits of genius surrounded by a sea of mediocrity.
Given the sharp decline, this highlights another way that /. could help alleviate /.ing of sites: stagger the time that a certain client gets informed of a new article.
... 2 hours for full distribution is going to be friendlier to the /.ed sites, but 1 hour total would probably still be effective.
/.
1) RSS feeds would get the update -last- or in some form of randomness.
2) Anonymous (no cookie) clients get the same treatment
3) People logged in get the article sooner but are also stretched out. An example:
a) If your UID is in the 25% of the oldest active users you get the article as soon as it is published (after going out to subscribers, who always get it first, another very mild reason to subscribe especially if you like to FP)
b) If your UID is in the 26-50% of the oldest active users you get the article 30 minutes after it is published.
c) If your UID is in the 51%-75% you get it 1 hour after it is published
d) If your UID is in the last block you get it 90 minutes after publishing.
e) If you are pulling from RSS or anonymously you get it 120 minutes after publishing.
This also gives a little treat to the folks who have been around the longest while not removing the benfit of subscribing.
Another example could work like the above but randomly change which order each block of UIDs will get the article (with RSS and Anon getting it last) if you wanted to not show preferrence to older users.
Increments could be adjusted
The only people this would affect negatively are FPers, SPAMboarders and people who have a cow-orker walk by and go "hey d00d, seen that new article yet?". No one else would probably even be aware of it unless they find it from another site that found it on
It is more productive to voice thoughtful opinions (reply) than to judge (moderate) others.
Why is it surprising that it follows an exponential dropoff? The only interesting questions are the coefficients of exponential dropoff, not that it's exponential--I'd sit upright and take notice if it was a linear decrease.
Anything which follows a steady fractional diminishment will have a curve of y = ke**-ax, where k and a are constants. You see this basic equasion pop up all the time in physics, economics, statistics... etc. Why should server slashdotting be any different?
Should have squid cache running inside their network, so only one request for a given file should be necessary.
Not sure how you get that from the graph. For myself, I didn't know what the subject matter was, so I opened the window, went "ugh, geology", and closed it more or less straight away. Ok, perhaps this proves the point - for subjects I'm not interested in I have a short attention span, but this doesn't mean I have a SAS for everything.
/.ted, did the graph decay at the same rate or did it take longer to go down? If it took longer that would suggest shortening ASs, but then did you have anything of special interest up at the time? Bung some pr0n up there and see if the, er, bulge is a different shape.
You get an exponentially decaying number of hits, yes, but how many of those are people doing exactly what I did and not staying, as opposed to those who stay a while because they find geology interesting?
The last time you were
And I am on a very reliable 600 up 400 down ADSL connection.
:-)
Apparently not
I would expect such blatant racism on Fark, but on Slashdot? Mods please ban this asshole.
Taco:Yes, of course! The Holy Slashdot of OSDL! 'Tis one of the sacred relics Brother Cowboy Neal carries with him. Brother Neal! Bring up the Holy Slashdot!
AC's chanting: Pie Iesu domine, dona eis requiem.
Brother Neal: Armaments, chapter two, verse nine to twenty one.
Brother Neal: And Saint Cowboy Neal raised the Slashdot up high, saying, 'O Lord, bless this Thy Slashdot that, with it, Thou mayest slashdot Thine enemies to tiny bits in Thy mercy'. And the Lord did grin, and the AC's did feast upon first posts, trolls, GNAA posts, and...
Taco: Skip it a bit, Brother.
Brother Neal: And the Lord spake, saying, 'First shalt thou click on the holy link called Slashdot. Then, shalt thou count to three. No more. No less. Three shalt be the number thou shalt count, and the the number of the counting shall be three. Four shalt thou not count, nor neither count thou two, excepting that thou then proceed to three. Five is right out. Once the number three, being the third number, be reached, then, clickest thou holy Slashdot of OSDL towards thy server, who being naughty in My sight, shall snuff it.'
Taco: Amen
It is called Geek A.D.D.
"There is no teacher but the enemy."-Mazer Rackham
The guy is talking about bursts of 42,000 hits per day, and talking about it "bringing their system down". Now I could see that on Windows, but not on Linux.
Now, before you think I'm talking out of my posterior orifice, when my company was young and bright, we had a server built on a single 450 MHz Pentium 2, and 256 megs of RAM. It ran both Apache and PostgreSQL. Many of our pages were database-driven, which of course is a much larger load on the server than simple static pages.
That little machine would peak out at around 60,000 hits per day. At that point, it was slow enough to be self-limitting, but there was never any fear (or realization) of having the machine "brought down".
So, still "back in the day", I replaced it with a dual P3/650. That machine would peak out at around 100,000 hits/day (database driven!), without much problem at all. Also, as time goes on, and we develop new apps that make further use of our data, we tend to need more power to generate every page. Even still, we could crank out 40,000 hits per day on what would now be a relatively anemic server.
Now, with 7 front-end web servers and a dedicated DB machine, we crank out 5 million hits/day without problem. And even when our systems have been IMMENSELY overloaded from both legitimate and illegitimate traffic, the systems have still responded, and never once have I ever worried about a machine "going down" from the load. Failed hardware, perhaps, but not the load.
steve
Oh, you're not stuck, you're just unable to let go of the onion rings.
Well, it's 8:20pm EST and I'm not getting through from a T1.
Marxist evolution is just N generations away!
I remember the days when I would treasure any new content and what I read or saw had more of an impact on me. Now information is just catalogued in my head and I feel this strange need to gather more all the time.
...and don't even get me started on modern day movie trailers. There are so many cutscenes in trailers now that I literally have to close my eyes in the theatre to avoid having an epileptic seizure.
I think the attention span problem is more widespread than just us slash-heads. People are now being inundated with constant 'quick clips' and cut scenes for every television show and commercial
How can our brains avoid being desensitized with so much information being thrown at us all the time?
- Simon
Remember also that even though it's a fact that a huge amount of Slashdot users are interested in articles like this, still I'd imagine that a "normal" article with actual information attracts more readers, and therefore causes more traffic and server load. I could be wrong too, but I doubt it. ;)
They survived this before, just saying that judging their performance now by this article may not be correct. Subject does matter.
Seems like an easy way to get your setup tested
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Buy Steampunk Clothing Online!
my server survived a slashdotting just fine - IIS 5.0 / win2k sp3, 512 ram, single IDE HDD, P3-800mhz, etc.
The problem was, as it is for most people who get slashdotted, I didn't have a big enough pipe. Nothing to be done about that. I can't afford an OC3.
Natural != (nontoxic || beneficial)
1. Not enough bandwidth
2. Poorly configured web server
3. 300k images & 20MB mpeg/avi downloads (see #1)
4. Not enough RAM (1GB is generally enough)
I host about 50 domains for friends on my webserver (an Athlon 1.2Ghz w/ 1GB RAM) and have survived a simultaneous Slashdot and Fark link.
--Brent
Anti SCO T-Shirt
So if your site is slashdotted, churn out a static version of the pages which are likely to be pulled most and hand them over to a TUX server. Sit back and enjoy the traffic!