Slashdot Mirror

← Back to Stories (view on slashdot.org)

Surviving Slashdotting with a Small Server

Posted by michael on from the soon-to-be-an-oreilly-book dept.

S.BartFarst writes "Our little departmental server has been slashdotted twice in the last year and survived! Implementation of a two-headed redundant hardware scheme using linux virtual server and backup and failover capabilities enhanced by the linux high-availability tools has produced a nifty low-cost solution. Gotta love those little white boxes! (also having a university-supplied BIG PIPE doesn't hurt). More interesting is the documentation of the apparent exponentially decaying attention span of slashdotters. Anybody else observed similar phenomena?"

33 of 307 comments (clear)

  1. Third Time's a Charm? by sweeney37 · · Score: 5, Funny

    Our little departmental server has been slashdotted twice in the last year and survived!

    Wait... is this a challenge?

    Mike

    1. Re:Third Time's a Charm? by BWJones · · Score: 5, Insightful

      Our little departmental server has been slashdotted twice in the last year and survived!

      Oh, come on. Even my little old G3 iMac is capable of handling quite a load from Slashdot and this site is serving up graphics intensive stuff. What you need to prevent a good Slashdotting is bandwidth that universities provide. T3 backbone connections are a wonderful thing. :-)

      Go ahead click all you want.

      --
      Visit Jonesblog and say hello.
    2. Re:Third Time's a Charm? by FCKGW · · Score: 5, Interesting

      Even though only a small percentage of Slashdot readers look at the comments, Slashdot's readership is so huge that the number of people reading the comments is still significant. It's not enough to kill a server, but I posted links to three images, around 80KB each, on my home server a few days ago fairly deep down in the discussion and got 3904 hits from it. It didn't kill my server (Pentium 133MHz, 64MB RAM, Debian 3.0, Apache 1.3.26, 3000/256 cable) and didn't result in any nasty letters from my ISP.

      OT: It was interesting reading the logs. There are quite a few Linux users on here (but even more Windows users), and I saw lots of people using Mozilla, Opera, Safari, etc. Compare that to sites aimed at the average user where 95% of visitors are using IE or AOL and don't know that there's anything better out there.

      --
      It's an operating system, not a religion.
    3. Re:Third Time's a Charm? by Syre · · Score: 4, Informative

      Static content is right.

      In 1998 I was working at a startup and we served Olympics news to Excite.com, which was one of the very largest sites on the web back then. Excite linked to pages on our server which were at some URL like olympics.excite.com that pointed to us.

      What they didn't know was that we were serving all this off of ONE Sun Ultra 1 workstation on my desk (which was all we could afford at the time). I had it set up with Squid so everything was coming out of memory.

      It worked fine, even at peak times everything popped up about as fast as any other page on Excite.

      Moral: if you need speed and the page doesn't really have to be dynamic make it static or cache it.

  2. Apparently... by yanbusa · · Score: 5, Funny

    They are asking for another test.

    --
    What's in a sig?
  3. Thou shall not by Anonymous Coward · · Score: 5, Funny

    Thou shall not survive thrice. You're insolence will not be tolerated. You'll servers will suffer a slashdotting not hence seen....

  4. Clever, clever by Frodo2002 · · Score: 4, Insightful

    Notice this comment was posted on a slow Sunday afternoon (EST). Very clever, because they know that /.'ers can't resist a challenge like that. Feel sorry for them on Monday morning though...

    1. Re:Clever, clever by ashitaka · · Score: 4, Funny

      Feel sorry for them on Monday morning though...

      Monday morning!!?? You're kidding, right?

      They have already noticed the "exponentially decaying attention span" of Slashdotters.

      By Monday morning this story and the site will be relegated to un-clicked graveyard of "Older Stuff"

      --
      If you don't want to repeat the past, stop living in it.
    2. Re:Clever, clever by Anonymous Coward · · Score: 4, Funny

      Commander Taco will repost it on Tuesday....

  5. Wouldn't you need an expensive switch? by Billly+Gates · · Score: 4, Interesting
    I find it hard to believe that a cheap dual channel switch for 2 pc's could handle the slashdot effect.

    I was under the impression that a 20k fiber or 100mbs one that can dynamically shift traffic would be needed.

    --
    http://saveie6.com/
    1. Re:Wouldn't you need an expensive switch? by Limburgher · · Score: 4, Interesting
      It's not the switch/throughput that's the issue. It's the amount of page generation being done by the servers themselves. The high throughput and individual links merely allows the maximum possible number of simulatneous hits from a bandwidth standpoint. The trick here is to do that, AND have your webserver survive, hence the dual box Linux Virtual Server setup. That's the piece that smells the /.ing and tries to load-balance the page generation. That ability should allow this setup to survive most /.ings.

      Very inexpensively, I might add. Beats having to buy the hardware for an, I dunno, . . .

      Beowulf cluster?

      --

      You are not the customer.

  6. well golly gee... by JeanBaptiste · · Score: 5, Insightful
    (also having a university-supplied BIG PIPE doesn't hurt).


    well there you go... having a massive amount of bandwidth will allow you to survive a slashdotting. In most cases of slashdotting, I dont think the server was the bottleneck... its no problem for a server to dish out static pages... its the bandwidth, especially for serving pictures or videos....
    1. Re:well golly gee... by Lumpy · · Score: 4, Informative

      nahhh... you can survuve a slashdotting much easier...

      simply use thttpd.

      it has throttling built in so that the slashdotting wont take down the server. It simply stops serving images at anything but a really slow rate and eliminates multiple requests from same hosts... (A bitch for NAT'ed companies)

      There are httpd servers out there that are much better than apache for handling insane loads.

      --
      Do not look at laser with remaining good eye.
    2. Re:well golly gee... by mo · · Score: 4, Insightful

      You're right. Bandwidth also serves a big purpose in finishing requests quickly. For example, let's see what happens when I have a 1.5mbps line with 512 concurrent requests. First of all, if you're using apache 1.3.X or 2.0 prefork, you've filled 2 gigs of ram by spawning 512 clients. Furthermore, you're bandwidth allocation per-client is 384 bytes/sec. This means you're spoon feeding your pages to your clients which makes it really hard for your server to get that 512 number down to something manageable.

      The problem here, is that the bandwidth bottleneck will make your server either (a) run out of processes/threads, (b) run out of ports/sockets, or (c) run out of memory from spawning all of the processes/threads to handle all of the stalled connections.

      Once this happens, people no longer can connect to you, and you're toast. The crazy thing here is that this can happen even at 10mbit/sec if you're machine is configured well enough, and if the content you're serving is large enough (IE: image/media serving).

      So cheers for these guys at keeping their bandwidth/server ratio high, I actually really like their architecture. But note that the greatest architecture in the world won't save you from a slashdotting if your server(s) are on a business dsl line.

    3. Re:well golly gee... by Jeremiah+Blatz · · Score: 4, Interesting
      JeanBaptiste writes:
      well there you go... having a massive amount of bandwidth will allow you to survive a slashdotting. In most cases of slashdotting, I dont think the server was the bottleneck... its no problem for a server to dish out static pages... its the bandwidth, especially for serving pictures or videos....

      With static web pages, server power is rarely a problem, it's all about the pipes. However, if the pages are dynamically generated, and don't have a lot of caching, then you've got yourself a big problem.

      So take, for example, loading a forum page in UltimateBB. AntiOnline handily tells you how many db requests it takes to create a page, and how long it took. This one over here says 61 requests and .3 seconds. Now, the poster claims to be peaking at ~37000 page views/hour, which is 10 hits per second. Now in that .3 seconds, where 61 database connections were established, there were another 3 requests coming in, making it an average of 240 database connections every .3 seconds. That's not an unreasonable number of connections, but what if your DB server can't keep up? What if, due to the load, the queries take 10x longer than usual? At that point, over .3 seconds, you get 240 connections, but only service 24 of them. Over the next .3 seconds, you get another 240 requests, but service only another 24, leaving you with 436 pending. After 30 seconds, you've serviced 2400 requests, but have another 21,600 pending. before too long, you're out of possible TCP ports.

      There are ways to keep your servers from crapping out under heavy load. One is to buy a studly, fire-breathing DB server that can process requests faster than your web servers can send them. Another (cheaper) solution would be to pool and marshall your DB requests, being sure to remove requests from the queue when the remote user times out (either by clicking the stop button or running up against a built-in limit of their browser). This way, your site may get slow, but nothing will crash. A final method is to use enough caching on the web server that you pages are, essentially, static. This is, for instance, what Vignette does, which is why all the major news sites use it. This method combines the flexibility of database-backed CMS systems with the database load of static web pages.

      So, essentially, there are many ways to let your database-backed web site survive a slashdoting, but embedding a bunch of PHP SQL queries against a locally-running installation of MySQL is not one of them. Unless you have a big honkin' cluster.

    4. Re:well golly gee... by seanadams.com · · Score: 5, Interesting

      It simply stops serving images at anything but a really slow rate

      What's the point? Either way you're slashdotted.

      Besides, I think in the case of server overload (as opposed to network overload), throttling will only exacerbate the problem by increasing the number of slow clients you have to deal with. This is the #1 bottleneck in web servers, the more clients you have, the longer it takes to deal with each one of them. Losts of processes to switch between, long arrays in an out of select(), etc.

      Also, when a user doesn't get a page in his browser, what does he do? He clicks the link again and again.... even more connections to handle.

      Really the only way to cure an overloaded server is to drop incoming SYNs. Any other measure is just pouring gasoline on the fire.

    5. Re:well golly gee... by evilviper · · Score: 4, Insightful

      We aren't talking about an actual error, that's just what it is called.

      What we are talking about is sending a different, low-bandwidth page, which will reload the main page after a certain ammount of time.

      Sure, you aren't immediately serving up the page to everyone who is requesting it, but people will only see a delay of a minute or so, rather than the server not serving up anyone, and crashing and burning for a day before anyone resets the thing. Or maybe just saturating the line, so nobody gets anything faster than 0.00001kbps, which is far far worse than having to leave the error page open in the background a couple minutes before it has a chance to load the main page.

      --
      Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
    6. Re:well golly gee... by Syre · · Score: 4, Informative

      Vignette is a rather trivial hack which was developed at C|Net and licensed to Vignette. It basically just writes out static pages, nothing fancy.

      As soon as Vignette licensed it, they were able to claim C|Net as a "Big Customer", which of course was a fib. This let them sell loads of software to organizations that didn't know any better.

      Moral of the story? Static pages are a great idea. If you can't do static easily, put a cache in front of your dynamic pages and decide how dynamic you really need them (Is 15 minutes delayed OK for you or is each page totally dynamic?). You don't need to spend $250K for Vignette to get this to work. In fact, you can do it with free software quite easily.

      Web caching and optimization is a big topic but in general the more clever you can be the less money you have to spend on big iron.

  7. Is it decaying attention span? by Eevee · · Score: 5, Insightful

    Or is it where the article is at any given time? Top of front page gives lots of hits. As it drifts down, the hits slow as fewer read; to the sidebar, fewer but still substantial hits; then off to the specialty pages such as Science or Games, then only a few will read.

    Of course, the only test would be to repost the article and see if there's the same number of hits... Nah, slashdot would never go for duplicate stories.

  8. A.D.D. crowd by Anonymous Coward · · Score: 5, Funny

    More interesting is the documentation of the apparent exponentially decaying attention span of slashdotters.

    Well, I was gonna reply, but I forgot what the post was about.

  9. Nope... by ryanvm · · Score: 5, Funny

    Anybody else observed similar phenomena?

    Nope. In our jobs they make us do work.

  10. You didn't get Slashdotted by 0x0d0a · · Score: 4, Insightful

    You didn't get Slashdotted if the server was still operating normally. You just had some people from Slashdot visit.

    --
    May we never see th
  11. The Slashdot worm. by caluml · · Score: 4, Funny

    Lets help them out.

    while :; do wget http://www.geology.smu.edu/~dpa-www/venus/mpeg/atl a1.mpg -O /dev/null -o /dev/null ; done

    Don't forget to fix the space in the URL.

    --
    Get your own free personal location tracker
    1. Re:The Slashdot worm. by Pharmboy · · Score: 4, Funny

      i already posted mine before i saw yours. my solution was

      while true
      do
      wget --delete-after -m -p http://www.geology.smu.edu/
      done

      which has the advantage of not only hammering their bandwidth, but since it requests everything on the entire site, including images and many multiple mb files, then the server cant cache it all, and will have to read from the disk alot. also, since it deletes after downloading, it doesn't take up your precious drive space, lol.

      but nice to know another sick fucker had the same basic idea.

      --
      Tequila: It's not just for breakfast anymore!
  12. Pretty simple... by seanadams.com · · Score: 5, Interesting

    My server has been slashdotted a few times and I can tell you it's pretty simple to not get overloaded.

    The first time I learned my lesson. The server was on a T1 line that was 2/3 full already, and slashdot linked to a page full of large photos. That'll kill your link pretty quickly. Low-budget solution: sign up for a burstable web hosting account somewhere and just put all your large images there.

    Later when we got some actual office space for the business, I moved the main server up to a colo facility in fremont. All slahdottable content is hosted there on a fast server with a 100mbps ethernet link. Other oddball services that need their own machine are hosted from the other end of a point-to-point T1 line going directly back to the office from the colo.

    So depending on your budget it's really not hard to set up your site to survive a slashdotting. If you don't have a lot of dough to spend but you want to run your own server for configurability/security reasons, just host the static stuff somewhere else. Or if you're serving enough to make it economical, get a colo account with a burstable link.

    There's a widespread misconception here that slashdotting is caused by server overload. In reality this is almost never the case. It's caused by insufficient bandwith. This in turn may cause server overload because of too many slow clients being connected, but that is purely a secondary effect.

  13. Testing, testing... by Mish · · Score: 5, Funny

    They're just begging for a 'real' test... ... such as everyone downloading this:

    ar405eng.exe (5.41 MB)

    from their webserver :p

    5.41MB per slashdot reader should provide a test worth of such a fat pipe ;)

  14. Re:Well its sorta dead by Latent+IT · · Score: 4, Informative

    Are you serious? Because... this is what I get:

    Pinging geology.heroy.smu.edu [129.119.223.84] with 32 bytes of data:

    Reply from 129.119.223.84: bytes=32 time=58ms TTL=232
    Reply from 129.119.223.84: bytes=32 time=61ms TTL=232
    Reply from 129.119.223.84: bytes=32 time=60ms TTL=232
    Reply from 129.119.223.84: bytes=32 time=56ms TTL=232
    Reply from 129.119.223.84: bytes=32 time=58ms TTL=232
    Reply from 129.119.223.84: bytes=32 time=74ms TTL=232
    Reply from 129.119.223.84: bytes=32 time=67ms TTL=232 ...blah,blah,blah...

    Ping statistics for 129.119.223.84:
    Packets: Sent = 16, Received = 16, Lost = 0 (0% loss),
    Approximate round trip times in milli-seconds:
    Minimum = 56ms, Maximum = 74ms, Average = 60ms

    I mean, that's pretty damn smooth for a 30 minute old story. That's probably peak /. effect time, too. I'm really proud of them, and their little beige boxes. =)

  15. Attention Span by cmacb · · Score: 4, Interesting

    I really don't think the Slashdotter attention span is any different (or if different, it is longer) than the average Internet user.

    When articles appear on the first page, they get attention, as they scroll to the bottom they get less, as they move to background pages they get significant;y less.

    While I often look beyond the front page, I am less likely to delve into the articles or discussions there, since almost everything that needs to be said HAS been said by then.

    I've carried on conversations with people regarding Slashdot articles long after the article appears. This can take place in journal entries or via e-mail where the discussion material can be easily kept as opposed to Slashdot comments which ultimately disappear anyway.

    The fact that people don't continue to click on the original source URLs doesn't mean anything.

  16. Here are the testing materials by krir · · Score: 5, Informative

    Here are some mpeg files from their server: 3.8mb , 3.6mb and 320kb

  17. More detail, please by embobo · · Score: 4, Interesting

    Congratulations on surviving /.ing. I have a few questions.

    How were LVS and HA configured? With two systems, I can only guess that each was a real server (using the LVS terminology). Also both would be load balancers, with one being selected as active using HA.

    How did using HA or LVS help surivive a /.ing? Were there failovers? How many? When? Why? If surviving /.ing consisted of a high rate of failovers then the hardware wasn't up to the job.

    What is the "automated backup system?" Are you rsyncing the contents? From each other? From another system? Or does it refer to regular "tar" backups to tape?

    Having separate UPSs is overkill, unless the one UPS could not handle the load of both systems.

    Is there any dynamic content on the servers? Databases? How was keeping these synchronized handled?

  18. Subscriber Slashdottings by MBCook · · Score: 4, Interesting

    What I'd really like to see would be a graph of a BIG site when we Slashdot them now. It would be very interesting to see the subscribers and what they do before the /.ing public sees it. I couldn't seem to see one on the graph that they posted. Is it just that small? Just wondering.

    --
    Comment forecast: Bits of genius surrounded by a sea of mediocrity.
  19. Exponential decay by rjh · · Score: 4, Insightful

    Why is it surprising that it follows an exponential dropoff? The only interesting questions are the coefficients of exponential dropoff, not that it's exponential--I'd sit upright and take notice if it was a linear decrease.

    Anything which follows a steady fractional diminishment will have a curve of y = ke**-ax, where k and a are constants. You see this basic equasion pop up all the time in physics, economics, statistics... etc. Why should server slashdotting be any different?

  20. exponentially decaying attention span? by jazman · · Score: 5, Funny

    Not sure how you get that from the graph. For myself, I didn't know what the subject matter was, so I opened the window, went "ugh, geology", and closed it more or less straight away. Ok, perhaps this proves the point - for subjects I'm not interested in I have a short attention span, but this doesn't mean I have a SAS for everything.

    You get an exponentially decaying number of hits, yes, but how many of those are people doing exactly what I did and not staying, as opposed to those who stay a while because they find geology interesting?

    The last time you were /.ted, did the graph decay at the same rate or did it take longer to go down? If it took longer that would suggest shortening ASs, but then did you have anything of special interest up at the time? Bung some pr0n up there and see if the, er, bulge is a different shape.