Handling the Loads

On Tuesday, something terrible happened. The effects rippled through the world. And Slashdot was hit with more traffic than ever before as people grabbed at any open line of communication. When many news sites collapsed under the load, we managed to keep stumbling along. Countless people have asked me questions about how Slashdot handled the gigantic load spike. I'm going to try to answer a few of these questions now. Keep reading if you're interested.

I woke up and it seemed like a normal day. Around 8:30 I got to the office and made a pot of coffee. I hopped on IRC, started rummaging through the submissions bin, and of course, began reading my mail. Within minutes someone told me on IRC what had happened just moments after the impact of the first plane. Just a minute or 2 later, submissions started streaming into the bin. And at 9:12 a.m. Eastern Time, I made the decision to cancel Slashdot's normal daily coverage of "News for Nerds, Stuff that Matters," and instead focus on something more important then anything we had ever covered.

I couldn't get to CNN, and MSBNC loaded only enough to show me my first picture of the tragedy. I posted whatever facts we had: these were coming from random links over the net, and from Howard Stern who syndicates live from NY, even to my town. Over the next hour I updated the story as events happened. I updated when the towers collapsed. And the number of comments exploded as readers expressed their outrage, sadness, and confusion following the tragedy.

Not surprisingly, the load on Slashdot began to swell dramatically. Normally at 9:30 a.m., Slashdot is serving 18-20 pages a second. By 10 we were up to 30 and spiking to 40. This is when we started having problems.

At this point Jamie and Pudge were online and we started trying to sort out what we could do. The database crashed and Jamie went into action bringing it back up. I called Krow: he's on Western time, but he knows the DB best, and I had to wake him up. But worst of all, I had to tell him what had happened in New York. It was one of the strangest things I've ever done: it still hadn't settled in. I had seen a few grainy photos but I don't have a TV in my office and hadn't yet seen any of the footage. After I hung up the phone I almost broke down. It was the first time, but not the last.

The DB problem was a known bug and the decision was made to switch to the backup box. This machine was a replicated mirror of Slashdot, but running a newer version of MySQL. We hadn't switched the live box simply because it meant taking the site down for a few minutes. Well we were down anyway, and the box was a complete replica of the live DB, so we quickly moved.

At this point the DB stopped being a bottleneck, and we started to notice new rate limits on the performance of the 6 web servers themselves. Recently we fixed a glitch with Apache::SizeLimit: Functionally, it kills httpd processes that use more then a certain amount of memory, but the size limit was to low and processes were dying after serving just a few requests. This was complicated by the fact that the first story quickly swelled to more than a thousand comments ... we've tuned our caching to Slashdot's normal traffic: 5000-6000 comments a day, with stories having 200-500 comments. And this was definitely not the normal story. Our cache simply wasn't ready to handle this.

Our httpd processes cache a lot of data: this reduces hits to the database and just generally makes everything better. We turned down the number of httpd processes (From 60 on each machine, to 40) and increased the RAM that each process could use up (From 30 to 40 and later 45 megs) We also turned off reverse hostname lookups which we use for geotargetting ads: The time required to do the rdns is fine under normal load, but under huge loads we need that extra second to keep up with the primary job: spitting out pages as fast as possible.

This was around noon or so. I was keeping a close eye on the DB and we noticed a few queries that were taking a little too long. Jamie went in and switched our search from our own internal search, to hitting Google: Search is a somewhat expensive call on our end right now, and this was necessary just to make sure that we could keep up. We were serving 40-50 pages/second ... twice our usual peak loads of around "Just" 25 pages a second. I drove the 10 minutes to get home so I could watch CNN and keep up better with what was happening.

We trimmed a few minor functions out temporarily just to reduce the number of updates going to frequently read tables. But it was just not enough: The database was now beginning to be overworked and page views were slowing down. The homepage was full of discussions that were 3-4x the average size. The solution was to drop a few boxes from generating dynamic pages to serving static ones.

Let me explain: most people (around 60-70%) view the same content. They read the homepage and the 15 or so stories on the homepage. And they never mess with thresholds and filters and logins. In fact, when we have technical problems, we serve static pages. They don't require any database load, and the apache processes use very little memory. So for the next few hours, we ran with 4 of our boxes serving dynamic pages, and 2 serving static. This meant that 60-70% of people would never notice, and the others would only be affected when they tried to save something ... and then they would only notice if they hit a static box, which would happen only one in 3 times. It's not the ideal solution, but at this point we were serving 60-70 pages a second: 3x our usual traffic, and twice what we designed the system for. We got a lot of good data and found a lot of bottlenecks, so next time something that causes our traffic to triple, we'll be much more prepared.

At the end of the day we had served nearly 3 million pages -- almost twice our previous record of 1.6M, and far more then our daily average of 1.4M. During the peak hours, average page serving time slowed by just 2 seconds per page ... and over 8000 comments were posted in about 12 hours, and 15,000 in 48 hours.

On Wed. we started to put additional web servers into the pool, but that ended up not being necessary. We stayed dynamic and had no real problems on all 6 boxes all day. We peaked at around 35-40 pages/second. We served about 2 million pages. Thursday traffic loads were high, but relatively normal.

Summary So here is what we learned from the experience.

• We have great readers. I had only one single flame emailed to me in 24 hours, and countless notes of thanks and appreciation. We were all frazzled over here and your words of encouragement meant so much. You'll never know.
• Slashteam kicks butt. Jamie, Pudge, Krow, Yazz, Cliff, Michael, Jamie, Timothy, CowboyNeal, you guys all rocked. From collecting links to monitoring servers, to fixing bits of code in real time. It was good seeing the team function together so well ... I can't begin to describe the strangess of seeing 2 seperate discussions in our channel: one about keeping servers working, and another about bombs, terrorists, and war. But through it all these guys each did their part.
• Slash is getting really excellent. With tweaks that we learned from this, I think that our setup will soon be able to handle a quarter million pages an hour. In other words, it should handle 3x Slashdot's usual load, without any additional hardware. And with a more monstrous database, who knows how far it could scale.
• Watch out for Apache::SizeLimit if you are doing Caching.
• Writing and reading to the same innodb MySQL tables can be done since it does row-level locking. But as load increases, it can start being less then desirable.
• A layer of proxy is desirable so we could send static requests to a box tuned for static pages. For a long time now we've known that this was important, but its a tricky task. But it is super necessary for us to increase the size of caches in order to ease DB load and speed up page generation time ... but along with that we need to make sure that pages that don't use those caches don't hog precious apache forks that have them. Currently only images are served seperately, but anonymous homepages, xml, rdf, and many other pages could easily be handled by a stripped down process.

What happened on Tuesday was a terrible tragedy. I'm not a very emotional person but I still keep getting choked up when I see some new heart breaking photo, or a new camera angle, learn some new bit of heart breaking information, or read about something wonderful that somebody has done. This whole thing has shook me like nothing I can remember. But I'm proud of everyone involved with Slashdot for working together to keep a line of communication open for a lot of people during a crisis. I'm not kidding myself by thinking that what we did is as important as participating in the rescue effort, but I think our contribution was still important. And thanks to the countless readers who have written me over the last few days to thank us for providing them with what, for many, was their only source of news during this whole thing. And thanks to the whole team who made it happen. I'm proud of all of you.

Comments

A request

[rosewood]

I know that a lot of shackers and other people on the net aren't christian or don't even beleive in God. Thats fine. Tomorrow (now today) you will hear a lot of people praying, asking you to pray, etc. This isn't the snickers comercial where they bring in a representative of every religion before the big game. It will feel weird. I feel that a week ago that if NBC was showing a service that someone would whine. Today, I ask ya just let it slide. When they say pray, interperate that as 'do what makes you feel comfortable. Please just be respectful like your mama would want you to be. But for today, just kinda chalk it up to all those people burned, crushed, flateneted, chocked, suffocated, etc. to death.

Thank you

Re:A request

[Nicolas MONNET]

"Come back to this when you lose someone dear to you"

If it happened, odds are it would be due to some religious fanatics.

Re:A request

[Happy Monkey]

They are "Christian" in the same way that those flying the suicide planes were "Muslim"; i.e., only in that they happen to use that term to describe themselves.

Here are some such people.

Re:A request

[ChickenMaster]

"I really believe that the pagans, and the abortionists, and the feminists, and the gays and the lesbians who are actively trying to make that an alternative lifestyle, the ACLU, People for the American Way -- all of them who have tried to secularize America -- I point the finger in their face and say, 'You helped this happen.' "

A statement like this is meant to strike fear in the hearts of those people he "fingered" or people who may associate with them. Isn't that the core definition of Terrorism...

Shame on you Falwell, for you are no better than those responsible for the thousands of deaths on Tuesday.

Re:A request

[SirKron]

It's ironic. All of these religions which these misguided fundamentalist-whacko "leaders" (such as Osama Bin Laden and Jerry Falwell) supposedly follow condemn the most the Bin Ladens and Falwells of the world, who use God's Name to mislead people, or cause people to commit terrible atrocities.

My beliefs are right with yours, however, we must know that bin Laden is not a Fundamentalist in the normal definition. He is not trying to run a society closer to the exact interpretation of the Muslim religion. Contraversially, he is funding the training of a new militant generation of soldiers who dispise any religion other than their grossly missinterpreted version.

Osama bin Laden must be dealt with my the US military, however, we can not become a martyr! As was said in the movie Gladiator, 'they must kill your name first.' Once his name has been reduced in popularity through his personal suffering, then we can truely erase his influence and destiny of USA disasters.

From a Navy reservist waiting to assist...
----------
# rm -rf /bin/laden

Re:A request

[Anonymous Coward]

(+1 Insightful) to the above. Give me a gun, show me Falwell, and the world shall be a better place...

Of course, even a hypocrite like Falwell is not in the same class as Bin Laden.

Re:A request

[Redline]

WTF?? How exactly is a request to pray a denial of most of america's ethnicity?

I agree. Where I am from (Houston, Texas) the population is 70 percent hispanic. Almost ALL of the latin population here is Christian. And they *certainly* pray.

Re:A request

[orangesquid]

AugstWest: Over the past few days, I've learned that no matter how important or interesting of a point you are making, people are bound to interpret it wrong, judge quickly, etc. Everybody's just anxious, nervous, and scared, for the most part; I think it's best not to worry about petty things like comment moderations :)

Re:A request

[Anonymous Coward]

Falwell has apparently backpedaled and apologized for some of his comments. I haven't read such myself, but I've been told that this is the case.

Regardless, I have always believed Jerry Falwell to be a massive hypocrite and his behavior since Tuesday only confirms that belief. I saw him on one of the networks on Tuesday night (I channel-surfed a lot that night, so I honestly don't remember which) essentially agreeing that the guilty parties need to be dealt with militarily.

Perhaps I'm naive, but I've always thought that true Christians believed that it's up to God to punish the wicked and that Christians should strive to find it in their hearts to forgive evil acts. I realize this is a nearly impossible goal for even the devout to achieve, but at least the prominent clergy shouldn't be espousing such blatantly violent rhetoric.

I've always admired Martin Luther King, Jr., who even after horrible violence such as church bombings killed innocent children, counciled peace and love as the only cure for violence. It may seem naive, but it's clear that violence begets violence.

Perhaps after the organizers of Tuesday's atrocities are captured and tried, the citizens of the US can find it in our hearts to extend kindness to people we currently view as enemies. After all, extremism in many middle-eastern countries is growing more because of wretched economic conditions than a fanatical belief that America is Satan.

Re:A request

[Theodrake]

This reminds why I told my wife I don't like organized religion. I told her "I doubt you'll ever hear a statement like: an extremist atheist bin-laden follower today killed himself and 100's of others". She just didn't get it. She doesn't see how the hatred of the so called christians like the Pat Robertson's of the world can take us down the same path as the Taliban and the Bin Laden's.

Remember it was the senior President Bush that said there is no place in America for atheists. I say, yes lets get the people behind the attacks, but don't destroy the very foundation of our democracy in the process.

Re:A request

[DunbarTheInept]

Normally people calling themselves athiest aren't so sompassionate towards other religions.

As an atheist who hangs out with a lot of other atheists, let me say that this is a strongly filtered perception. It is filtered by the fact that most of the time, an atheist isn't going to really say or do anything that would clue you in to the fact that he's an atheist. It's not a religion - it's not a belief. It's a lack thereof. So it rarely comes up in conversation unless in response to something someone has said that is highly insulting to an atheist (for example the 700 club quote in this thread). So you never realize someone is an atheist until they've been riled up by such a statement and are (justifiably) argumentative about it. This gives the false impression that atheists aren't compassionate toward other religions. It's not that we aren't compassionate. It's just that we don't tend to say much about it when we are in that mood. Consider today's prayers across the country, and the "moment of silence" that is typically used for personal prayers. Think about whichever crowd you were in at the time, or were watching on TV. During that moment of silence, if everyone in the crowd was an atheist - would you have even noticed the difference? Not really. It was a moment to respect the dead, not to start arguments.

I suspect you are committing the statistical error of counting the hits and ignoring the misses. My compassion for a religious person dissapears when he tells me I am incapable of being a good person because I'm an atheist. But until then, I view him as a person just like any other.

Over the years I've noticed that religions are so varied and complex that they don't *really* dictate what morals people believe in, like they claim to do. Rather, what happens is the other way around: that the morals a person holds dear determines which religion, and which subsect of that religion he migrates toward, and this is mostly subconsious. You can twist a lot of religious teachings into whatever you like by selectively picking which parts to take literally vs which parts to take as "metaphor". Thus when someone tells me he's a Christian, or he's a Muslim, or he's a Jew - I try as hard as I can to *not* draw any predjudiced conclusions about what that really means. I wait to observe his actions and his words first. Or at least I *try* to. I will admit to not always succeeding at being gracious and polite as I should be - especially if I've just recently heard some hateful rhetoric such as that of today's 700 club (which has been floating around the net all day).

As a child I was brought up as a Ba'hii (Not sure where the apostrophe is supposed to go there - it's one of those Arabic words that just isn't easy to spell in English letters.) They are a modern offshoot from Islam, in much the same way that Christianity started as an offshoot of Judaism. (But the Ba'hii offshoot is much more recent - from the 1800's). Anyway, one of the core tenets of the Ba'hii faith is that one must spend time studying the other religions of the world as well as one's own, because they all came from the same god, and all have useful things to say. Given enough time studying, one is supposed to realize that the relgions all teach the same message, and that the similarities between them must be the "pure" message from God and the differences are supposed to be the minor points that don't matter as much, perhaps are even manglings of the original message a bit. It's a nice idea, and Ba'hiis tend to be *extremely* tolerant of other religions because of this. It is common to mix up different words from different relgions in a 'reading'. (The televised event today from the National Cathedral, with the list of different leaders from different faiths speaking is very reminiscent of what I remember from Ba'hii services I used to go to.) Anyway, where am I going with this? I don't really know.

I guess just one day I started toying idly with the idea that maybe the reason the religions overlap so much in what they say isn't because they all come from the same god, but because they don't come from a god at all, and instead come from people's own hearts and desires. Perhaps their similarities merely reflect the similarities cultures have because we're all human, and at some basic level we all think similarly when confronted with the taks of explaining the unknown. Over the years that little seed of doubt grew until finally I started thinking it the more plausable explanation to the one offered by the Bah'ii faith. When I turned 15 (the age at which one can officially make the declaration to become a Bahii for real), that doubt was so strong that I didn't feel comfortable doing it, and so I didn't. And I never have since, and what was once a small doubt is now what I see as the most plausable explanation. I called myself "agnostic" and decried atheists until much later when I realized they don't actually *have* a belief like I thought they did. I was applying the atheist label to a strawman that doesn't often exist. That's when I started calling myself one too.

But enough rambling. On this day, we shouldn't argue. Comfort yourself with the things you hold dear. I often think of religion as just a crutch, a teddy bear to make people feel better. But today is the last day to be taking people's teddy bears away. They need them now more than ever.

I've been impressed overall

[WinDoze]

Not only with Slashdot (did that REALLY say 2-thousand-something comments on the front page?!?!), but with CNN, ABCNews, the NY Times, and just about every other major news source I can think of. Tuesday afternoon was tough. By Tuesday evening all these sites were responding as though I was the only connected user. The server power that must have been thrown at some of these sites is staggering.

Kudos to Slashdot and the Slashteam

[Whyte Wolf]

I've spent the last few days in something of a daze, waiting for the real ramifications of Tuesdays horror to sink in. Many of my collegues up here in Canada are not sure what to make of the events, and possible response, but we're sure it will be bad.

That said, in all my experiences on the net over the last couple of days, it was Slashdot I came back to for my info feed/dump. Who had their site up and running in the face of the massive demand? Slashdot.

CNN was there during the Gulf War. Slashdot was there for the start of this new era, and I'm sure will be there in the face of whatever is to come. You guys are just another indication of the strength the US can have in the face of adversisty.

Thank you.

Very interesting stuff.

[XretsiM]

As a part owner of an internet developer/consultant, one of the more interesting things about Tuesday's tragedy was watching how various sites responded to the incredible load demands placed on them. Even watching the situation from the outside, it was clear that clear heads at Slashdot were doing something remarkable behind the scenes. Thanks for the insite into what was actually going on. I'll be passing this on to our staff, many of who came to rely on Slashdot's coverage on Tuesday morning.

thank you very much.

[garcia]

It is not common for people to recieve thanks for the great service that they do for a community but I am going to go ahead and give you thanks for feeding us the information that I was not able to get through TV and the basically non-exitant other news-sites.

I am normally a critic of /. and the editors but this entire week I felt that they did an extraordinary job of keeping us informed. For once I am going to applaud you.

I got links to personal experiences on the tragedies, movies, images not seen on TV, and personal reflection on the entire ordeal by people that seem to have valid ideas (not the crap that you hear from most people about the attacks)

Thank you again /. for making sure we got the news we needed.

Way to go Slashdot

[Sentry21]

When I first heard about this, after being woken up to it, I checked CNN's homepage - which was down. I checked several other news sites, and the only working one was CTV News. Then I thought to check slashdot - lo and behold, it was the only other site I could get to. I posted in one of the discussions that ctvnews.ca was working, and by the time I had hit 'submit', it wasn't.

Kudos to the Slashdot team for having the only satisfactorally working news service on the net. Combined with the people that made their own websites and posted their own pictures, and the people that mirrored news reports they COULD get to, it was an amazing triumph of technology. It's just too bad that this great moment in Slashdot history had to come at such a horrid moment in world history.

--Dan

A request for future

[tzanger]

Could you please cache the stories in NESTED mode instead of threaded? When the site is being hammered I would imagine it is far better to have guys grab a single large, cached page than a smaller cached page and then have to try to have teh system survive thosands of clicks for more information.

I really do thank you guys for this site and your decision to carry the news. I have a new respect for the amount of bandwidth you throw around with impunity on a daily basis.

one final request: get search back online so I can get to the old stories! Google doesn't have them (even now!)

Thank you

[sulli]

I posted this elsewhere, but I'll say it again: this has been Slashdot's finest hour. The eyewitness accounts and individual stories have been so meaningful, and the readers have been great - almost zero harassment and trolling (a bit more in the last day) and very honest, heartfelt comments. Also great were the mirrors in the first day that many participants posted, to handle the excess load for the news sites; the many Red Cross donation links; the updates and corrections of the news; and more.

Slashdot itself did very, very well in my experience. I experienced far fewer delays and errors than on other sites. Thanks to everyone who worked so hard to keep it running. You've made a huge difference for thousands of people.

Well Said

[Wyatt Earp]

That was really well said.

During my life I've always taken "bow your head and pray" as "shut up and look serious".

And I thought Slashdot did a very good job this week. I woulda emailed Taco that, but I figured there was enough traffic over the Internet.

Really good job guys. Between Slashdot and Drudge I felt as informed as a guy can be.

Good job to /., but forgive CNN and MSNBC

[Jayde Stargunner]

First off kudos to Slashteam. You kept a valuable news source up and running while most people were too stunned to do anything other than watch, horrified, at the TV. Good work. You provided a valuable service to many people in this crisis.

Also, to those who are getting down on CNN and MSNBC... From what I've heard, those sites are already tuned--and regularly do--serve around 45 pages per second...even with loads of media.

Crashing them was likely no small feat, either. Likely every person with internet typed in the very familiar cnn.com or msnbc.com just on instinct. It probably didn't help MSNBC or CNN that the MSN and AOL/Netscape portals, respectivly, link to them directly.

I was actually pretty impressed with how they handled the load...it was a little slower than /.'s recovery, but it was rather impressive given the HUGE load they were experiencing. First, they stripped down the page content to low-bandwidth versions, then phased in their site. I'm not sure about CNN, but MSNBC added static mirrors to their pool, and got Akamai servers to serve all their media. By around noon, both sites were running their normal full-content versions, even though they were probably still getting hammered to high-heaven.

Personally, I give many thanks to all the techs for all the news sites who worked like mad to ensure that people were able to understand what was happening. It must not have been easy to work in conditions like that: especially considering the stress that was put on them.

-Jayde

Of Slash and Slashdot

[ajs]

When I started reading this, I was disgusted. I was expecting something like CNN's ads after the Gulf war, touting the fact that they were the ones who got most of the scoops.

By the time I got half-way through the actuall content (not the front-page piece) I was in awe of how much went on. Usually when a massive load spike happens on my watch, I try to get everyone's fingers out of the pie so that we have a good chance of the machines just doing their jobs. The fact that these folks were able to make emergency changes in real-time to compensate for the load is just astounding.

CNN should be rolling out a Slash-based discussion forum for top stories. Heck, so should Whitehouse.gov!

Thanks guys, and good luck with your ongoing coverage of News For Nerds, Stuff That Matters!

I used slashdot to gauge public opinion

[uchian]

I first heard what was going on from Slashdot, and I had to turn the television on to believe it - it sounded too much for a prank story when I first read it.

For me, the television was more important than Slashdot for recieving information on what was going as and when it happened.

But for me, Slashdot has been much more important as a place where I could see what other people from all over the world were thinking about this tradegy. I hope that the different pesrpectives and posts which I have read have allowed me to more maturely handle how I feel about the situation than I otherwise would have been able to.

/.ers: Don't get too cocky...

[Alomex]

While my heartfell thanks go to /. for keeping this site up; others who are dissing the major news organizations must keep in mind that while
Slashdot was serving 50 pages per second, CNN was peaking at about an estimated 50,000 hits per second.

In light of this it was amazing that CNN was up at all, slow as it was.

Re:/.ers: Don't get too cocky...

[hattig]

And CNN.com is served from a single PII 533MHz server! Totally amazing.

Honestly, CNN's website will be composed of more than 10 times the servers that house Slashdot, possibly 100 times. The web server software will be serving static content, which is a lot easier than serving non-static content, even if the static content is larger (images, video). So 100x the servers plus 1/10th the work of Slashdot sounds fair to me...

Re:/.ers: Don't get too cocky...

[Datafage]

That says nothing about the quality of code/administration, CNN has incredibly more money than /. for boxes and bandwidth.

A solution for the links bogging down

[barnaclebarnes]

We were discussing how well /. stayed up just last night and how other news servers seemed to melt. thanks.n you guys gave us people with no tv a valuable news source.

What also impressed me were the people who put up pics/videos/news stories on their own servers to help people get news, even if they only had a dsl connection. Of course these sites soon got /.'ed as well...

So that led me to a new feature idea for news sites like this:

- People 'donate' a section of their web site to be a mirror for overloaded news stories.
- Whenever a link is /.'ed these sites replicate the site and store the data on their server.
- Slashdot keeps track of what sites have replicated and changes the url each time it serves a page with that link in it. That way the orginal site is now spread across 100's of dsl connections instead of one.
- After a set time (say a week?) the mirrors then delete the site from there servers and deregister their site from the mirrors list.
- Of course all this could be scripted with no input from users. All the /. admin would need to do is add some form of switch to say 'mirror this link' and the process would be put in place to start the morroring process.

And then you have your own distributed news network that handles major news stories with out getting slashdotted as much.

\well it sounds like a good idea...any comments?\

My only gripe..

[shayne321]

I want to thank the slash team as well for doing a great job of keeping slashdot up and running! It was my primary news source all day long when I wasn't able to get news from the "major" media sites.

My only gripe is I think it was very out of place and a bit insensitive that right in the middle of this (around 12pm if IIRC) Jon Katz took this tragedy as an opportunity to post some rant about how technology led us to this evil situation we were in and how technology was changing the way people get news or some such. I'm normally not a Katz-basher, but I think this was WAAAY out of place and insensitive to the people that died that day. Not only that, but it was unnecessary noise while people were still scrambling to get to the FACTS of what was going on. We really didn't need some insensitive wanna-be journalist's opinion on technology, of all things, in the middle of all of this. Maybe it would have been more appropriate on Wednesday or Thursday, but (to me) it was out of line at 12pm on Tuesday. Not to mention the whole crux of his article was off base (people killed people Tuesday, not technology).

Okay, I'll stop bitching now. Thanks again Slashdot, for stepping up to the plate and knocking one out of the park!

Shayne

Some say the Internet failed, I disagree.

[NetJunkie]

I've read a few reports about how the Internet failed during this disaster since almost all news sites were too busy to respond. I disagree with that. Slashdot was here, as well as things like IRC.

On the channel I've frequented for years I got more up to the minute information than anyone in the office. Everyone was wondering where my news was coming from, especially since it was so accurate. While some people were sitting around watching CNN we were discussing and talking about what was going on with people very close (too close) to the events.

This doesn't even take in to consideration email. With cell phones and land lines too congested people were sending emails back and forth to get word on loved ones or just to talk about the events.

I think the Internet did a great job.

Thanks, Slashdot Crew

[FFFish]

While the television remained my primary news feed, Slashdot was my primary web feed. It provided the community side of the equation: a finger on the pulse of the world and, particularly, America.

Thanks to the Slashdot crew for scrambling to provide the best possible service during a time when many other people were in emotional and occupational shutdown.

And thank-you to the people who form this community. On the whole, the discussions have been remarkably insightful and rational.

I'm hopeful that this web community is representative of the American population, and that we will see your political and military leaders taking sane action. This tragedy could all too easily throw us into devastating war with continuing long-term consequences.

I'll also take this opportunity to apologise for the several postings where I lost my head. While most of what I've written has attempted to educate a broadly ill-informed public as to why this attack took place, and to preach sanity in dealing with the attack, I have also lost my head in responding to some of the more dreadfully ignorant folk. For that, I am sorry: I should have been more patient and tolerant.

In closing, I'd like to assure our American friends that this has been a global tragedy. The outpouring of support, and demonstrations of grief and sorrow, have encircled the globe. Every nation mourns with you, and every nation feels a sense of shock and loss.

You are not alone.

Re:No need to reverselookup hostnames for geotarge

[Anonymous Coward]

I'm an AC who works for a large, well-known player in the computer industry. Your comments about what you all had to do to keep Slashdot going will help us and our customers to do a better job the next time, God forbid, that we will need this kind of capability.

I realize that you need the ad revenue to pay your bills. But, I do feel that when there is an emergency situation like this, perhaps it's better just to simply turn off the ads, or use a pre-selected static, omni-geographic, generic ad,
and explain to your customers later. If they don't understand, then they don't deserve our attention and support. I'm sure that all of the major news outlets in North America greatly reduced or eliminated commercials during their coverage.

Thanks for working so hard, and working so hard under such trying circumstances (FWIW, I agree with the comment about 25% productivity, BTW).

I think that after the weekend, when everyone has had time to sit at home with their families and friends to discuss Where We Go From Here, there will be a renewed vigor in the USA, and among peace-loving democracies around the world, and we will get the economy back on its feet and proceed to kick terrorist ass big-time. The outpouring of support from Canada, Europe, Australia, New Zealand, Russia, and other countries, both here on Slashdot and elsewhere, is greatly appreciated by this Yank. Thank You !!

Light Mode

[rho]

In the future, you might consider making the "HTML Light" mode the default mode under heavy load.

Granted, it doesn't alleviate the DB problem, but it does limit the images sent down the pipe.

(more ideas pulled out of the ass) Perhaps another Apache instance or a Perl script (horrors!) to watch traffic and to ratchet the options down as traffic increases, based on a weighted system (level 1: no sigs, level 2: drop journals, level 3: no search, ... level N serve only static HTML)

This is an interesting problem, and I'm impressed with y'all ability to handle it.

Automatic static page failover?

[Ed Avis]

Couldn't the switch to static pages happen _automatically_ if the database goes down? The only difference to most users would be inability to post comments.

Hmm, that is actually quite a problem (though still better than just having the site go down). Maybe a 'comment spool' where the comments can be saved as flat files, ready to be inserted when the DBMS comes back up?

Anyway, kudos to Taco and the gang for keeping Slashdot up. Three million pages in 24 hours... how does that compare with the really big sites like Yahoo, AOL and CNN?

Thank you slashdot

[o'bryon]

I live in Manhattan on the Lower East Side. I watched the the events of Sept 11th from my roof top in stunned disbelief. My disbelief turned to tears when the second tower fell. Throughout that day and the days that have followed, I've turned to the TV and the internet for any information that I can get, but my patience for what is passing for news on TV is wearing thin. I almost feel as though I can't trust what they say on TV anymore given
that last night the reports were claiming that they had arrested 10
people at JFK and LGA with compelling evidence that they were potential
hijackers. This morning when I got up story had changed and now it turns
out that they hadn't arrested anybody nor was there any real evidence.
There are similar stories are floating around regarding people rescued
from the rubble. As far as I know, every story that I've heard so far
about survivors being dug out, has turned out to be false. Every
channel has a different number of survivors and sometimes they are
policemen and sometimes they are firemen. And then it turns out that
they were two rescue workers
who had been trapped in the course of the rescue efforts! Can someone
please confirm or deny if they have pulled anyone out who was trapped at
the time that the towers fell?

The part of it that makes me really mad is that I haven't heard very
much in the way of apologies for these mistakes, nor have I heard any of
the reporters comment on the sheer volume of misinformation that is
being reported as fact. I understand that the reporters are human and
there is a great deal of competition that drives them to report
information before they can do a throrough fact check, in order to scoop
their rivals. But they've begun to lose their credibility and are
unconsciously furthering the terrorists' agenda of fear and uncertainty.

My apologies for the rant. I guess I'm feeling the fear and anger that
is inescapable right now...

Thanks to the slash team for keeping slashdot up and running and thank you to all of the slashdot readers, who've provided both information and commentary that is sorely lacking on all of the major television networks.

eirik

Pray Or Meditate Or Whatever For President Bush

[cybrpnk]

Folks, I've got a very very bad feeling and if it's true then the worst is yet to come and President Bush is going to need all the support he can get. The other day when he got off the phone from the mayor and governor of New York (neither of which I can spell), President Bush started speaking off the cuff (undoubtably to the horror of his PR people) and after rambling a little he said, "...I'm a likable guy....but I've got a job to do...and I'm going to do it..." and he said this with tears in his eyes. Several people in my office including me think they've decided to use a nuke and Bush is getting shook up about how HE is the one who is going to go down in history for authorizing it. This is a terrible burden for him, no matter what. He deserves your thoughts and support....

Thanks

[TeachingMachines]

CmdrTaco,

Thanks for the site. Not only was it the only site that was up, but better coverage was provided. I think we're all a little traumatized, but if it wasn't for Slashdot, I don't know what I'd be doing right now. I have to admit, I was really proud of you guys for working so hard on this story. I think that we all are. Thanks again.

Steve

Re:The irony

[edunbar93]

Well, it's in no small part because religion isn't about crusades and blowing stuff up. That's not the way it's supposed to be.

"Supposed to be" and "are" are two different things of course. Religion sometimes becomes a justification for slaughter, but it's never really the root cause, just an excuse.

Time for some highly unpopular opinion...

[tzanger]

Jerry Falwell and Pat Robertson blaming the events on liberals, feminists, etc. etc. etc.

While I wholly condemn the actions of the terrorists I do have to critically ask, "Did the government of the United States of America have this coming?

You'd have to be blind to see that the U.S. government has been supplying arms and training and money to factions around the world for over 50 years. You'd have to be blind to see the American government change its mind mid-stride -- first by supporting a group (again, with weapons and money), then by turning face, cutting off support or even condemning the actions of the group they supported.

You'd have to be insane to believe the 1973 crap propaganda article by Gordon Sinclair is a clear and frank view of the United States of America and its leaders and their policies.

The government of the United States of America has been bullying and harassing nations for a very long time, flaunting themselves as a superpower which is untouchable. They've stuck their noses in other nations' business too many times and someone had decided to cut it off.

I don't agree entirely with this Guardian article but it does rise a very strong and important point: The U.S. must change the way it carries itself in foreign affairs. The American people must stand up and take active interest in their nation's government. The American media must stop downplaying foreign affairs.

an aside: the Canadian people aren't much (any) better in this regard. Canadian readers: How much interest do you show in your government??

I do not believe that this is the act of one nation, or even of a nation. And I am frightened because I do not think this is the last.

The U.S. government and media is running around crying "Why me? Why us?" and you have the President standing frail and shaken, telling his nation that "He's gotta do what he's gotta do" instead of analyzing the situation properly and keeping cool.

I must give Bush credit -- he did not spout off about Arabs or "them guys" as Clinton did with OKC -- Bush remained calm and rational. I fear that this is quickly fizzling out because his anger is taking over and as President, he is not allowed to have those emotions. He is a man with the power of a very large, wealthy and military nation. He is not allowed to be angry. I think he is grappling with those emotions and his reserve is failing.

As a Canadian, I demand retribution for what happened in the United States this week. I am not saying "forgive and forget." Blood will be shed, and rightly so. Check out my /. userpage for views on what I personally feel is acceptable for retaliation. I also think the President should send a strong message that it is not acceptable to hate the middle eastern people -- Just as there was no witchhunt against all white people with OKC, there should be no anger towards the Arab, Muslim and other middle-eastern people within or outside the U.S. This is not an attack by the middle eastern people nor their religion; this is an attack by terrorists and cowards too cowardly to stand up and fight.

And I fear that we will be brought into a world war because of it.

Re:Time for some highly unpopular opinion...

[NMerriam]

So you're telling me that you're damned if you do and you're damned if you don't. I'm asking you about gradients between the extremes. Somehow I don't think that people would be as pissed about helping civilians as they would about the one-sided arming, financial and technical backing of one side of a faction, do you?

So you're suggesting a repeat of Somalia -- where we sent food with troops to protect it, but stayed out of the actual conflict? We simply went to provide support for civilians. The thanks we got was a lot of dead US soldiers. It seems combatants don't like for others to help civilians when THEY want to control the civilians.

So now we're back to the question of helping civilians and having both sides hate us, picking a side and having the other side hate us, or staying out and having the rest of the world hate us for "doing nothing".

Perhaps it is because the United States government makes their nation out to be the end-all, be-all of nations. Capable of anything, impenetrable and wealthy beyond all comparison.

Maybe it goes both ways. We certainly aren't lacking for suitors (to butcher my "prettiest girl at the party" metaphor). Maybe we're just a plain-looking girl but everyone else is desperate? Or wearing beer goggles? :)

But the track record your government shows indicates that they tend to be more interested in serving their own interests than genuinely helping others. That, as a foreign policy, will get you into trouble. Do you not agree?

I'm not sure -- will we be doing people a favor to be driven by 100% benevolent intentions? These nations were all very capable of playing the US off of the USSR, getting money and goods from both while avoiding any real commitments for 50 years. They are no strangers to capitalism or self-interest. Now that only the US is left, they seem shocked that we're not willing to keep sending them money just for being nice people.

We HAVE had mutually beneficial relationships with small nations who are willing to genuinely "grow" -- look at Turkey, for example, which is STILL shunned by the EU but has consistently been a close friend of the US. We have military bases there, but what we have REALLY gained is a stable nation in that part of the world, truly dedicated to democracy and ready to join the first world.

I understand what you're saying, and yes it would be great if we could all be friends, but ultimately I think Washington was right when he warned us to beware of foreign entanglements. We would love to go back to isolationism if it was possible, which of course it isn't.

BTW, sorry for all the cursing earlier -- I've been going through mood swings all week, from angry to despondent and everything in-between. There was actually a shouting match with three people in the office on thursday, over a video tape! Irritability is at an all-time high, that's for sure. Right now I'm at the "why bother?" stage of foreign policy. Its a good thing I'm not president, thats for sure...

Re:DNS & mod_gzip

[tshak]

You don't "suffer" from advertisers. Rather, you benefit from the services they finance like "/.".

Uh...

[Anonymous Coward]

Lenin. 1917. Do you know the figures on the
number of Christians killed by atheists for
religious reasons? Look it up, it's quite
enlightening.

Re:Correction

[Heem]

I'd love to say that I hope we dont use a nuke. I'd love to be able to take that stance. I'm sure I'm going to be disagreed with, maybe even modded down, flamed. Probably rightfully so. But you know what, I hope we do nuke those fuckers. It certainaly worked on the Japenese

to quote Leonard Pitts, From the Miami Herald - 9/12/2001

"This is the lesson Japan was taught to its bitter sorrow the last time
anyone
hit us this hard, the last time anyone brought us such abrupt and monumental
pain. When roused, we are righteous in our outrage, terrible in our force.
When provoked by this level of barbarism, we will bear any suffering, pay
any
cost, go to any length, in the pursuit of justice."

All I hope is that we have a well planned attack.

Re:Atheism != communism

[Anonymous Coward]

But, Lenin was athiest. He used his hatred of religion to get people to exterminate religious people.

Re:CNN and MSNBC take note.

[Anonymous Coward]

If you don't know what you're talking about, keep quiet.

During the 2000 elections, CNN set the world record for News (if not all sites) pages served, with peaks of 1.2 MILLION per minute. (~400 times slashdot's peak this week)

Tuesday, CNN eclipsed that, even with the downtime, at 164,200,000 page views. Wednesday was still larger yet, at more than 300,000,000 page views.

I heartily congratulate slashdot and thier team in doing an exceptional job. However, they are no where _near_ CNN's or MSNBC's league.

slashdot has performed...

[freaq]

...admirably.

thank you, rob & crew.

upon hearing rumour of what happened, and not being able to access any of my bookmarked news sites, take a wild guess what url i typed in.
up it came, immediately, with usable information.

good? yes. you've done well. if you still need to break down, go ahead. i find no fault in tears.

tragedy is not a strong enough word for the recent calamity. i (a canadian, if it matters) cried also.

peter

(i sin ø)
e

Re:Pray [etc] For President Bush

[!coward]

Let me just start by stating 2 things:

- I'm portuguese (as in Europe) which means that I might not take things as personal as some of you.

- I don't like the guy (G. W. Bush). I honestly believe you chose an underqualified person for President (although the voting majority didn't actually vote for him, your system made him the winner -- which is ok, it _is_ the same system you've used for years.. if it's wrong, change it and it won't happen again).

First of all, IMHO, there is no excuse, no cause, no ideal, nothing that can justify the killing of people. We define certain circumstances in the law in which this is tolerable: self-defense, etc.. Terrorism, whatever form it may take, isn't one of them. It is intolerable. We all live on the same planet, depend on basically the same things to survive and, most of all, belong to the same species. We all have to accept and live by a certain number of rules in order to make it possible for _everyone_ to have as good of a life as possible. To think that one is superior to any other because you or him/her think/act/feel different is not only stupid, it's a waste of the cognitive faculties we've enhanced, as a species, over thousands of generations.

But the truth is, things don't work. The "system" doesn't work for most people. People die every day of hunger, dehidration, or lack of medicine as common as rain in most industrialized countries. I'm not saying we should try to understand the people who commit acts of terror. At least, not to look for excuses for there are none. But it does seem a terrible waste (not to say an insult to all the deceased) when we fail to realize that any 'normal' human being would never do something as hideous as this Tuesday's attack if things were ok.

The Al-Koran is, unlike 'popular' opinion, a very 'open-minded' text. The message is not about hate, or pain, or punishment, but rather of self-enlightenment, respect and tolerance, much like the "New Testament" for christians or catholics. So why do we see this kind of fanatic behaviour (suicide attacks, I mean) repeatedly associated with arabic people? Could it be that they feel, somehow, _we_ are to blame for a lot of their problems? Could it be that, to some extent, they're actually right?

When you see priests or vicars, like Falwell, feeding intolerance into people do you imediately generalize it to the whole population of priests and vicars? Or their believers? Why don't we extend the same curtesy to arabic people? Why don't we accept the likely possibility that these people have been mis-lead by other people who should know best, but don't? If you feel down, feel that the whole world is against you, wouldn't you be a little more willing to embrace such extremist views?

Should the people involved in the terrorist attacks of the 11th be brought to justice? Damn right! But notice I said 'justice'. We can't fight terrorism with some new form of terrorism (state or country-sponsored assassinations/attacks ARE forms of terrorism). There has been no declaration of war. No state or country or protectorate has declared war against any NATO country. The point I'm trying to make is: you either consider every person in Afghanistan (sorry, can't spell it) a terrorist or otherwise guilty of the attacks on the planes, WTC and Pentagon -- which would justify the envolvement of armies (it _would_ be the same as a declaration of war), or you stipulate that there _are_ people in Afghanistan who have _nothing_ to do with what happened last Tuesday. In this later case, there is nothing that justifies treating those people the same way you/me/we all want to treat terrorists.

What am I trying to say? Generically speaking: I find it already troubling that my country can go to war for reasons I totally disagree with and, I, along with every other citizen of my country (regardless of their views towards that conflict) would pay the price. But it troubles me more to know that, if NATO really gets involved and we end up bombing the hell out of Afghanistan, we will be opening up a precedent whereas any criminal actions by a citizen or group of citizens of any country, or living in any country, can lead to that country becoming extinct. I shudder to think that anyone can be killed because of someone else's actions. In a way, however inexcusable the reasons might have been, the people on the airplanes, Pentagon, WTC towers and those who came to help, who died last Tuesday, died just because of that. They payed the price _someone_ stipulated for things they did not do.

This is the only oportunity we have to really do things right. To show that we DO believe in democracy, in tolerance, and in the principle 'innocent until _proven_ guilty'. And to prove that, no matter how much our hearts call for blood, from sheer shock and pain, we _can_ act as we all say we all should.

I'm not worried if G W Bush is sad because he doesn't want to be the 1st person to use a nuke in a non-war scenario. I'm worried about the thousands, maybe millions of people who will suffer, directly or indirectly should that come to pass.

I sympathise with you all. I may not agree with a lot of the USA's foreign policy, but I do not confuse it with the american people. Please don't do the same with the afghans. If their regime, government, whatever, is somehow responsible (aiding and abeding, for example), then there _are_ solutions out there better than massive destruction. Economic sanctions (Iraq, Germany after WWII), blocking that country out. International Courts to judge all the people suspected of involvement, instead of summary executions (P.R. of China, anyone?). Isn't that how things are supposed to work?

These people have shown they do not know what democracy, freedom and tolerance mean. Let's try to show everyone that we, at least, do.

Re:You are going to be scared....

[Moonshadow]

Clancey is getting to me.

In Executive Orders, terrorists flew a plane into the White House, killing most of the major gov't officials that were there for a ceremony. Sound famaliar?

Don't remember which book it is (Never finished it), but there's one about a small terrorist group gathering materials and machining a nuclear bomb to set off in a football stadium in Colorado. The possibility of it seemed to be almost zero...but so did the possibility of a hijacked plane being flown into the Pentagon. What's scary is that it's not as hard as it seems. With bin Laden's resources, he could have procured the materials and gotten people to carry a nuke-in-a-briefcase into the WTC, and taken out all of Manhattan. Not that I'm saying what happened isn't terrible - it is - but the possibilities for terrorists these days are WAY past frightening.

Terrorists don't care about diplomacy, they don't care about the politics, they care about causing pain, terror, death, and destruction. How are we supposed to fight an enemy that is free to use weaspons many times more destructive than our government will dare to use? How can you guarantee safty?

Look at me...I'm turning into a paranoid conspiracy theorist.