Are Internet News Sites Ready for Major World News?

An anonymous reader asks: "Heading says it all really - are Internet news websites ready for the next big world event? news.bbc.co.uk already switches format under heavy load (not sure if this is automatic or not) and i'm sure some other sites do the same. But should a major world event take place in the coming months/years, the Internet is going to be the primary news source for many millions of people, particularly those without access to a quality television news service. How will / can it cope?"

Comments

In a word, no.

[FreeLinux]

No they can't cope. It's been proven already. Even giants like the BBC and CNN had several moments where they could not handle the load on September 11th.

I'm sure that they have taken steps to improve things in the future but, there is only so much that you can do, or at least do cost effectively. There is no substitute for hardware and bandwidth but, maintaining enough to support the entire planet at one critical moment in time, that may or may not come, is not cost effective.

When the time comes, the news sites will buckle under the load, just as the telephone system does. The best source for news, during times of disaster are television and more so, radio. Even in the most remote places, you can still get radio and with new satellite radio, you can get it anywhere.

September 11th

[sql*kitten]

Keynote have oublished a reporton the performance of major web sites on September 11th, 2001.

Of course, there's a lot of dark fibre around, so the capacity is there if it's really needed. Once the current recession is over, we can expect to go back to the days of massive overprovision and redundancy as content and bandwidth providers seek to build in capacity to handle peaks. What will really help is multicasting for video streams, and well-designed caches at ISPs.

Re:September 11th

[drdink]

As a SlashNET administrator, I appreciate the fact that people acknowledge and appreciate the efforts of our users to provide up to the minute news in times of crisis. We hope to continue doing this in the future as time dictates, and we've improved our ability to do so in the future as necessary.

The problem with the web is that it is graphics intensive. When you go to CNN, you have to download a ton of graphics, you have to initiate a new connection with the server on each request, etc. With IRC, you don't have any of the graphics and you don't have to reconnect to it in order to get updates 5 minutes later.

The history of IRC is pretty spotty. Most of the times it can be pretty lame and pointless, but it has always become a useful communications tool. I hope this practice continues.

9/11/01 and CNN

[Anonymous Coward]

On 9/11/01, many people may have had trouble accessing CNN. There is a good reason for this. CNN was in the middle of a server upgrade. If memory serves me correctly (and keep in mind this is second-hand), CNN only had eight servers running for cnn.com at the time.

Needless to say, cnn.com really had to get more servers into production quickly. They worked with Sun to get several hundred servers on site and running.

I don't know why cnn.com had such an upgrade strategy, but it is what happened....

People in offices perhaps?

[Jack_Frost]

I have web access here at work without ready access to cable news services on TV. I watched/read about the events of September 11th on the web.

Re:Did it do that badly last time?

[Anonymous Coward]

Actually the BBC site was one of the sites I was able to get to. I think perhaps because people in the US didn't even think of going to a British site for US news. CNN, MSNBC, etc were the first ones we thought of.

interesting fact

[hype7]

one of the passengers on one of the planes that came down on 9/11 (it was the one that crashed in the field, IIRC) was a founder of Akamai Networks, one of the load sharing/distribution companies that allow bandwidth to scale according to demand. As his plane came down, his company was entering one of the most demanding days in its history, as more people were targeting news sites at once than ever before.

It's organisations like that which will assist in the next big news item.

-- james

How sites seem to cope now

[jht]

Ever since 9/11, I've noticed that the heavily-trafficked sites cope with sudden floods of hits by switching over to static pages with minimal graphics. The NY Times, for instance, did this when the AA flight went down in Queens last November. CNN's done it a couple of times as well.

When we're looking at scale, though, it's useful for us to remember that these sites can handle way more traffic than even the typical slashdotting can deliver. Most breaking major news can be handled by them with only a little bit of slowdown. It's only the 9/11-scale events that can really bring the news sites to their knees - so lets hope that we don't have to see anything that brings on a overload scenario for the big news sites.

The other thing to consider is that most of the news providers are still investing some money in their infrastructure - just less than before. It's very well possible that a 9/11-scale event might not hammer the servers the way they were hammered last year. A lot of web sysadmins learned valuable lessons that day that I'm sure have been applied since then.

Re:How sites seem to cope now

[MtViewGuy]

I think the news sites should have the ability to run multiple mirrors of their own web site all over the world. That way, instead of everyone trying to log onto one centralized site they just log onto the closest mirror site. After all, Microsoft now uses Akamai as their primary means to distribute patches and updates online, and Akamai is one of the companies specializing in this type of business.

Re:won't replace TV

[nicklott]

The BBC has its bandwidth graphs online here.

Sept 11 is just on the left of graph at the bottom. Interestingly a normal day's traffic is now greater than the sept 11th spike, maybe they could handle major news events.

(I can't think what the early april spike is, but the raised traffic in june/july is the world cup)

A Few Ideas

[limekiller4]

When 9/11 hit, the first thing I did was wget about two dozen news sites and thousands of blogs immediately. CNN, in particular, got blacked out really, really hard, and was reduced to one image on the front. I wish I had my archives available to post but they're rather deeply gzipped ...somewhere. =)

Akamai had their work cut out for them that day, I can tell you. I was lucky. I called out sick.

But none of this really answers the question -- how do you cover your butt and insure that you keep getting a news feed when/if you need it? I noticed that when I go to www.php.com, it's quite slow. So I started using uk.php.net and it zips right along. The moral of this story is that you might want to find 3-5 news sites that you consider good (and a factor in this probably should be how fast news gets to their site), then find some printer-friendly version/low bandwidth links to their front pages. Those are far less likely to be used when things get crazy. Drop some admins an email, perhaps certain versions of their site is located on entirely seperate servers and might go unscathed during a 9/11-ish rerun.

Re:Usenet Example: news.announce.important

[grainofsand]

Recent posts to this important announcement newsgroup include:

17 Sep 2002 Blind Vigilantes
23 Apr 2002 Art and all that Jazz
16 Oct 2001 My car was recently struck by a United Parcel

Maybe not a such a great source of breaking news - there are no Sept 11-related posts at all.

Initial notification only?

[Toby Moray]

It's already been kinda mentioned, but doesn't the web just provide the initial notification that something big is happening, then we all switch to "standard" media news to find out more?

CNN and others provide email alerts for breaking news (which notified me of 911), the web then provides initial reports, then we switch the TV on and get realtime news as the web grinds to a halt.

Though if the next major event happens on the same day as a game demo or a new Matrix trailer are released, we're truly stuffed...

CNN.com

[Whizard]

William LeFebvre (of CNN.com) has an excellent talk that he's given at least twice at Usenix events (I saw it at Usenix '02, he also gave it at LISA '01), that gives a lot of detail as to the things that had to be coped with to keep CNN.com running on September 11. I can't find the full-text of the speech anywhere online, but there's some information at this site that at least gives you an idea. Interesting stuff!

Streaming Media and Large Audiences

[ellbee]

I ran a major streaming media distribution net on 9/11. We saw a steady 75k - 100k simultaneous users (mostly audio) for the next several days as people used webcasts to get live news while at work. We had a few glitches as video streams were inserted by customers without warning us of the oncoming load, but they were mostly transient as we adjusted for capacity. At the edge we were seeing between two and four terabits/second being sent out, and could have turned up more if we needed it.

I've since built some even larger systems; I've no doubt that it's possible to scale Internet streaming media distribution to millions or even tens of millions of simultaneous viewers using today's technology and protocols.

ellbee