Slashdot Mirror
Are Internet News Sites Ready for Major World News?
An anonymous reader asks: "Heading says it all really - are Internet news websites ready for the next big world event? news.bbc.co.uk already switches format under heavy load (not sure if this is automatic or not) and i'm sure some other sites do the same. But should a major world event take place in the coming months/years, the Internet is going to be the primary news source for many millions of people, particularly those without access to a quality television news service. How will / can it cope?"
Hopefully news.bbc.co.uk can cope with having it's link on Slashdot's homepage...
Certainly when the events of September 11th took place, for those of us at work in the UK without a television at hand the only way to keep up with events was via the web.
News sites failed to cope with the load - millions of people trying to access the same sites meant that no amount of bandwidth could cope with demand.
For this reason, I don't think that the web is going to replace television as a source of live news coverage anytime soon.
How many people have internet access, but no access to TV, radio, or other broadcast recievers? For major news stories boardcast medium will always be the main method of disseminating information to the masses, client-server systems aren't really designed for this purpose.
Why? Are you planning one?
Al.The Daily ACK - Eclectic posts by yet another hacker
No they can't cope. It's been proven already. Even giants like the BBC and CNN had several moments where they could not handle the load on September 11th.
I'm sure that they have taken steps to improve things in the future but, there is only so much that you can do, or at least do cost effectively. There is no substitute for hardware and bandwidth but, maintaining enough to support the entire planet at one critical moment in time, that may or may not come, is not cost effective.
When the time comes, the news sites will buckle under the load, just as the telephone system does. The best source for news, during times of disaster are television and more so, radio. Even in the most remote places, you can still get radio and with new satellite radio, you can get it anywhere.
The BBC coped because of two main things. The first is because they switched to a low-graphics version. The other reason is that the BBC's servers are geographically spread out. They have servers on several European backbones, and also have seperate servers in New York Telehouse which serves all the content for the people on the other side of the Atlantic.
Thats how they coped, my old mucker.
Cheers,
Ian
Keynote have oublished a reporton the performance of major web sites on September 11th, 2001.
Of course, there's a lot of dark fibre around, so the capacity is there if it's really needed. Once the current recession is over, we can expect to go back to the days of massive overprovision and redundancy as content and bandwidth providers seek to build in capacity to handle peaks. What will really help is multicasting for video streams, and well-designed caches at ISPs.
They can do it the same way I cope when my power goes off... A cheep battery operated shortwave radio tuned to the BBC or other quality station. IMHO, I'm pretty sure if they can't get access to a TV then what chances do they have at getting the internet?
------
Never underestimate the power of stupid people in large groups.
"...particularly those without access to a quality television news service."
Gee, that's pretty much everyone.
On September 11th, major news sites like Yahoo, BBC, CNN were entirely flooded with traffic much like the phone system was, going as far as taking down some fairly large servers altogether. What ended up happening was that a bunch of IRC channels (specifically on SlashNET) cropped up with people giving live webcam shots, rumours and snippets of information, mirrors. Then the CNN closed captioning bots started relaying to IRC for those without the cable service. It was interesting as it showed the Internet both failing at and succeeding in its primary designed function, as a communications and information network that could survive a major catastrophe.
s200.org - visit it (me), love it (me).
My observation for 9/11 waw that major news site crawled under the load. However, less often visited news site were responsive all day and gave the same news with the same level of coverage than the big news sites.
So I must say, find some smaller news site and bookmark them. When your big-shot news site will crawl under load, just go to the small one and you will get your news.
BTW if you just want nice video, the Intenet is not the place to go, turn on your TV, you'll get far better image quality and you don't have to wait until the video is buffered.
Being a Brit, the BBC was the first place I turned to for news and basically the whole thing ground to a halt and that was despite the BBC News outfit having upgraded systems substantially to cope with the 2001 UK General Election. Both the UK and US mirror were swamped and basically stopped working. Interestingly the US Mirror site was in New York, not far from the WTC, and despite the fact the power was lost in the entire area, the servers kept going for several days on backup generators until those generators died due to the dust.
It tended to be the second-tier news service like Ananova that could cope, simply because in times of crisis people will always turn to familiar names first.. the BBC, NBC, CBS, CNN etc.
I seem to remember that the low-graphics option came after 9/11, but it's only a partial solution to the problem.. several times since then the BBC have switched to low-graphics but there haven't been any events of the magnitude of 9/11 since then.
Look at it this way.. lets say the US has 50 million office workers with access to the Internet (a pure guesstimate) and they all try to access the same news sites within a window of 30 minutes. On 9/11 people were trying to download videos of the attacks so they could understand what was going on - don't forget that those now familiar images we all know now were completely unthinkable. This combination of huge numbers of users and very high demand for streaming video is almost impossible to keep up with.
In short, on 9/11 the web let us down and the only people who knew what was going on were those with access to televisions. The world has not moved on that much in the past 12 months, so basically the same thing will happen all over again if (God forbid) the same thing happens all over again..
Never email donotemail@WeAreSpammers.com
or at least this is what I think:
http://robots.cnn.com
however, I still think that the best medium for broadcast is not an interactive media like the Internet, but a one-way media like radio or TV;
Anyway, I would rather prefer a text-only information source like during the Gulf War the BBC did on IRC. But I may be wrong on that.
-- There are two kind of sysadmins: Paranoids and Losers. (adapted from D. Bach)
Get yourself a battery powered portable radio. Make sure that it is the type that can receive shortwave frequencies and you will never be without a BBC broadcast. The are lots of small cheap portable radios on the market that receive AM/FM/SW/TV and I'd also expect to see satellite portable radios soon but, I can't imagine paying the subcription for such a thing, especially when SW is availalble.
news.google.com should hold up under even the heaviest loads, and while you might not get the actual site it links to, you should at least be able to get the idea of what's going on based on the headlines.
In a time of crisis, is it really necessary to know the details of a major world event immediately? If a nuke goes off somewhere, I'm not too concerned about who did it--I'm driving to some remote place, THEN I'll start asking the questions.
On Sept. 11th, what did we know for certain:
*4 planes were hijacked
*Two towers fell
*The Pentagon was hit
*A plane went down in PA
everything else was mere specualtion at the time, and everything above could be read by headlines alone.
Just a thought,
sig--we don't need no goddamn sig
The answer to scalability has been there for years, and it's multicast. Multicast is a protocol that implement a one-to-many distribution of the information, allowing very efficient distribution of contents on the internet (the target is that the information should not pass more than once on any given physical line), and dynamic group joining and leaving.
However, ISP and users are confronted to a chicken-and-egg problem: ISP pretend there is no demand for multicast, so that can't justify the investment in increased NOC knowledge, users don't know what it is, and content providers have no support from ISP or user.
Multicast is however the scalable answer for live broadcast and scheduled replay, it's been there for years and I do not loose hope that it will be better used one day.
It was Larry Niven who predicted the idea of "flash crowds". Of course, he was envisioning physical crowds via teleportation, but the basic idea still holds. It's only going to get worse as more and more people use the net.
Look at it this way: in a primative society, a clan or village would usually have a storyteller or sage who gathered the news of the world in story form and re-told as appropriate. We should not be supprised that it takes millions, perhaps even hundreds of millions of people to be the story-tellers to 6 billion (that's a US billion).
If the Internet had a higher percentage of useful sites for news (not just talking jpeg-heads, but innovative ways of conveying the STORIES that the news represents), then no one of them would be loaded down and the backbones would be the only bottleneck. Notice that so many of us flocked to Slashdot when the towers fell? Wonder why? Because Slashdot, for good or ill, is our community's storyteller, and we instinctively come here to understand how our community is reacting.
Folks,
If you don't have a portable AM/FM radio, or even better a shortwave receiver, then get one TODAY. Get some spare battteries for it as well.
The simple fact is if you want to hear what's going on during a "major event" radio is the best way to do it. And you have evacuate in a hurry, you sure as hell aren't going to be taking your 60" flat screen TV with you. You want pictures, wait for the evening news, if you want to know what's going on NOW, get a radio.
Even better, get yourself licensed as a ham radio operator so you can be part of the communication solution if needed (yes, amateur radio is still important, even today).
You know you're a geek if you've ever replied to a tagline.
particularly those without access to a quality television news service.
Isn't that a oxymoron?
Higher Logics: where programming meets science.
During the horror of the attacks last year, I was surprised and thankful for CNN's approach which allowed them to withstand the barrage of hits:
They switched to an old-school, how-the-web-used-to-be, no-nonsense design. It was basic HTML, with some embedded pictures that contribued to the information. No frills, no ads, no sidebars about the latest crap-news, just the information we were looking for. Needless to say, it also ate a lot less bandwidth.
Of course, they were down part of the morning, but when they came back in the altered format, I thought it was a great move. A few other sites were doing the same thing, and I think they'll remember the technique for the next time something big goes down (hopefully something pleasant next time? I can hope...)
To state the obvious, the major news sites would have to have not only leaner pages, but also have the infrastructure to withstand a slashdotting-with-hair-on-it. Leaner, lower bandwidth web pages benefit every one, every day, but for daily needs the infrastructure is going to be expensive overkill.
In contrast, more of the tech sites were already used to heavy loads and I would guess that his brought in a larger than normal number of new and infrequent visitors. Maybe it was my imagination, but it seemed that after that many mainstream newspapers, magazines, and radio magazines started to carry more cutting edge tech info and topics and providing in a much more timely manner - days instead of weeks or months.
It would be interesting to map how much the coverage and timeliness of tech issues by the mainstream press changed, when it changed, and how much was related to being able stay on line.
Beta is broken and the link to classic doesn't work. Stop wasting our time or there won't be anybody left here.
It isn't just the news sites we have to think about. We should also be asking, when the next big event does happen, will people even be able to get online to access the news sites?
I'm not talking about some sort of damage to the communications network. I'm talking about ISPs that enforce strict rules on how many of their customers can get online simultaneously. They are the real threat to the Net as a primary source of urgent information, and it's all about money. They take on millions of customers but total capacity is measured in tens of thousands.
For example, on September 11th there were a few hours when tall buildings in London and other British cities were being evacuated, but many people over here couldn't get online to access vital information because our ISPs have notoriously low capacity and only allow a small percentage of their customers online at any one time.
Obviously this is a greater threat in rural areas because the only available connection method is dial-up.
I am subscribed to a couple of worldwide mailing lists and I have found that email simply rocks in high 'net traffic situations.
During the New York tragedy, much of the traffic on those lists was along the lines of "I can't get to the major sites because the web is clagged solid - can anyone tell me the latest?". And thankfully for a couple of days, the rules about straying from the topic of the mailing list were ignored.
Granted, many of the complaints were actually related to individual corporate firewalls, http gateways and proxy servers, rather than the sites themselves, but the situation stands: for whatever reason, you can't get to the site. Our web proxy fell over under the load, but our SMTP gateway just kept on going. And so did most others around the world. And I imagine that NNTP stuff worked just as well the SMTP stuff.
Remember folks, the Internet is a lot more than the Web!
Ahh - My eye!
The doctor said I'm not supposed to get Slashdot in it!
Multicast news services worked well during 9/11 and there is no reason to think that they won't the next time. Multicast is specifically designed not to "melt down" under extreme changes in audience.
The trouble is that not everyone is multicast enabled, but this shows real promise in handling news and emergency information over the Internet.
one of the passengers on one of the planes that came down on 9/11 (it was the one that crashed in the field, IIRC) was a founder of Akamai Networks, one of the load sharing/distribution companies that allow bandwidth to scale according to demand. As his plane came down, his company was entering one of the most demanding days in its history, as more people were targeting news sites at once than ever before.
It's organisations like that which will assist in the next big news item.
-- james
Ever since 9/11, I've noticed that the heavily-trafficked sites cope with sudden floods of hits by switching over to static pages with minimal graphics. The NY Times, for instance, did this when the AA flight went down in Queens last November. CNN's done it a couple of times as well.
When we're looking at scale, though, it's useful for us to remember that these sites can handle way more traffic than even the typical slashdotting can deliver. Most breaking major news can be handled by them with only a little bit of slowdown. It's only the 9/11-scale events that can really bring the news sites to their knees - so lets hope that we don't have to see anything that brings on a overload scenario for the big news sites.
The other thing to consider is that most of the news providers are still investing some money in their infrastructure - just less than before. It's very well possible that a 9/11-scale event might not hammer the servers the way they were hammered last year. A lot of web sysadmins learned valuable lessons that day that I'm sure have been applied since then.
-- Josh Turiel
"2. Do not eat iPod Shuffle."
It was Princess Di's death. I was on shift the night it happened and it pretty much brought all news websites to their knees. That was the first time I noticed the low bandwidth version of CNN. At first I thought the site was choking because it looked like some graphics were not loading.
Still, I'll give it to Slashdot and to IRC. I spent most of 9/11 on IRC transcribing what was being reported on CNN, since for a while the site was pretty much useless. A bunch of us where also taking screen captures and posting them online so people could see the horror. I still have captures of the first flyover of the Pentagon, which is less than 10 miles from my office.
Pedro
----
The Insomniac Coder
When 9/11 hit, the first thing I did was wget about two dozen news sites and thousands of blogs immediately. CNN, in particular, got blacked out really, really hard, and was reduced to one image on the front. I wish I had my archives available to post but they're rather deeply gzipped ...somewhere. =)
Akamai had their work cut out for them that day, I can tell you. I was lucky. I called out sick.
But none of this really answers the question -- how do you cover your butt and insure that you keep getting a news feed when/if you need it? I noticed that when I go to www.php.com, it's quite slow. So I started using uk.php.net and it zips right along. The moral of this story is that you might want to find 3-5 news sites that you consider good (and a factor in this probably should be how fast news gets to their site), then find some printer-friendly version/low bandwidth links to their front pages. Those are far less likely to be used when things get crazy. Drop some admins an email, perhaps certain versions of their site is located on entirely seperate servers and might go unscathed during a 9/11-ish rerun.
My
Limekiller
the Internet is going to be the primary news source for many millions of people, particularly those without access to a quality television news service. How will / can it cope?
Huh?? I'm not sure what you're talking about but I'm pretty sure I don't recognize it. Quality television service is much more widespread than the Internet. I'm N. American, but I've lived for years in Africa and Asia. I can assure you that in "None of the above" has the web surpassed broadcast media as a source of news for any but an elite few. And the comment is irrelevant for the elite since they have access to "all of the above"-plus.
Seriously, even in the smallest, poorest villages around the world several people will have radios and access to VOA, BBC, a national broadcast network and one or two regional stations. In addition most villages will have at least one television.
The internet is a bit player if it's a player at all
Yes.
I contacted them all and they said they're ready.
A spokesman for CNN.com said "after talking to several vendors including Sun, IBM and Microsoft, hotnakedteens.com won the business by showing they routinely handle traffic 10 times the traffic we received around Sept 11."
Read reviews of shopping cart software
But should a major world event take place in the coming months/years, the Internet is going to be the primary news source for many millions of people, particularly those without access to a quality television news service.
Please be advised that your set needs adjusting... It's pretty clear from the evidence (and from a phenomenological point of view if you observe your own reactions) that the experience of watching a major event on television as it unfolds barely qualifies as useful information, due in part to the nature of the medium, but largely due to the nature of media filters and techniques. When you see something like 9/11 going on, it's much closer to entertainment, unfortunately, than providing one with reconnaisance leading to rational behaviour. The drama of the moment helps you develop powerful emotions in relation to the event, but what kind of info do you really get?
When it comes to war, TV obscures. For instance, see this study on media and the gulf war. [Remember that? Oh wait, it's still happening.] A salient quote:
In other words, you'd actually be better off combing through usenet than sucking on the immediacy of the glass teat.
Qualifier: I've worked in media-democracy-oriented film/video for years, I'm involved and devoted to the medium!
Damn those pesky terrorists
If multicast were ubiquitous then things could have been much better. If people could received the html only web page and turn to the mbone or some other multicast network for the streaming video then the net could probably shrugged almost any event off. Since porn is one of the few things that makes money on the net I am suprised that multicast for streaming smut hasn't become more prevelant.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
I've since built some even larger systems; I've no doubt that it's possible to scale Internet streaming media distribution to millions or even tens of millions of simultaneous viewers using today's technology and protocols.
ellbee
You can't fight in here - this is the war room!
Damn, all of us in the US are screwed!
Read the EFF's Fair Use FAQ
Unless it's on slashdot, and then there's bound to be 17 moderators who mod it down for the hell of it before people start realizing it on their own.