Slashdot Mirror
Are Internet News Sites Ready for Major World News?
An anonymous reader asks: "Heading says it all really - are Internet news websites ready for the next big world event? news.bbc.co.uk already switches format under heavy load (not sure if this is automatic or not) and i'm sure some other sites do the same. But should a major world event take place in the coming months/years, the Internet is going to be the primary news source for many millions of people, particularly those without access to a quality television news service. How will / can it cope?"
personally, I didn't like watching TV on 9/11/02. They were repeating the same garbage all day long. The reporters were rather boring and the news coverage the same.
/. (as most other news sites were unreachable due to traffic) b/c of not only the news but also discussion w/others. It was interesting to read what other people were feeling, especially those that were not in the US.
I preferred to read
1. Resistance to large amounts of sudden traffic.
2. Meta-news from other sites.
Simple really.
-----
fat chicks need love too
What type of backend is running most of the news sites? Are some of them distrubuted? (I know some are, but to what extent and how? )
If you mean a major bandwith spike, then where is the weakest link? Will the pipe fill up before the processing power is toped out?
I know that some ISP's had their bandwidth bursting at the seams during 911, so even if there was nothing wrong with the news/internet/network - the ISP was fragile.
Not really a post - in that I am not giving much in the way of answers, but just trying to ask the right questions. There is so much to consider in such a situation, rather than looking (drooling?) at their massive server farm(s), don't forget about the pipe that feeds it(them).
Create music
The BBC coped because of two main things. The first is because they switched to a low-graphics version. The other reason is that the BBC's servers are geographically spread out. They have servers on several European backbones, and also have seperate servers in New York Telehouse which serves all the content for the people on the other side of the Atlantic.
Thats how they coped, my old mucker.
Agreed. The 2nd plane hit the 2nd tower just as I was pulling into my parking lot at my (then) office. I thought it was a total and tasteless joke (considering the juvenile humor the morning show I listed to at the time was known for). When I unlocked the office, I jumped on CNN's webpage. By the time that the rest of the engineers and the admin staff arrived, CNN was almost unreachable. We did get to watc about 15 minutes more of the coverage before we lost connection.
I tried several other news sites (MSNBC, ABCNews, etc) only to find the same congestion.
No, the internet isn't ready to handle the bandwidth associated with millions of people logging on to get the latest information.
Which leads me to a question: Any *decent* (and FREE) newstickers out there that are totally customizable, and run under Windows? I already checked SourceForge.. I've been using Netropa, but its not set up to allow me to add whatever channels I want. I tried Swen (from Tucows) but it doesn't work at all..
Cruising the internet on my TI-99/4A @ a whopping 300 baud!
A lot of news sites got a taste of what covering big events on the internet is like. Some did okay, most didn't. Even Slashdot learned a few things about handling loads.
Also, it in part led to Google News. I'm actually kinda comfortable with Google handling news, as I think if such an event happens again, Google can just cache the important news.
TV and radio, though, will likely always have the advantage that viewer load doesn't affect them. So, even if someday we move beyond traditional TV/Radio broadcasting, emergency radio broadcasting should be kept in some form.
On September 11th, major news sites like Yahoo, BBC, CNN were entirely flooded with traffic much like the phone system was, going as far as taking down some fairly large servers altogether. What ended up happening was that a bunch of IRC channels (specifically on SlashNET) cropped up with people giving live webcam shots, rumours and snippets of information, mirrors. Then the CNN closed captioning bots started relaying to IRC for those without the cable service. It was interesting as it showed the Internet both failing at and succeeding in its primary designed function, as a communications and information network that could survive a major catastrophe.
s200.org - visit it (me), love it (me).
My observation for 9/11 waw that major news site crawled under the load. However, less often visited news site were responsive all day and gave the same news with the same level of coverage than the big news sites.
So I must say, find some smaller news site and bookmark them. When your big-shot news site will crawl under load, just go to the small one and you will get your news.
BTW if you just want nice video, the Intenet is not the place to go, turn on your TV, you'll get far better image quality and you don't have to wait until the video is buffered.
Being a Brit, the BBC was the first place I turned to for news and basically the whole thing ground to a halt and that was despite the BBC News outfit having upgraded systems substantially to cope with the 2001 UK General Election. Both the UK and US mirror were swamped and basically stopped working. Interestingly the US Mirror site was in New York, not far from the WTC, and despite the fact the power was lost in the entire area, the servers kept going for several days on backup generators until those generators died due to the dust.
It tended to be the second-tier news service like Ananova that could cope, simply because in times of crisis people will always turn to familiar names first.. the BBC, NBC, CBS, CNN etc.
I seem to remember that the low-graphics option came after 9/11, but it's only a partial solution to the problem.. several times since then the BBC have switched to low-graphics but there haven't been any events of the magnitude of 9/11 since then.
Look at it this way.. lets say the US has 50 million office workers with access to the Internet (a pure guesstimate) and they all try to access the same news sites within a window of 30 minutes. On 9/11 people were trying to download videos of the attacks so they could understand what was going on - don't forget that those now familiar images we all know now were completely unthinkable. This combination of huge numbers of users and very high demand for streaming video is almost impossible to keep up with.
In short, on 9/11 the web let us down and the only people who knew what was going on were those with access to televisions. The world has not moved on that much in the past 12 months, so basically the same thing will happen all over again if (God forbid) the same thing happens all over again..
Never email donotemail@WeAreSpammers.com
or at least this is what I think:
http://robots.cnn.com
however, I still think that the best medium for broadcast is not an interactive media like the Internet, but a one-way media like radio or TV;
Anyway, I would rather prefer a text-only information source like during the Gulf War the BBC did on IRC. But I may be wrong on that.
-- There are two kind of sysadmins: Paranoids and Losers. (adapted from D. Bach)
Get yourself a battery powered portable radio. Make sure that it is the type that can receive shortwave frequencies and you will never be without a BBC broadcast. The are lots of small cheap portable radios on the market that receive AM/FM/SW/TV and I'd also expect to see satellite portable radios soon but, I can't imagine paying the subcription for such a thing, especially when SW is availalble.
news.google.com should hold up under even the heaviest loads, and while you might not get the actual site it links to, you should at least be able to get the idea of what's going on based on the headlines.
In a time of crisis, is it really necessary to know the details of a major world event immediately? If a nuke goes off somewhere, I'm not too concerned about who did it--I'm driving to some remote place, THEN I'll start asking the questions.
On Sept. 11th, what did we know for certain:
*4 planes were hijacked
*Two towers fell
*The Pentagon was hit
*A plane went down in PA
everything else was mere specualtion at the time, and everything above could be read by headlines alone.
Just a thought,
sig--we don't need no goddamn sig
The answer to scalability has been there for years, and it's multicast. Multicast is a protocol that implement a one-to-many distribution of the information, allowing very efficient distribution of contents on the internet (the target is that the information should not pass more than once on any given physical line), and dynamic group joining and leaving.
However, ISP and users are confronted to a chicken-and-egg problem: ISP pretend there is no demand for multicast, so that can't justify the investment in increased NOC knowledge, users don't know what it is, and content providers have no support from ISP or user.
Multicast is however the scalable answer for live broadcast and scheduled replay, it's been there for years and I do not loose hope that it will be better used one day.
Seems like the best answer would be automatic load balancing between disparate servers. But how would we get the services to cooperate? E.g. rushlimbaugh.com not be too keen on sharing resources with cnn.com. :) And that begs the question, would the "rescuing" site be entitled for a fee for their failover support?
3. Profit!
2. ???
1. On Soviet Slashdot, a Beowulf cluster of alien Natalie Portman overlords welcomes YOU!
During the horror of the attacks last year, I was surprised and thankful for CNN's approach which allowed them to withstand the barrage of hits:
They switched to an old-school, how-the-web-used-to-be, no-nonsense design. It was basic HTML, with some embedded pictures that contribued to the information. No frills, no ads, no sidebars about the latest crap-news, just the information we were looking for. Needless to say, it also ate a lot less bandwidth.
Of course, they were down part of the morning, but when they came back in the altered format, I thought it was a great move. A few other sites were doing the same thing, and I think they'll remember the technique for the next time something big goes down (hopefully something pleasant next time? I can hope...)
It isn't just the news sites we have to think about. We should also be asking, when the next big event does happen, will people even be able to get online to access the news sites?
I'm not talking about some sort of damage to the communications network. I'm talking about ISPs that enforce strict rules on how many of their customers can get online simultaneously. They are the real threat to the Net as a primary source of urgent information, and it's all about money. They take on millions of customers but total capacity is measured in tens of thousands.
For example, on September 11th there were a few hours when tall buildings in London and other British cities were being evacuated, but many people over here couldn't get online to access vital information because our ISPs have notoriously low capacity and only allow a small percentage of their customers online at any one time.
Obviously this is a greater threat in rural areas because the only available connection method is dial-up.
I am subscribed to a couple of worldwide mailing lists and I have found that email simply rocks in high 'net traffic situations.
During the New York tragedy, much of the traffic on those lists was along the lines of "I can't get to the major sites because the web is clagged solid - can anyone tell me the latest?". And thankfully for a couple of days, the rules about straying from the topic of the mailing list were ignored.
Granted, many of the complaints were actually related to individual corporate firewalls, http gateways and proxy servers, rather than the sites themselves, but the situation stands: for whatever reason, you can't get to the site. Our web proxy fell over under the load, but our SMTP gateway just kept on going. And so did most others around the world. And I imagine that NNTP stuff worked just as well the SMTP stuff.
Remember folks, the Internet is a lot more than the Web!
Ahh - My eye!
The doctor said I'm not supposed to get Slashdot in it!
It was Princess Di's death. I was on shift the night it happened and it pretty much brought all news websites to their knees. That was the first time I noticed the low bandwidth version of CNN. At first I thought the site was choking because it looked like some graphics were not loading.
Still, I'll give it to Slashdot and to IRC. I spent most of 9/11 on IRC transcribing what was being reported on CNN, since for a while the site was pretty much useless. A bunch of us where also taking screen captures and posting them online so people could see the horror. I still have captures of the first flyover of the Pentagon, which is less than 10 miles from my office.
Pedro
----
The Insomniac Coder
Whether the fibre is dark or not isn't the problem. "The Internet" can handle the ammount of traffic that an event would create. The problem is the servers that everyone want to access will fail under that load. I bet dollars to donuts that news sites were going down on 9/11, but the sites where you can see chicks going down on each other were just fine that day.
You're probably right; the report notes that the infrastructure was fine but the web servers were overwhelmed. Lighting up that dark fibre would make it easier to deploy Akamai-like solutions to replicate content to distribution points closer to the consumer.
If any one is interested about how the BBC's network can handle a /.ing there are network diagrams here
The internet was designed to be a somewhat peer-to-peer infrastructure.
IRC, while admitedly incorporating a client/server architecture, is still more peer to peer (it is, after all, Internet relay chat) than a news site (which is completely server/client).
The "failures" were those parts of the 'net that didn't obey p2p, and the "successes" were the systems that did.
Even Kazaa lit up with ripped/pirated CNN broadcasts. I didn't have access to a TV that day, either. I got my footage from Kazaa at School.
for distribution of widely needed information such as news?
I've read nearly 100 comments so far and while many mentioned the inherent flaws in the "massive webserver" model we currently use for virtually all web news traffic, no one mentioned the alternatives. For things like news distribution television and radio have an incredible advantage (they don't have to double any output to reach twice as many people as normal) but peer to peer systems would also work wonders in situations like this.
Maybe that is the rationale behind project IRIS, the recent US government backed research into a complete network founded on the peer to peer methodology.
www.newscientist.com/news/news.jsp?id=ns99992861
iris.lcs.mit.edu/
"The best argument against democracy is a five minute chat with the average voter."
--Winston Churchill
After September 11th last year, many news sites found they could not cope. CNN is now hosted by Akamai's Edgesuite system. The BBC still serves all of it's own content. The switching between light and heavy pages is not automatic and both are kept up to date so it is easy to switch between them. Also due to the closeness of Telehouse New York to the WTC, half of the BBC's servers were out of action when NY lost power and the backup generators could not cope with all the dust in the air. This actually meant that the traffic was sent all the way to the UK. In light of this, a lot of effort was put into increasing the serving architecture to be ready for another such event...Although it has yet to be tested.
If multicast were ubiquitous then things could have been much better. If people could received the html only web page and turn to the mbone or some other multicast network for the streaming video then the net could probably shrugged almost any event off. Since porn is one of the few things that makes money on the net I am suprised that multicast for streaming smut hasn't become more prevelant.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
Seriously though, a major distributed system could handle a massive load. Maybe that's what news.google.com is about. If the news sites all mirrored the same content (which they pretty much did on sept 11th anyway and do in most major events) they could probably handle the traffic increase between all of them.
As for people using the net versus TV, it happens because TV doesn't provide as much information as people want sometimes. Websites often link to additional info that TV won't cover as it's time to repeat the same report in 5 minutes.
Oh and I thought the net coped pretty well with the last event. Phones were down all day but my b/f in NYC was able to call me in San Francisco using dialpad and keep a connection long enough to wake me up and let me know what was going on.
Introducing the new Occam Fusion! Now with sqrt(-1) fewer blades!