Slashdot Mirror
Are Internet News Sites Ready for Major World News?
An anonymous reader asks: "Heading says it all really - are Internet news websites ready for the next big world event? news.bbc.co.uk already switches format under heavy load (not sure if this is automatic or not) and i'm sure some other sites do the same. But should a major world event take place in the coming months/years, the Internet is going to be the primary news source for many millions of people, particularly those without access to a quality television news service. How will / can it cope?"
Hopefully news.bbc.co.uk can cope with having it's link on Slashdot's homepage...
Certainly when the events of September 11th took place, for those of us at work in the UK without a television at hand the only way to keep up with events was via the web.
News sites failed to cope with the load - millions of people trying to access the same sites meant that no amount of bandwidth could cope with demand.
For this reason, I don't think that the web is going to replace television as a source of live news coverage anytime soon.
How many people have internet access, but no access to TV, radio, or other broadcast recievers? For major news stories boardcast medium will always be the main method of disseminating information to the masses, client-server systems aren't really designed for this purpose.
On sep. 11 last year I watched the first hour of it live from BBC's homepage without to many problems.
Seemed fine to me (not that I was thinking much about the quality). Was it really that bad?
TC - My Photos..
Why? Are you planning one?
Al.The Daily ACK - Eclectic posts by yet another hacker
Sites can do their best to anticpate heavy load, but off-the-map events like 9/11 tend to reveal weaknesses in systems (which potentially can be elsewhere in the network). Also, it's pretty expensive to engineer to contantly be ready for such rare occurances.
Here's what I do: Bitty Browser & Andromeda
This is how.
1. Resistance to large amounts of sudden traffic.
2. Meta-news from other sites.
Simple really.
-----
fat chicks need love too
No they can't cope. It's been proven already. Even giants like the BBC and CNN had several moments where they could not handle the load on September 11th.
I'm sure that they have taken steps to improve things in the future but, there is only so much that you can do, or at least do cost effectively. There is no substitute for hardware and bandwidth but, maintaining enough to support the entire planet at one critical moment in time, that may or may not come, is not cost effective.
When the time comes, the news sites will buckle under the load, just as the telephone system does. The best source for news, during times of disaster are television and more so, radio. Even in the most remote places, you can still get radio and with new satellite radio, you can get it anywhere.
What type of backend is running most of the news sites? Are some of them distrubuted? (I know some are, but to what extent and how? )
If you mean a major bandwith spike, then where is the weakest link? Will the pipe fill up before the processing power is toped out?
I know that some ISP's had their bandwidth bursting at the seams during 911, so even if there was nothing wrong with the news/internet/network - the ISP was fragile.
Not really a post - in that I am not giving much in the way of answers, but just trying to ask the right questions. There is so much to consider in such a situation, rather than looking (drooling?) at their massive server farm(s), don't forget about the pipe that feeds it(them).
Create music
The BBC coped because of two main things. The first is because they switched to a low-graphics version. The other reason is that the BBC's servers are geographically spread out. They have servers on several European backbones, and also have seperate servers in New York Telehouse which serves all the content for the people on the other side of the Atlantic.
Thats how they coped, my old mucker.
Cheers,
Ian
Keynote have oublished a reporton the performance of major web sites on September 11th, 2001.
Of course, there's a lot of dark fibre around, so the capacity is there if it's really needed. Once the current recession is over, we can expect to go back to the days of massive overprovision and redundancy as content and bandwidth providers seek to build in capacity to handle peaks. What will really help is multicasting for video streams, and well-designed caches at ISPs.
They can do it the same way I cope when my power goes off... A cheep battery operated shortwave radio tuned to the BBC or other quality station. IMHO, I'm pretty sure if they can't get access to a TV then what chances do they have at getting the internet?
------
Never underestimate the power of stupid people in large groups.
That was my first thought. On that day Slashdot, Kuro5hin, and other places became "rip'n'read" sites and held up quite well under the load.
Best Slashdot Co
A lot of news sites got a taste of what covering big events on the internet is like. Some did okay, most didn't. Even Slashdot learned a few things about handling loads.
Also, it in part led to Google News. I'm actually kinda comfortable with Google handling news, as I think if such an event happens again, Google can just cache the important news.
TV and radio, though, will likely always have the advantage that viewer load doesn't affect them. So, even if someday we move beyond traditional TV/Radio broadcasting, emergency radio broadcasting should be kept in some form.
"...particularly those without access to a quality television news service."
Gee, that's pretty much everyone.
On September 11th, major news sites like Yahoo, BBC, CNN were entirely flooded with traffic much like the phone system was, going as far as taking down some fairly large servers altogether. What ended up happening was that a bunch of IRC channels (specifically on SlashNET) cropped up with people giving live webcam shots, rumours and snippets of information, mirrors. Then the CNN closed captioning bots started relaying to IRC for those without the cable service. It was interesting as it showed the Internet both failing at and succeeding in its primary designed function, as a communications and information network that could survive a major catastrophe.
s200.org - visit it (me), love it (me).
With the prevalence of the internet as a means for distribution of all forms of data, new ways of meeting these needs are needed. No longer can one use traditional methods of increasing pipe size or basic colocation and assume that you're back will be covered. We're seeing increasing occurrences of sites being hammered (for whatever reason) and not just the small ones. While the internet may be a massively distributed thing, it still has some major Hopefully this is an area which the methodology of P2P systems and on-the-fly mirroring can help with. If something is in high demand, it should be made _easier_ to get hold off, not harder.
My observation for 9/11 waw that major news site crawled under the load. However, less often visited news site were responsive all day and gave the same news with the same level of coverage than the big news sites.
So I must say, find some smaller news site and bookmark them. When your big-shot news site will crawl under load, just go to the small one and you will get your news.
BTW if you just want nice video, the Intenet is not the place to go, turn on your TV, you'll get far better image quality and you don't have to wait until the video is buffered.
Being a Brit, the BBC was the first place I turned to for news and basically the whole thing ground to a halt and that was despite the BBC News outfit having upgraded systems substantially to cope with the 2001 UK General Election. Both the UK and US mirror were swamped and basically stopped working. Interestingly the US Mirror site was in New York, not far from the WTC, and despite the fact the power was lost in the entire area, the servers kept going for several days on backup generators until those generators died due to the dust.
It tended to be the second-tier news service like Ananova that could cope, simply because in times of crisis people will always turn to familiar names first.. the BBC, NBC, CBS, CNN etc.
I seem to remember that the low-graphics option came after 9/11, but it's only a partial solution to the problem.. several times since then the BBC have switched to low-graphics but there haven't been any events of the magnitude of 9/11 since then.
Look at it this way.. lets say the US has 50 million office workers with access to the Internet (a pure guesstimate) and they all try to access the same news sites within a window of 30 minutes. On 9/11 people were trying to download videos of the attacks so they could understand what was going on - don't forget that those now familiar images we all know now were completely unthinkable. This combination of huge numbers of users and very high demand for streaming video is almost impossible to keep up with.
In short, on 9/11 the web let us down and the only people who knew what was going on were those with access to televisions. The world has not moved on that much in the past 12 months, so basically the same thing will happen all over again if (God forbid) the same thing happens all over again..
Never email donotemail@WeAreSpammers.com
or at least this is what I think:
http://robots.cnn.com
however, I still think that the best medium for broadcast is not an interactive media like the Internet, but a one-way media like radio or TV;
Anyway, I would rather prefer a text-only information source like during the Gulf War the BBC did on IRC. But I may be wrong on that.
-- There are two kind of sysadmins: Paranoids and Losers. (adapted from D. Bach)
Needless to say, cnn.com really had to get more servers into production quickly. They worked with Sun to get several hundred servers on site and running.
I don't know why cnn.com had such an upgrade strategy, but it is what happened....
Get yourself a battery powered portable radio. Make sure that it is the type that can receive shortwave frequencies and you will never be without a BBC broadcast. The are lots of small cheap portable radios on the market that receive AM/FM/SW/TV and I'd also expect to see satellite portable radios soon but, I can't imagine paying the subcription for such a thing, especially when SW is availalble.
This is where proxies come in handy. If there are 1000 people in a large corporation trying to access the web at once on such a day, then a proxy would reduce the number of duplicate requests being made to the web site involved.
At the same time maybe the HTTP procotol needs a version that is capable of UDP broadcasts in special cases?
Jumpstart the tartan drive.
news.google.com should hold up under even the heaviest loads, and while you might not get the actual site it links to, you should at least be able to get the idea of what's going on based on the headlines.
In a time of crisis, is it really necessary to know the details of a major world event immediately? If a nuke goes off somewhere, I'm not too concerned about who did it--I'm driving to some remote place, THEN I'll start asking the questions.
On Sept. 11th, what did we know for certain:
*4 planes were hijacked
*Two towers fell
*The Pentagon was hit
*A plane went down in PA
everything else was mere specualtion at the time, and everything above could be read by headlines alone.
Just a thought,
sig--we don't need no goddamn sig
The answer to scalability has been there for years, and it's multicast. Multicast is a protocol that implement a one-to-many distribution of the information, allowing very efficient distribution of contents on the internet (the target is that the information should not pass more than once on any given physical line), and dynamic group joining and leaving.
However, ISP and users are confronted to a chicken-and-egg problem: ISP pretend there is no demand for multicast, so that can't justify the investment in increased NOC knowledge, users don't know what it is, and content providers have no support from ISP or user.
Multicast is however the scalable answer for live broadcast and scheduled replay, it's been there for years and I do not loose hope that it will be better used one day.
Seems like the best answer would be automatic load balancing between disparate servers. But how would we get the services to cooperate? E.g. rushlimbaugh.com not be too keen on sharing resources with cnn.com. :) And that begs the question, would the "rescuing" site be entitled for a fee for their failover support?
3. Profit!
2. ???
1. On Soviet Slashdot, a Beowulf cluster of alien Natalie Portman overlords welcomes YOU!
It was Larry Niven who predicted the idea of "flash crowds". Of course, he was envisioning physical crowds via teleportation, but the basic idea still holds. It's only going to get worse as more and more people use the net.
Look at it this way: in a primative society, a clan or village would usually have a storyteller or sage who gathered the news of the world in story form and re-told as appropriate. We should not be supprised that it takes millions, perhaps even hundreds of millions of people to be the story-tellers to 6 billion (that's a US billion).
If the Internet had a higher percentage of useful sites for news (not just talking jpeg-heads, but innovative ways of conveying the STORIES that the news represents), then no one of them would be loaded down and the backbones would be the only bottleneck. Notice that so many of us flocked to Slashdot when the towers fell? Wonder why? Because Slashdot, for good or ill, is our community's storyteller, and we instinctively come here to understand how our community is reacting.
Folks,
If you don't have a portable AM/FM radio, or even better a shortwave receiver, then get one TODAY. Get some spare battteries for it as well.
The simple fact is if you want to hear what's going on during a "major event" radio is the best way to do it. And you have evacuate in a hurry, you sure as hell aren't going to be taking your 60" flat screen TV with you. You want pictures, wait for the evening news, if you want to know what's going on NOW, get a radio.
Even better, get yourself licensed as a ham radio operator so you can be part of the communication solution if needed (yes, amateur radio is still important, even today).
You know you're a geek if you've ever replied to a tagline.
particularly those without access to a quality television news service.
Isn't that a oxymoron?
Higher Logics: where programming meets science.
I mean ok performance degraded on 11/9 but that was the only event I can think of that made the news sites shudder.
..Err, I believe that was the author's intent when he said the next "major world news event". Obviously, that doesn't have to be "the next time that planes crash into buildings in the US, but when in recent times has newsworthy incident happened? Or at least one that draws that level of coverage? The question was not "Can major news sites cover the England vs. Brazil soccer game this weekend?", it was "If a major world event were to happen in the near future, will the current news sites be able to handle it?" The only point of asking at all is the fact that as you even noted, the last time no, they unarguably did not.
;)
Do they have any obligation to serve under high load?
No. If you walk into my store, I have no obligation to sell to you. This becomes a matter of self-appointed corporate responsibility. When it really comes down to the wire, are you about providing the public with vital, up to date information, or are you about providing content to generate revenue? If many of the advertisers' links were slowing up (as was already posted somewhere above), you're not generating all that many more hits, and if they have to click the ad, forget it.
Do we even care? Maybe the radio is a better source of news sometimes, hell try CNN
Do you get cable at work? I don't. I don't have a radio either. This happened when most people were at work, getting ready for work, or on their way to work, most of them probably have internet access, but relatively few have access to cable. Radio is a possibility, but on average probably less ubiquitous in the work place than internet access.
--- What
During the horror of the attacks last year, I was surprised and thankful for CNN's approach which allowed them to withstand the barrage of hits:
They switched to an old-school, how-the-web-used-to-be, no-nonsense design. It was basic HTML, with some embedded pictures that contribued to the information. No frills, no ads, no sidebars about the latest crap-news, just the information we were looking for. Needless to say, it also ate a lot less bandwidth.
Of course, they were down part of the morning, but when they came back in the altered format, I thought it was a great move. A few other sites were doing the same thing, and I think they'll remember the technique for the next time something big goes down (hopefully something pleasant next time? I can hope...)
To state the obvious, the major news sites would have to have not only leaner pages, but also have the infrastructure to withstand a slashdotting-with-hair-on-it. Leaner, lower bandwidth web pages benefit every one, every day, but for daily needs the infrastructure is going to be expensive overkill.
In contrast, more of the tech sites were already used to heavy loads and I would guess that his brought in a larger than normal number of new and infrequent visitors. Maybe it was my imagination, but it seemed that after that many mainstream newspapers, magazines, and radio magazines started to carry more cutting edge tech info and topics and providing in a much more timely manner - days instead of weeks or months.
It would be interesting to map how much the coverage and timeliness of tech issues by the mainstream press changed, when it changed, and how much was related to being able stay on line.
Beta is broken and the link to classic doesn't work. Stop wasting our time or there won't be anybody left here.
I have web access here at work without ready access to cable news services on TV. I watched/read about the events of September 11th on the web.
If you have to point out that something is funny, the chances are it isn't funny.
Athletic Scholarships to universities make as much sense as academic scholarships to sports teams.
It isn't just the news sites we have to think about. We should also be asking, when the next big event does happen, will people even be able to get online to access the news sites?
I'm not talking about some sort of damage to the communications network. I'm talking about ISPs that enforce strict rules on how many of their customers can get online simultaneously. They are the real threat to the Net as a primary source of urgent information, and it's all about money. They take on millions of customers but total capacity is measured in tens of thousands.
For example, on September 11th there were a few hours when tall buildings in London and other British cities were being evacuated, but many people over here couldn't get online to access vital information because our ISPs have notoriously low capacity and only allow a small percentage of their customers online at any one time.
Obviously this is a greater threat in rural areas because the only available connection method is dial-up.
I am subscribed to a couple of worldwide mailing lists and I have found that email simply rocks in high 'net traffic situations.
During the New York tragedy, much of the traffic on those lists was along the lines of "I can't get to the major sites because the web is clagged solid - can anyone tell me the latest?". And thankfully for a couple of days, the rules about straying from the topic of the mailing list were ignored.
Granted, many of the complaints were actually related to individual corporate firewalls, http gateways and proxy servers, rather than the sites themselves, but the situation stands: for whatever reason, you can't get to the site. Our web proxy fell over under the load, but our SMTP gateway just kept on going. And so did most others around the world. And I imagine that NNTP stuff worked just as well the SMTP stuff.
Remember folks, the Internet is a lot more than the Web!
Ahh - My eye!
The doctor said I'm not supposed to get Slashdot in it!
Multicast news services worked well during 9/11 and there is no reason to think that they won't the next time. Multicast is specifically designed not to "melt down" under extreme changes in audience.
The trouble is that not everyone is multicast enabled, but this shows real promise in handling news and emergency information over the Internet.
one of the passengers on one of the planes that came down on 9/11 (it was the one that crashed in the field, IIRC) was a founder of Akamai Networks, one of the load sharing/distribution companies that allow bandwidth to scale according to demand. As his plane came down, his company was entering one of the most demanding days in its history, as more people were targeting news sites at once than ever before.
It's organisations like that which will assist in the next big news item.
-- james
Slashdot seems to be the Gray Hat QA engineer in testing concurrent site capacity. Maybe it should get a salary and benefits....
moto411.com
Ever since 9/11, I've noticed that the heavily-trafficked sites cope with sudden floods of hits by switching over to static pages with minimal graphics. The NY Times, for instance, did this when the AA flight went down in Queens last November. CNN's done it a couple of times as well.
When we're looking at scale, though, it's useful for us to remember that these sites can handle way more traffic than even the typical slashdotting can deliver. Most breaking major news can be handled by them with only a little bit of slowdown. It's only the 9/11-scale events that can really bring the news sites to their knees - so lets hope that we don't have to see anything that brings on a overload scenario for the big news sites.
The other thing to consider is that most of the news providers are still investing some money in their infrastructure - just less than before. It's very well possible that a 9/11-scale event might not hammer the servers the way they were hammered last year. A lot of web sysadmins learned valuable lessons that day that I'm sure have been applied since then.
-- Josh Turiel
"2. Do not eat iPod Shuffle."
... the headline reads, "Internet knocked out by multi-city EMP attack"
It was Princess Di's death. I was on shift the night it happened and it pretty much brought all news websites to their knees. That was the first time I noticed the low bandwidth version of CNN. At first I thought the site was choking because it looked like some graphics were not loading.
Still, I'll give it to Slashdot and to IRC. I spent most of 9/11 on IRC transcribing what was being reported on CNN, since for a while the site was pretty much useless. A bunch of us where also taking screen captures and posting them online so people could see the horror. I still have captures of the first flyover of the Pentagon, which is less than 10 miles from my office.
Pedro
----
The Insomniac Coder
When there are big world events, the amount of net traffic does increase overall, but not hugely, as instead of wasting time reading/working, we all go and look at news sites instead.
One way around this problem is bandwidth insurance. What is this? Large groups of averagely popular websites all get their bandwidth from certain sources. When there's a sudden move in traffic, those really big providers can simply deallocate the bandwidth from gardening.com and reallocate it to the BBC .
I might be talking out of my ass here, as the BBC already has peering agreements with Telehouse etc it's so big. Alternatively ISPs could implement decent caching systems. Otherwise, FreeNet released 0.5rc1 earlier :)
When 9/11 hit, the first thing I did was wget about two dozen news sites and thousands of blogs immediately. CNN, in particular, got blacked out really, really hard, and was reduced to one image on the front. I wish I had my archives available to post but they're rather deeply gzipped ...somewhere. =)
Akamai had their work cut out for them that day, I can tell you. I was lucky. I called out sick.
But none of this really answers the question -- how do you cover your butt and insure that you keep getting a news feed when/if you need it? I noticed that when I go to www.php.com, it's quite slow. So I started using uk.php.net and it zips right along. The moral of this story is that you might want to find 3-5 news sites that you consider good (and a factor in this probably should be how fast news gets to their site), then find some printer-friendly version/low bandwidth links to their front pages. Those are far less likely to be used when things get crazy. Drop some admins an email, perhaps certain versions of their site is located on entirely seperate servers and might go unscathed during a 9/11-ish rerun.
My
Limekiller
I'm far to cynical for my own good. My first thought was "no they won't, they'll add more to get the increased revenue".
Maran
the Internet is going to be the primary news source for many millions of people, particularly those without access to a quality television news service. How will / can it cope?
Huh?? I'm not sure what you're talking about but I'm pretty sure I don't recognize it. Quality television service is much more widespread than the Internet. I'm N. American, but I've lived for years in Africa and Asia. I can assure you that in "None of the above" has the web surpassed broadcast media as a source of news for any but an elite few. And the comment is irrelevant for the elite since they have access to "all of the above"-plus.
Seriously, even in the smallest, poorest villages around the world several people will have radios and access to VOA, BBC, a national broadcast network and one or two regional stations. In addition most villages will have at least one television.
The internet is a bit player if it's a player at all
Yes.
I contacted them all and they said they're ready.
On Sept 11th (and you all know what happened there, save the ribbons for a different soapbox), I used the internet as my primary source for what was happening. Somebody here had a radio, and the news channels were spouting lie after lie, rumours on air, digging up unchecked sources, because that's what the mainstream media does.
I, instead, got my news from "switchboard" type sites (/., drudge and a few forum sites), keeping an eye on who was up, mirroring important pages, and basically exchanging as much info as possible. It lagged a bit...I was 10 minutes out of the loop when the tower fell, for example...but I also wasn't supplied rumours like "there are nukes in the air" or "A fifth plane is on its way to chicago."
By the way, BBC had amazing realtime coverage plus rm video that stayed online pretty well. NYTimes was slow as hell. CNN got swamped, as did MSNBC.
Hey freaks: now you're ju
A spokesman for CNN.com said "after talking to several vendors including Sun, IBM and Microsoft, hotnakedteens.com won the business by showing they routinely handle traffic 10 times the traffic we received around Sept 11."
Read reviews of shopping cart software
The internet is not most people's main source of news.
But it is most people's main source of news _during the working day_. If an event happens during the evening people are most likely to tune to their TV. But if it happens during the day they are more likely to use the internet, simply because that's more convenient. And as you say, they want real time audio and video - from the internet if it's during the day.
There will always be bottlenecks, simply because it doesn't make economic sense to plan for such rare events. But as traffic in general grows on the internet, available bandwidth/server capacity will grow to meet the average demand (including pictures). This should make it easier for news sites to cope with peaks in demand by switching to low graphics formats.
CNN and others provide email alerts for breaking news (which notified me of 911), the web then provides initial reports, then we switch the TV on and get realtime news as the web grinds to a halt.
Though if the next major event happens on the same day as a game demo or a new Matrix trailer are released, we're truly stuffed...
So we have questions about bandwidth, okay -- but we also have questions about how and whether television and newspaper editorial process might break down in trying to get "instant" stories up on a Web site. A process set up to approve stories for tomorrow's paper doesn't necessarily apply to stories that need to go up now. (My two local dailies have really felt their way with that, too.)
particularly those without access to a quality television news service.
Okay, I'll bite... What quality television news service? Gotta get me some of that action. You must not be viewing the local sludge we get here, with the jocular anchors' repartee and all...
I've seen one U.S. "news" program -- Dateline, maybe? -- ask a scant few questions about the preparedness of New York's emergency Fire and Police responses, mentioning specifically the failure to improve the same communications gear that had failed in the earlier WTC attacks. The show mentioning those problems in passing, almost rhetorically -- "Some people wonder..." was the tone. (Apparently the TV network didn't wonder itself. Only some vague "critics" -- that's the tone I mean.) The New York Times published an article about those same problems, around a full year later if I remember right -- and the article's theme was "Why isn't anyone asking these questions?"
If we had quality "news" on TV, the shows would be investigating controversial events, not just... what, commemorating momentous ones? Journalism is about intelligent enquiry. If you had to choose between "intelligent enquiry" and "advocacy" in describing the Fox "News" Network, which would you choose? That network is about reinforcing people's political leanings, not reporting the news. No thanks.
"Fundamentalism" isn't about divine morality. It's about human authority.
But should a major world event take place in the coming months/years, the Internet is going to be the primary news source for many millions of people, particularly those without access to a quality television news service.
Please be advised that your set needs adjusting... It's pretty clear from the evidence (and from a phenomenological point of view if you observe your own reactions) that the experience of watching a major event on television as it unfolds barely qualifies as useful information, due in part to the nature of the medium, but largely due to the nature of media filters and techniques. When you see something like 9/11 going on, it's much closer to entertainment, unfortunately, than providing one with reconnaisance leading to rational behaviour. The drama of the moment helps you develop powerful emotions in relation to the event, but what kind of info do you really get?
When it comes to war, TV obscures. For instance, see this study on media and the gulf war. [Remember that? Oh wait, it's still happening.] A salient quote:
In other words, you'd actually be better off combing through usenet than sucking on the immediacy of the glass teat.
Qualifier: I've worked in media-democracy-oriented film/video for years, I'm involved and devoted to the medium!
Damn those pesky terrorists
If multicast were ubiquitous then things could have been much better. If people could received the html only web page and turn to the mbone or some other multicast network for the streaming video then the net could probably shrugged almost any event off. Since porn is one of the few things that makes money on the net I am suprised that multicast for streaming smut hasn't become more prevelant.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
William LeFebvre (of CNN.com) has an excellent talk that he's given at least twice at Usenix events (I saw it at Usenix '02, he also gave it at LISA '01), that gives a lot of detail as to the things that had to be coped with to keep CNN.com running on September 11. I can't find the full-text of the speech anywhere online, but there's some information at this site that at least gives you an idea. Interesting stuff!
A local SAGE chapter had the senior sysadmin for CNN come in to give a presentation on managing a large webserver farm. I remember the admin said their weekly staff meetings frequently discussed the answer to the question "What if the president declares war this week?" and the servers' readiness for the load, projected from the traffic they received during Desert Storm. In general, I seem to recall their main strategy revolved around scalable or easily-expanded network connections to the data center, and a large pool of servers used as a testbed and development set that could be switched over to production use (I believe they were using a round-robin DNS strategy similar to Netscape's ftp server system in Netscape's early days.
I atttended this presentation, so while the description above is first-hand, my memory of the details may well have dimmed with time.
So tell me again what possible motivation the news industry has for upgrading their online capabilities?
Are any of them even making a penny on their websites? So why pour more money into upgrades? What's the reward? So they can pay more for bandwidth and lose more money?
I was at work, away from a television on September 11th I heard vague news of a plane crash on the radio. I logged in for details:
msnbc.com - down
cnn.com - down
cbsnews.com - down
abcnews.com - down
drudgereport.com - down
I turned the radio back on. Yep. Still works.
Why? Because radio can charge enough for ad space to pay for a working transmitter and a studio and a full-time staff. Cable news makes enough money to support their operation as well.
But online news, for the most part, loses money, and thus can exist only as an offshoot from an offline operation like a TV news broadcast or a newspaper. Therefore it winds up acting only as 1) a supplement and a promotional tool for the broadcast or publication 2) a reader feedback time-waster.
It's always this way; follow the money and you get your answer. And right now the answer is none of the online operations have the desire or motivation to be "the" online news source when the next 9/11 breaks. Let the site go down. Who cares?
Phallic Symbols in LOTR
A far cry from the days of Walter Cronkite.
You mean back when we had exactly 22 minutes of world news for the entire day?
You mean before the days of 24 hour news channels? And 24 hour Headline News channels?
You mean before the days of live congressional coverage via C-Span?
And are you aware that virtually every TV network went commercial-free during 9/11 coverage?
TV news deserves its criticism, for sure... but be fair. And don't pretend there was this golden age of news when reporters and newscasters worked for free because of an altruistic love of the truth. They've always been under pressure to make the news presentable, entertaining, to package it for consumption. If they don't we stop watching. But there are a HELL of a lot more TV news resources now than there were then.
Phallic Symbols in LOTR
CNN HEADLINE NEWS, baby.
From the AP:
CNN Looks to Get Hip, Think Young
Wed Oct 2, 5:02 PM ET
NEW YORK (AP) - Is CNN Headline News down with it?
The cable network is trying, judging from an effort emanating from its executive suite to think young.
CNN Headline News general manager Rolando Santos told the San Francisco Chronicle this week that he's looking to mix 'the lingo of our people' -- words like 'whack' and 'ill' -- into newscasts to attract young people.
And the New York Daily News on Wednesday quoted from an e-mail sent by a network manager to his headline writers, sending them a copy of a slang dictionary so they can be 'as cutting edge' as possible.
'Please use this guide to help all you homeys and honeys add a new flava to your tickers and dekkos,' the message said, referring to graphics on the Headline News screen.
The list of phrases included 'fly,' meaning sexually attractive.
Santos said Thursday that the e-mail was designed to point out resources that might help headline writers.
'The e-mail was informational, not a policy or directive from me,' Santos said. 'With that said, I should point out that I want the language used in our tickers and dekkos to be real, current and relevant to the people who watch us.'
CNN underwent a makeover a year ago to add busy graphics to make its screen look like a computer screen. Its ratings have been improving among young viewers.
--------------------
Eh, maybe that wasn't such a great example. "Yo, that suicide bombing is wack!"
Phallic Symbols in LOTR
I've since built some even larger systems; I've no doubt that it's possible to scale Internet streaming media distribution to millions or even tens of millions of simultaneous viewers using today's technology and protocols.
ellbee
You can't fight in here - this is the war room!
If I had to design a site that had to stay up during a world event, I'd try to talk to the people that were able to keep there site up(mostly) during the last world event. At least that would give me some ideas to work with.
Ha, I did the whole post about world events and didn't mention 9/11 once!. . . D'oh
The Kruger Dunning explains most post on
On Sept 11 I woke up to NPR's description of what happened. (8:00 PST, so after everything was over) Then, I turned on the TV to CNN. Soon, I had to go to work. I quickly found out that ALL the news sites had crumbled under the load (except slashdot). So, I had to turn to an alternative source. I turned to ShoutCast. There were dozens of broadcasts that had switched from music to a feed from CNN, I was really impressed.
I heard a stat that the internet traffic had quadrupled that day, but that hits had stayed the same. Many people complained that the internet had "failed", but we all know that only a few sites had failed, the internet as a whole behaved beautifully.
Travis
P.S. MSNBC also has automatic triggers that remove the graphics from the site when the load gets high.
Damn, all of us in the US are screwed!
Read the EFF's Fair Use FAQ
Let's just say, they are well aware of the issues, and a lot of thinking and planning has gone into how they handle the load of major news events.
-me
Love many, trust a few, do harm to none.
Seriously though, a major distributed system could handle a massive load. Maybe that's what news.google.com is about. If the news sites all mirrored the same content (which they pretty much did on sept 11th anyway and do in most major events) they could probably handle the traffic increase between all of them.
As for people using the net versus TV, it happens because TV doesn't provide as much information as people want sometimes. Websites often link to additional info that TV won't cover as it's time to repeat the same report in 5 minutes.
Oh and I thought the net coped pretty well with the last event. Phones were down all day but my b/f in NYC was able to call me in San Francisco using dialpad and keep a connection long enough to wake me up and let me know what was going on.
Introducing the new Occam Fusion! Now with sqrt(-1) fewer blades!
Unless it's on slashdot, and then there's bound to be 17 moderators who mod it down for the hell of it before people start realizing it on their own.
To put in a plug (though I no longer have any ownership interest in it) one of the things about
ClariNet is that it can handle any load.
Because it feeds out news in USENET format from the major wire services, the load is placed on the local server. ClariNet's servers never even feel increased demand. Even highly saturated internet pipes would only slow things slightly, USENET doesn't care about the latency of the pipe.
And all this using 20 year old technology, oddly enough. People always talk about the news sites failing during things like the Olympics, Sept. 11 etc. but the distributed technology never has that problem.
On top of that, USENET is designed for serial news, so that it shows you what's new. You don't have to sit there constantly refereshing a page to see if there is new material, you only see the new material. We even had a system so that urgent stories could be fed directly to your screen, and it's not a polling style of "push" like PointCast was.
Generally the newsreader is, surprise surprise, a great way to read news. What surprises me is that all these years later -- ClariNet was the first of the dot-com companies -- nobody has done the same. I sold it 5 years ago, but it's still running, if a bit shrunk from the economy.
Has it been over a year since you last donated to the Electronic Frontier Foundation
And yet, even in the throes of 9/11, Slashdot itself fared better than CNN, MSNBC, Foxnews, and others.
The great thing about the internet generally is the ease with which content can be ripped off. If I get into a news site, I can easily mirror / cut and paste into a Yahoo chat room, onto /. etc...
/. was a damn good source of news when all the news sites were knackered. If slash-knitting, slash-boarder and slash-hump all join in on the act that takes a decent chunk of the population away from the big news sources - freeing them up for others.
As the use of the internet develops more granularity - i.e. people spend more time in smaller groups, not all huddling around google, bbc and yahoo, this will become a viable route for this 'big news' to get through.
About a year ago
Then your just waiting for the whole net to crumple under the load of a hundred million people IMing each other with "do you have any new news???"
The BBC took the load just fine, and now take a larger load daily!
--
Internet Explorer (n): Another bug -- that is, a feature that can't be turned off -- in Windows.
Akamai (and its competitors, AT&T and Speedera) have a business model that says they put lots of caching servers out in the network and sell caching to the web content providers who want to get their content out. By contrast, the original web caching was a pull model - businesses with firewalls and some ISPs use either transparent or explicit-proxy-based caching to cache *incoming* content at their gateways or other concentration points (e.g. cable modem network head ends), and they cover the cost of the caching equipment by reducing their bandwidth needs as well as by giving users service that's perceived to be better. Flooding networks like Usenet are good for non-realtime multicast-like behaviour, and multicast is good for streaming but could also be integrated with caching systems. Back during the Internet boom, there were several companies such as I-Beam that used satellite broadcasting to push content out to caching servers, but alas, Chapter 11 has eaten most of them.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
I probably shouldn't be starting a new thread this late, but I can't see an appropriate place to post this.
I was a sys-admin for a non-US news website during September 11. Basically we were hit hard, mainly because of the time-zones. Because the organization I worked for (I don't name them, but you should be able to figure it out if you know me) mainly produces old-media news, and re-purposes most of that for the Internet. Because of the time-zone difference, most of the action was over by the time I got into work.
Apparently, the late TV news shift had just finished the last broadcast for the night and was heading home when the news broke. They turned around and stayed on deck until the morning shift came in to take over.
We received a weeks worth of hits in less then 24 hours. Our load-balanced redundant web servers were purring along, not quite maxed out but very little room to spare.
The biggest killer was bandwidth. Looking at the bandwidth stats our international PVC (about 1/4 of our capacity), maxed out early in the morning, and was taken down briefly twice to increase its share of the total bandwidth.
In the end, we reached over 80% utilisation of our total pipe. This may not sound much, but at the time we had never used more then 40% of what we had available that day. I think the only reason we didn't go any higher was that something upstream was maxed out. My guess is the US link out of my country.
Several steps were taken to improve the performance of the website during the day. The main page was replaced by a news summary with a link to the old main-page. Most people only wanted the latest news on New York, so they could get that without hitting the rest of the content. We had to fine-tune the web servers a few time, and I've already mentioned the tuning to increase the share of international traffic.
HTTP was not the bandwidth killer. Because of the extended news coverage, the video from the news studio was streamed directly onto the Internet. Usually we use static video files or live stream specific shows, but the video stream was on for something like 12 hours, and that killed our pipe. I presume the streamed radio stations were also popular, but I haven't seen the statistics on that.
We survived, just. The biggest problem was that we were not ready, and that we had to react. If we were fully ready, or we could have reacted more quickly, then we would have done much better. The trouble was that these events happened during the night our time, and the staff on at that stage didn't know that there were things that the day staff could have done to help the load problems.
I don't think that you can expect a news site to be able to fully deal with an event like September 11. You can't justify having 10 times the bandwidth you normally use, just for a once in a decade event. You have to aim for the once-a year event, and try to deal with the other cases as best you can.
Democracy isn't about no one telling you what to do. It's about everyone telling you what to do.
Way back during the Gulf War, I was playing on a MUD (Dartmud) and was getting scud raid news from a guild member based in Israel before the news about it came on (usually just before or after he headed to or came back from the bomb shelter). I also usually knew more about what was going on than my roommates, who were watching the war on TV.
So yes, sometimes the internet is better than TV, and no, I don't need to talk about what's on TV, as they sometimes don't really know what's going on, either.
The fact that the parent is moderated "Insightful" proves my point I think...