Slashdot Mirror
Blackout Worse For Internet Than Previously Thought?
An anonymous reader writes "Renesys (the people who previously brought you cool animated graphs of the US/Canada power outage has a new report out. It challenges the widely
held belief that the Internet was largely unaffected by the power outage. Lots of important networks lost connectivity, including banks, hospitals, government organizations and investment funds. There's a cool appendix on the huge Italian power outage in September as well. They conclude that the Internet is not ready to be critical infrastructure."
It has always seemed to me that the internet isn't all that de-centralized, but a few major companies ran most of the backbones. Since it isn't a huge ad-hoc network, most of the data for an area probably goes out through no more than 5 connections. Especially in rual areas, I wouldn't doubt that at least one routing station in each of those chains doesn't have good long term backup facilities.
Slashdot - the place where you can look like a genius by restating the obvious
Ready or not, the internet is increasingly being used for critical infrastructure. At best, failures like the power outage should motivate governments and industry to bolster the internet up to where it needs to be for reliability standards.
But seemingly no less so than the power grid.
-Peter
Exactly how does one system's dependance on a critical infrastructure (the power grid) and it's failure when that infrastructure fails imply that it's not ready?
Bah, I could have told you that. I work for an ISP that serves 15 states. I get calls from people who put 100% of their business into a DSL line - with no backup to other carriers or mediums. When a hardware failure or trunk line failure occures - they go postal.
Sorry, but uptime is not 100% never was, never will be - plan for it, or deal with it when your connection goes down.
Even though we have multiple connections to the backbone - local trunks can go down. Aka backhoe attacks on burried fiber, or dove hunters blasting pole run fiber (don't laugh - it happened last week). If you don't have a backup DSL,ISDN, or heck even dialup connection for your business - then stfu and wait while we repair.
And don't even get me started on residential accounts that call in 'I use this for work I need it up now - send someone out today.' And it's Sunday evening... no - you didn't pay for a business account, so you get residential service levels which include 24-72 hour turn around on repairs.
If power *is* a critical infrastructure, and lack of power is what caused these problems, how can that support a conclusion that the Internet is not ready to be considered critical?
I'm not saying there isn't other evidence that would support such a conclusion, but the real failure here was the power infrastructure, upon which the net relied "critically" in the first place...
I lost slashdot for a day. I almost had to commit a suicide to relieve the pain.
The reason everybody said that the internet survived was that they were able to visit most of the sites they cared about during the blackout. The chart seems to show that many links and servers were down (presumably without power) during the blackout (including some major components of the internet), yet most people basically unaffected. This seems to suggest that as long as the server itself isn't in the middle of a blackout, the Internet can survive rather well. How many of your learned about the blackout from Slashdot or some other online news source?
I read the internet for the articles.
Air Canada lost it's reservations/bookings/everything servers, and couldn't operate anything approaching normally for one reason. The servers were based in the midst of the blackout.
Out here on the left coast, there were no effects. So why, don't international org.s and government departments have duplicate facilities on independant grids? That's always bugged me.
bwh
I live in denmark and recently we had a blackout that lasted maybe 10 hours.
While I was unable to make any phone calls, I could get on the internet with GPRS and surf to our server with my laptop for as long as the laptop batteries lasted.
The server is hosted in a colo datacenter which was also in the middle of the affected area. We run a mud on the server, and most of the players are from USA. They never discovered the blackout as the datacenter went on emergency diesel backup and apparently knew to make business with backbone providers that also knew their stuff.
So to the people saying that internet can only route around blackout areas but not _through_ them, this is not true. Seems at least here in denmark all the infrastructure on the backbones got backup power and just keeps working when everyone else is busy lighting candles.
The people affected by the downed routers were people who were in the blackout and couldn't turn on their computers anyway, so it doesn't matter that those machines were down. People outside the blackout were able to route around it, and THAT is the relevant part of the statement that the internet did well during the blackout.
Don't label something "offtopic" unless you know the topic well enough to tell what's on topic.
Well, look at it this way... they say "an UPS is good enough, if power goes out it will go out a few seconds or maybe a half hour", and don't plan for a "worse-case" scenario, in which you have a few hours of "power outage"... so instead of saving everything, commiting caches and so on, they just keep on hoping "in a few seconds power will be back on"... I just hope they DID learn their lesson now, and cut back on cutbacks (lol).
By reading this signature you agree to not disagree with the post you just read.
I work in a hospital in Toronto. There were almost NO facilities or services that functioned in the early parts of the blackout. Would you claim phones are a critical infrastructure? It's true that they worked during the power outage, but very quickly all the phone networks were too congested to provide service - this lasted for several hours. Radio stations continued to broadcast until their backups ran out and we were left with dead air. Thankfully, the hospitals had sufficient emergency generation to support several days without external power, but I wonder how could such a heavy power consumer as the internet rely on backup? It is really a question of "how many other essential services require internet connectivity during a blackout", because every citizen surely doesn't need it right now.
Internet outages when they start putting high speed internet on power lines...
--- Grow a pair, liberals... stop letting the Republicans bully you!
The vast majority of the networks that went dark were 24-bit in size. That is generally either small to medium businesses or home office, or a division of a larger business. I think we can all agree that outages at that level, though undesired, are not the end of the world. Small outfits and home office workers can afford the down time in the case of a general crisis (ie the buses aren't running, either, so go have a coffee and read the WSJ) and 4-8 hour outages on their DSL are not uncommon either. I know that is the case where I work, and we have a global presence too.
We invested in a very large portable battery backup system for our server room back when California was having its own blackouts. The stack would probably stay up an hour or so, which we figure is enough to manage most blackouts nicely, and anything longer than that is a "major cockup" that we need to wait out. But if we go down who will care? Just us, and not all that much.
I think that the general expectation regarding the internet is not that it will stay up 100% in a crisis, but that it will continue to operate in cells of functionality during most kinds of disaster, then recover quickly on its own as soon as it can built remote connections again. Compare that to the electric grid, where most or all cells of function were sucked empty and driven into the ground when the grid dried up, and engineers spent days coordinating their recovery so that the first cell to go online didn't feed the entire electric grid on its own. Tricky stuff.
TCP/IP is built to understand rolling outages and uncoordinated recovery. The electric grid still is not. That, I would submit, is the main issue and not that routers on the edge of small networks didn't have generator backup.
=^..^= all your rodent are belong to us
If power *is* a critical infrastructure, and lack of power is what caused these problems, how can that support a conclusion that the Internet is not ready to be considered critical?
Because the internet is communication, not power. So the correct comparison is the telephone company, not the power company.
Power can be backed up locally. Communication can not. So power only needs to be available MOST of the time, with backups on any critical services, to achieve its "critical infrastructure" level of reliability. Communications, on the other hand, requires an infrastructure with multiple links, routing around failures, and local power backup at the active nodes to achieve its own "critical infrastructure" service levels.
The telephone company HAS this level of backup power built in. Switching centers, for instance, run their equipment directly from TWO banks of 48v batteries suitable for days of operation, and run battery chargers continuously when there's power available. Repeaters on long copper trunks are powered from the endpoints - and can run with only one endpoint hot. Telephone instruments are powered from the central office switch via the copper wire. Active customer premesis equipment has battery backup for critical features or is designed to connect at least one POTS phone directly to a copper pair to the switch in case of blackout, and so on. SONET nodes are wired as rings rather than trees, so you have to cut TWO fibers in different places to isolate them. Other trunks are redundant and switch over automatically in case of outage. I could go on. About the only place a single cable cut can cut you off is the line to your house - and if you pay (a lot!) extra (as some businesses do) you can get another run in by a different path, so no single backhoe or downed pole can isolate you.
The Internet was ORIGINALLY designed with this kind of redundancy built in. Individual links were via the tellco's infrastructure, with its power-failure resistance. Routing was automatic, and would find a route between any two nodes if one still existed. (It WAS designed by people who were at least THINKING about surviving a nuclear attack, after all.)
But with the "inflation" of the commercial internet this robustness was lost. The explosion of active IP addresses made routing tables impossibly large, while most sites were connected via a local ISP rather than ad-hoc connection to two (or more) internet neighbors.
So the internet split into a "backbone" with SOME of the old routing redundancy, interconnecting ISPs, who in turn give you a default route JUST to their own servers. If your ISP fails you're cut off, and if the backbone connections to your ISP fail, ditto (even if you in principle COULD reach the rest of the net through somebody with a two-ISP feed.)
The ISP buisness has FIERCE price competition, and one BIG way to cut costs is to reduce redundant routing internally and neglect backup power.
At the backbone level the long-haul networks carrying the data had an even FIERCER price war, due to the excessive long-haul buildout of the internet bubble. Perhaps some of the upstarts powered their switches and repeaters with local power (on the assumption that the could slough any site that had a local power failure and that they'd have a path with all equipment powered between any two customers still live). A major blackout would violate that assumption, cutting off not just the dead area but others who could only reach the rest of the net by routing through it.
How about your DSL or cable IP feed? Did your cable company include battery backup power in the repeaters, pole-mounted routers, and fiber/cable bridges? Is you settop box battery backed up? How about your DSL modem? If you're corporate, are all your routers, your VoIP bridge, and any desktops running a softphone on the UPS? Do your SIP phones run if the power fails? (Home users ditto for your PC.)
Until all these are fixed the internet is NOT running at "critical infrastructure" reliability levels. So you'll want to think VERY CAREFULLY before disconnecting your POTS line and depending on Internet-VoIP. B-)
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
From having been around the Internet for the last 15 years now..
The Internet was a lot MORE capable of being infrastructure, before *.com happened. Since it has been commercialized, the backbones have become more and more important, and routing/re-routing less and less important.
"Error: No Route To Host" at one point in history, literally meant that the computer directly connecting the computer you were trying to reach was offline. Now, "No Route To Host" means that there was a power failure somewhere in the world that just happened to be in the way of your provider routing through a few other providers, or that a janitor somewhere kicked out a plug in Minnesota, while you were trying to connect from Michigan to Texas.
The system used to be able to route around virtually ANY connectivity issue. Now, it can't route it's way out of a wet paper bag.
"Champagne for my real friends - and real pain for my sham friends!" http://ericblade.postalboard.com/
Newsflash: the internet is already critical infrastructure, and the power grid that failed is critical infrastructure and has been for the better part of a Century.
If you're saying that lack of failure defines whether something is critical or ready to be critical then I guess by that definition the electrical distribution grid isn't ready to be critical infrastructure. That is preposterous because it is and manages quite nicely for the most part. The rest is down to cost benefit.
I live and work in the NYC metro area, and was at work when the blackout started. I didn't notice there was a blackout until I walked outside and saw our generators on. For the record, I work for a company that provides services to large internet datacenters. Any datacenter worth its monthly fee wasn't affected by the power outage. Yes, individual institutions including banks etc etc who weren't prepared did lose connectivity, but backbone providers and large carrier centers in the area didn't skip a beat.
A slip of the foot you may soon recover, but a slip of the tongue you may never get over. -Benjamin Franklin
The data they present indicates that the blackout had a severe regional impact. I see nothing that shows that there was a significant global impact (meaning that I can't get data from AS 12374 to AS 553, for example).
n dex.html
y s-030502 -NRC-911.pdf
That's correct. In fact, our data showed that it clearly did _not_ have global impact. (Compare with various worm events, which do generally have global impact: http://www.renesys.com/projects/bgp_instability/i
cod red ii and nimda report)
The WTC collapse probably had more impact on global routing (some large carriers had primary and backup equipment in both basements).
Actually, it did not. It did affect some regions outside the US that had trans-Atlantic connectivity straight into NYC, but otherwise it was geographically well localized. This report (PDF slides) compares it to Code Red and Nimda:
http://www.renesys.com/projects/911/renes
9/11 report
The internet impresses me every day. Its the ridiculous expectations of people that blow my mind.
When the world trade center went down, I worked at a major ISP. Verizon is right next door to the WTC, not surprisingly all the main trunks were destroyed. Connectivity for much of the Atlantic including Europe was disrupted. Many carriers had cell towers on top of the building as well. Even from California I didn't need to be told the internet was going to be f#@*ed up on the east. Yet some how people in NYC who had to travel to NJ to find a working phone would call me and ask why their DSL was down.
The fact that the whole north east had no power, and the majority of the internet worked shows the internet has done a very good job of doing just what its designed for. It could do a better job. So lets work on that instead of just talking about it.
They conclude that the Internet is not ready to be critical infrastructure.
Huh? It would seem to me that the fucking power grid is not yet ready to be critical infrastructure but hey, here we are. Shit. There is nothing in the world (except for the sun, oceans, etc.) that is 100.00000% dependable.
Our top story tonight: humans, human inventions imperfect. Tomorrow: sky blue, water wet.
Dear Slashdot: next time you want to mess with the site, add a rich-text editor for comments.
> They conclude that the Internet is not ready to be critical infrastructure.
So. They conclude that the internet is supposed to be more reliable than the power grid...or else it's not ready for prime-time?
Sounds like they're setting their standards a bit too high.
Really? It already is a mission critical infrastructure for my company and most others, I suspect. When some idiot with a backhoe takes the region down for a few hours, we're in serious doo-doo (no second carrier where I am). We switch to ye ole spreadsheet as a backup, but we're crippled without Internet access.
I agree with the article - there are some serious architectural flaws that need to be addressed; however, fact of the matter is that the internet has already become a mission critical technology despite these shortcomings.