Slashdot Mirror
Blackout Worse For Internet Than Previously Thought?
An anonymous reader writes "Renesys (the people who previously brought you cool animated graphs of the US/Canada power outage has a new report out. It challenges the widely
held belief that the Internet was largely unaffected by the power outage. Lots of important networks lost connectivity, including banks, hospitals, government organizations and investment funds. There's a cool appendix on the huge Italian power outage in September as well. They conclude that the Internet is not ready to be critical infrastructure."
It has always seemed to me that the internet isn't all that de-centralized, but a few major companies ran most of the backbones. Since it isn't a huge ad-hoc network, most of the data for an area probably goes out through no more than 5 connections. Especially in rual areas, I wouldn't doubt that at least one routing station in each of those chains doesn't have good long term backup facilities.
Slashdot - the place where you can look like a genius by restating the obvious
Ready or not, the internet is increasingly being used for critical infrastructure. At best, failures like the power outage should motivate governments and industry to bolster the internet up to where it needs to be for reliability standards.
But seemingly no less so than the power grid.
-Peter
Exactly how does one system's dependance on a critical infrastructure (the power grid) and it's failure when that infrastructure fails imply that it's not ready?
Bah, I could have told you that. I work for an ISP that serves 15 states. I get calls from people who put 100% of their business into a DSL line - with no backup to other carriers or mediums. When a hardware failure or trunk line failure occures - they go postal.
Sorry, but uptime is not 100% never was, never will be - plan for it, or deal with it when your connection goes down.
Even though we have multiple connections to the backbone - local trunks can go down. Aka backhoe attacks on burried fiber, or dove hunters blasting pole run fiber (don't laugh - it happened last week). If you don't have a backup DSL,ISDN, or heck even dialup connection for your business - then stfu and wait while we repair.
And don't even get me started on residential accounts that call in 'I use this for work I need it up now - send someone out today.' And it's Sunday evening... no - you didn't pay for a business account, so you get residential service levels which include 24-72 hour turn around on repairs.
If power *is* a critical infrastructure, and lack of power is what caused these problems, how can that support a conclusion that the Internet is not ready to be considered critical?
I'm not saying there isn't other evidence that would support such a conclusion, but the real failure here was the power infrastructure, upon which the net relied "critically" in the first place...
I lost slashdot for a day. I almost had to commit a suicide to relieve the pain.
The reason everybody said that the internet survived was that they were able to visit most of the sites they cared about during the blackout. The chart seems to show that many links and servers were down (presumably without power) during the blackout (including some major components of the internet), yet most people basically unaffected. This seems to suggest that as long as the server itself isn't in the middle of a blackout, the Internet can survive rather well. How many of your learned about the blackout from Slashdot or some other online news source?
I read the internet for the articles.
Air Canada lost it's reservations/bookings/everything servers, and couldn't operate anything approaching normally for one reason. The servers were based in the midst of the blackout.
Out here on the left coast, there were no effects. So why, don't international org.s and government departments have duplicate facilities on independant grids? That's always bugged me.
bwh
I live in denmark and recently we had a blackout that lasted maybe 10 hours.
While I was unable to make any phone calls, I could get on the internet with GPRS and surf to our server with my laptop for as long as the laptop batteries lasted.
The server is hosted in a colo datacenter which was also in the middle of the affected area. We run a mud on the server, and most of the players are from USA. They never discovered the blackout as the datacenter went on emergency diesel backup and apparently knew to make business with backbone providers that also knew their stuff.
So to the people saying that internet can only route around blackout areas but not _through_ them, this is not true. Seems at least here in denmark all the infrastructure on the backbones got backup power and just keeps working when everyone else is busy lighting candles.
The vast majority of the networks that went dark were 24-bit in size. That is generally either small to medium businesses or home office, or a division of a larger business. I think we can all agree that outages at that level, though undesired, are not the end of the world. Small outfits and home office workers can afford the down time in the case of a general crisis (ie the buses aren't running, either, so go have a coffee and read the WSJ) and 4-8 hour outages on their DSL are not uncommon either. I know that is the case where I work, and we have a global presence too.
We invested in a very large portable battery backup system for our server room back when California was having its own blackouts. The stack would probably stay up an hour or so, which we figure is enough to manage most blackouts nicely, and anything longer than that is a "major cockup" that we need to wait out. But if we go down who will care? Just us, and not all that much.
I think that the general expectation regarding the internet is not that it will stay up 100% in a crisis, but that it will continue to operate in cells of functionality during most kinds of disaster, then recover quickly on its own as soon as it can built remote connections again. Compare that to the electric grid, where most or all cells of function were sucked empty and driven into the ground when the grid dried up, and engineers spent days coordinating their recovery so that the first cell to go online didn't feed the entire electric grid on its own. Tricky stuff.
TCP/IP is built to understand rolling outages and uncoordinated recovery. The electric grid still is not. That, I would submit, is the main issue and not that routers on the edge of small networks didn't have generator backup.
=^..^= all your rodent are belong to us
If power *is* a critical infrastructure, and lack of power is what caused these problems, how can that support a conclusion that the Internet is not ready to be considered critical?
Because the internet is communication, not power. So the correct comparison is the telephone company, not the power company.
Power can be backed up locally. Communication can not. So power only needs to be available MOST of the time, with backups on any critical services, to achieve its "critical infrastructure" level of reliability. Communications, on the other hand, requires an infrastructure with multiple links, routing around failures, and local power backup at the active nodes to achieve its own "critical infrastructure" service levels.
The telephone company HAS this level of backup power built in. Switching centers, for instance, run their equipment directly from TWO banks of 48v batteries suitable for days of operation, and run battery chargers continuously when there's power available. Repeaters on long copper trunks are powered from the endpoints - and can run with only one endpoint hot. Telephone instruments are powered from the central office switch via the copper wire. Active customer premesis equipment has battery backup for critical features or is designed to connect at least one POTS phone directly to a copper pair to the switch in case of blackout, and so on. SONET nodes are wired as rings rather than trees, so you have to cut TWO fibers in different places to isolate them. Other trunks are redundant and switch over automatically in case of outage. I could go on. About the only place a single cable cut can cut you off is the line to your house - and if you pay (a lot!) extra (as some businesses do) you can get another run in by a different path, so no single backhoe or downed pole can isolate you.
The Internet was ORIGINALLY designed with this kind of redundancy built in. Individual links were via the tellco's infrastructure, with its power-failure resistance. Routing was automatic, and would find a route between any two nodes if one still existed. (It WAS designed by people who were at least THINKING about surviving a nuclear attack, after all.)
But with the "inflation" of the commercial internet this robustness was lost. The explosion of active IP addresses made routing tables impossibly large, while most sites were connected via a local ISP rather than ad-hoc connection to two (or more) internet neighbors.
So the internet split into a "backbone" with SOME of the old routing redundancy, interconnecting ISPs, who in turn give you a default route JUST to their own servers. If your ISP fails you're cut off, and if the backbone connections to your ISP fail, ditto (even if you in principle COULD reach the rest of the net through somebody with a two-ISP feed.)
The ISP buisness has FIERCE price competition, and one BIG way to cut costs is to reduce redundant routing internally and neglect backup power.
At the backbone level the long-haul networks carrying the data had an even FIERCER price war, due to the excessive long-haul buildout of the internet bubble. Perhaps some of the upstarts powered their switches and repeaters with local power (on the assumption that the could slough any site that had a local power failure and that they'd have a path with all equipment powered between any two customers still live). A major blackout would violate that assumption, cutting off not just the dead area but others who could only reach the rest of the net by routing through it.
How about your DSL or cable IP feed? Did your cable company include battery backup power in the repeaters, pole-mounted routers, and fiber/cable bridges? Is you settop box battery backed up? How about your DSL modem? If you're corporate, are all your routers, your VoIP bridge, and any desktops running a softphone on the UPS? Do your SIP phones run if the power fails? (Home users ditto for your PC.)
Until all these are fixed the internet is NOT running at "critical infrastructure" reliability levels. So you'll want to think VERY CAREFULLY before disconnecting your POTS line and depending on Internet-VoIP. B-)
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
The data they present indicates that the blackout had a severe regional impact. I see nothing that shows that there was a significant global impact (meaning that I can't get data from AS 12374 to AS 553, for example).
n dex.html
y s-030502 -NRC-911.pdf
That's correct. In fact, our data showed that it clearly did _not_ have global impact. (Compare with various worm events, which do generally have global impact: http://www.renesys.com/projects/bgp_instability/i
cod red ii and nimda report)
The WTC collapse probably had more impact on global routing (some large carriers had primary and backup equipment in both basements).
Actually, it did not. It did affect some regions outside the US that had trans-Atlantic connectivity straight into NYC, but otherwise it was geographically well localized. This report (PDF slides) compares it to Code Red and Nimda:
http://www.renesys.com/projects/911/renes
9/11 report
They conclude that the Internet is not ready to be critical infrastructure.
Huh? It would seem to me that the fucking power grid is not yet ready to be critical infrastructure but hey, here we are. Shit. There is nothing in the world (except for the sun, oceans, etc.) that is 100.00000% dependable.
Our top story tonight: humans, human inventions imperfect. Tomorrow: sky blue, water wet.
Dear Slashdot: next time you want to mess with the site, add a rich-text editor for comments.