Slashdot Mirror
Blackout Worse For Internet Than Previously Thought?
An anonymous reader writes "Renesys (the people who previously brought you cool animated graphs of the US/Canada power outage has a new report out. It challenges the widely
held belief that the Internet was largely unaffected by the power outage. Lots of important networks lost connectivity, including banks, hospitals, government organizations and investment funds. There's a cool appendix on the huge Italian power outage in September as well. They conclude that the Internet is not ready to be critical infrastructure."
Industry is more than willing to "bolster the internet up to where it needs to be for reliability standards", it's called Spend the Money. You want 5 9's connectivity, you gotta pay. The government get involved? I thought you were looking for MORE reliable? :)
The proper conclusion from the data would be that many businesses in the blackout area, despite handling large sums of money daily, did not have sufficient redundant power or connectivity.
Whether anyone could have anticipate such a large scale blackout (and prepare accordingly) is another topic.
Anything is possible given time and money.
The vast majority of the networks that went dark were 24-bit in size. That is generally either small to medium businesses or home office, or a division of a larger business. I think we can all agree that outages at that level, though undesired, are not the end of the world. Small outfits and home office workers can afford the down time in the case of a general crisis (ie the buses aren't running, either, so go have a coffee and read the WSJ) and 4-8 hour outages on their DSL are not uncommon either. I know that is the case where I work, and we have a global presence too.
We invested in a very large portable battery backup system for our server room back when California was having its own blackouts. The stack would probably stay up an hour or so, which we figure is enough to manage most blackouts nicely, and anything longer than that is a "major cockup" that we need to wait out. But if we go down who will care? Just us, and not all that much.
I think that the general expectation regarding the internet is not that it will stay up 100% in a crisis, but that it will continue to operate in cells of functionality during most kinds of disaster, then recover quickly on its own as soon as it can built remote connections again. Compare that to the electric grid, where most or all cells of function were sucked empty and driven into the ground when the grid dried up, and engineers spent days coordinating their recovery so that the first cell to go online didn't feed the entire electric grid on its own. Tricky stuff.
TCP/IP is built to understand rolling outages and uncoordinated recovery. The electric grid still is not. That, I would submit, is the main issue and not that routers on the edge of small networks didn't have generator backup.
=^..^= all your rodent are belong to us
To a certain extent you may be correct . . . . .
But you have to look at it in a slightly different light
If the power goes out hospitals , telephone networks , and other "essential" services tend to have backup generators and backup batteries.
Now for the internet to be ready to reach the legendary uptime of POTS it will have to improve
This means that we should not be routing information on which if it doesnt get there people die exclusively over the internet
The so called essential services must all be willing to accept that one or more of the essential services will fail (hence the amazing backup batteries , generators etc. found at hospitals and telphone companies)
If power *is* a critical infrastructure, and lack of power is what caused these problems, how can that support a conclusion that the Internet is not ready to be considered critical?
Because the internet is communication, not power. So the correct comparison is the telephone company, not the power company.
Power can be backed up locally. Communication can not. So power only needs to be available MOST of the time, with backups on any critical services, to achieve its "critical infrastructure" level of reliability. Communications, on the other hand, requires an infrastructure with multiple links, routing around failures, and local power backup at the active nodes to achieve its own "critical infrastructure" service levels.
The telephone company HAS this level of backup power built in. Switching centers, for instance, run their equipment directly from TWO banks of 48v batteries suitable for days of operation, and run battery chargers continuously when there's power available. Repeaters on long copper trunks are powered from the endpoints - and can run with only one endpoint hot. Telephone instruments are powered from the central office switch via the copper wire. Active customer premesis equipment has battery backup for critical features or is designed to connect at least one POTS phone directly to a copper pair to the switch in case of blackout, and so on. SONET nodes are wired as rings rather than trees, so you have to cut TWO fibers in different places to isolate them. Other trunks are redundant and switch over automatically in case of outage. I could go on. About the only place a single cable cut can cut you off is the line to your house - and if you pay (a lot!) extra (as some businesses do) you can get another run in by a different path, so no single backhoe or downed pole can isolate you.
The Internet was ORIGINALLY designed with this kind of redundancy built in. Individual links were via the tellco's infrastructure, with its power-failure resistance. Routing was automatic, and would find a route between any two nodes if one still existed. (It WAS designed by people who were at least THINKING about surviving a nuclear attack, after all.)
But with the "inflation" of the commercial internet this robustness was lost. The explosion of active IP addresses made routing tables impossibly large, while most sites were connected via a local ISP rather than ad-hoc connection to two (or more) internet neighbors.
So the internet split into a "backbone" with SOME of the old routing redundancy, interconnecting ISPs, who in turn give you a default route JUST to their own servers. If your ISP fails you're cut off, and if the backbone connections to your ISP fail, ditto (even if you in principle COULD reach the rest of the net through somebody with a two-ISP feed.)
The ISP buisness has FIERCE price competition, and one BIG way to cut costs is to reduce redundant routing internally and neglect backup power.
At the backbone level the long-haul networks carrying the data had an even FIERCER price war, due to the excessive long-haul buildout of the internet bubble. Perhaps some of the upstarts powered their switches and repeaters with local power (on the assumption that the could slough any site that had a local power failure and that they'd have a path with all equipment powered between any two customers still live). A major blackout would violate that assumption, cutting off not just the dead area but others who could only reach the rest of the net by routing through it.
How about your DSL or cable IP feed? Did your cable company include battery backup power in the repeaters, pole-mounted routers, and fiber/cable bridges? Is you settop box battery backed up? How about your DSL modem? If you're corporate, are all your routers, your VoIP bridge, and any desktops running a softphone on the UPS? Do your SIP phones run if the power fails? (Home users ditto for your PC.)
Until all these are fixed the internet is NOT running at "critical infrastructure" reliability levels. So you'll want to think VERY CAREFULLY before disconnecting your POTS line and depending on Internet-VoIP. B-)
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
From having been around the Internet for the last 15 years now..
The Internet was a lot MORE capable of being infrastructure, before *.com happened. Since it has been commercialized, the backbones have become more and more important, and routing/re-routing less and less important.
"Error: No Route To Host" at one point in history, literally meant that the computer directly connecting the computer you were trying to reach was offline. Now, "No Route To Host" means that there was a power failure somewhere in the world that just happened to be in the way of your provider routing through a few other providers, or that a janitor somewhere kicked out a plug in Minnesota, while you were trying to connect from Michigan to Texas.
The system used to be able to route around virtually ANY connectivity issue. Now, it can't route it's way out of a wet paper bag.
"Champagne for my real friends - and real pain for my sham friends!" http://ericblade.postalboard.com/
The data they present indicates that the blackout had a severe regional impact. I see nothing that shows that there was a significant global impact (meaning that I can't get data from AS 12374 to AS 553, for example).
n dex.html
y s-030502 -NRC-911.pdf
That's correct. In fact, our data showed that it clearly did _not_ have global impact. (Compare with various worm events, which do generally have global impact: http://www.renesys.com/projects/bgp_instability/i
cod red ii and nimda report)
The WTC collapse probably had more impact on global routing (some large carriers had primary and backup equipment in both basements).
Actually, it did not. It did affect some regions outside the US that had trans-Atlantic connectivity straight into NYC, but otherwise it was geographically well localized. This report (PDF slides) compares it to Code Red and Nimda:
http://www.renesys.com/projects/911/renes
9/11 report