Blackout Worse For Internet Than Previously Thought?

An anonymous reader writes "Renesys (the people who previously brought you cool animated graphs of the US/Canada power outage has a new report out. It challenges the widely held belief that the Internet was largely unaffected by the power outage. Lots of important networks lost connectivity, including banks, hospitals, government organizations and investment funds. There's a cool appendix on the huge Italian power outage in September as well. They conclude that the Internet is not ready to be critical infrastructure."

Comments

Obvious?

[Huogo]

It has always seemed to me that the internet isn't all that de-centralized, but a few major companies ran most of the backbones. Since it isn't a huge ad-hoc network, most of the data for an area probably goes out through no more than 5 connections. Especially in rual areas, I wouldn't doubt that at least one routing station in each of those chains doesn't have good long term backup facilities.

Re:Obvious?

[Anonymous Coward]

Well, duh. (Yes I did read the article) If 1/3 of the country goes out, we are sure as hell going to loose *some* connectivity.

Its pretty cool though that it can be observed in terms of routing activity.

Yes, ideally everyone would have backup power (and enough of it). If power outages were common, it might be a good selling point for ISPs, but they aren't so not many people want to may more $ per month just to have battery backup. (Especially residential customers who won't have it at home anyway).

I don't like big government either, but an FTC law (or whatever) mandating backup power for ISPs/backbones of sufficient size or type of service (business vs residenial) might be what's needed.

If phone companies have such a requirement, then the internet probably should to.

(Unfortunately, most phones are powered from the phone line, but I can't say the same about my cable modem...)

OTOH, did many businesses care to have backup power for sufficient length? Just because the some routers went out, it might not have mattered if their end users were already without power.

A robust internet is a great thing, but not near as great as a robust internet with robust users.

Re:Obvious?

[tmu]

The ability to observe the outage (sharply) through routing activity is definitely the part that we thought was coolest.

People are saying two different things here: 1) well, duh, if power is out lots of people can't connect to the web; 2) if the core of the internet routes around that who cares. These are both interesting points. Here are some thoughts:

1) We agree. That's what I though. But read the keynote press releases. Or just google on 'blackout Internet' and you'll find glowing stories about how 'the Internet' didn't even blip under the blackout. We prove pretty conclusively that this is incorrect.

2) The core of the Internet did, indeed, route around the outage. This is good. What is less good is that thousands of networks within the outage area lost connectivity, either due to lost power themselves, or upstreams that lost power (or telcos who lost battery backup on csu/dsu units, or whatever). These are *not* DSL customers (or that grade, anyway). All of these are BGP-speaking networks with their own Autonomous Systems and their own prefixes.

The fact that so many networks went down is significant, given that many organizations are coming to rely on the Internet as a critical communications infrastructure.

That's fine

[dschl]

They conclude that the Internet is not ready to be critical infrastructure.

Apparently, neither is the electrical network. Back to candles we go.

Ready or not, here we come.

[Dav3K]

Ready or not, the internet is increasingly being used for critical infrastructure. At best, failures like the power outage should motivate governments and industry to bolster the internet up to where it needs to be for reliability standards.

Re:Ready or not, here we come.

[RedHat Rocky]

Industry is more than willing to "bolster the internet up to where it needs to be for reliability standards", it's called Spend the Money. You want 5 9's connectivity, you gotta pay. The government get involved? I thought you were looking for MORE reliable? :)

The proper conclusion from the data would be that many businesses in the blackout area, despite handling large sums of money daily, did not have sufficient redundant power or connectivity.

Whether anyone could have anticipate such a large scale blackout (and prepare accordingly) is another topic.

Infrastructure

[pete-classic]

They conclude that the Internet is not ready to be critical infrastructure.

But seemingly no less so than the power grid.

-Peter

And the power system is?

[David Frankenstein]

Exactly how does one system's dependance on a critical infrastructure (the power grid) and it's failure when that infrastructure fails imply that it's not ready?

Re:And the power system is?

[shotfeel]

That was my question. The core of the conclusions seems to be,

"We find that Internet connectivity in the blacked-out region was far more seriously affected than has been publicly revealed."

Pointing out that areas without power didn't have internet connectivity seems rather redundant to me. The big question is how did it affect people outside that area? The fact that the rest of the world just plugged right along seems contrary to the conclusion they seem to want to draw.

Re:And the power system is?

[Rick.C]

Pointing out that areas without power didn't have internet connectivity seems rather redundant to me.

For home users and small businesses, you are quite right. What about large businesses that invested in generators so they could stay online 24/7? They were prepared to remain online to conduct their business. They depended on the Internet and it failed them.

I work for a large bank. We were not hit by the power outage, but we were scrambling to find routes around the areas that were.

Re:And the power system is?

[orangesquid]

I think the implied problem was the connectivity that was provided by ISPs and backbone segments running off the affected sections of the power grid.

If the Internet were more redundant and ad-hoc (less backbone-centric), it would recover from problems better. That's how it was originally envisioned; unfortunately, the commercialization of NSFNet has largely destroyed this approach, for better or worse.

We have a more organized network, but it's very dependent on critical points because of it's multiplexing organization strategy, so when that fails...

Critical Infrastructure?

[Anonymous Coward]

Bah, I could have told you that. I work for an ISP that serves 15 states. I get calls from people who put 100% of their business into a DSL line - with no backup to other carriers or mediums. When a hardware failure or trunk line failure occures - they go postal.

Sorry, but uptime is not 100% never was, never will be - plan for it, or deal with it when your connection goes down.

Even though we have multiple connections to the backbone - local trunks can go down. Aka backhoe attacks on burried fiber, or dove hunters blasting pole run fiber (don't laugh - it happened last week). If you don't have a backup DSL,ISDN, or heck even dialup connection for your business - then stfu and wait while we repair.

And don't even get me started on residential accounts that call in 'I use this for work I need it up now - send someone out today.' And it's Sunday evening... no - you didn't pay for a business account, so you get residential service levels which include 24-72 hour turn around on repairs.

critical infrastructure

[Anonymous Coward]

If power *is* a critical infrastructure, and lack of power is what caused these problems, how can that support a conclusion that the Internet is not ready to be considered critical?

I'm not saying there isn't other evidence that would support such a conclusion, but the real failure here was the power infrastructure, upon which the net relied "critically" in the first place...

I'm not sure I agree with their conclusions

[jandrese]

The reason everybody said that the internet survived was that they were able to visit most of the sites they cared about during the blackout. The chart seems to show that many links and servers were down (presumably without power) during the blackout (including some major components of the internet), yet most people basically unaffected. This seems to suggest that as long as the server itself isn't in the middle of a blackout, the Internet can survive rather well. How many of your learned about the blackout from Slashdot or some other online news source?

Re:I'm not sure I agree with their conclusions

[lunartik]

How many of your learned about the blackout from Slashdot or some other online news source?

I learned about it when my card wouldn't swipe me out of the parking garage. And then when all of the traffic lights were out downtown. And then after searching the dial and finding the one AM station still on the air.

worked just fine here during blackout

[bbn]

I live in denmark and recently we had a blackout that lasted maybe 10 hours.

While I was unable to make any phone calls, I could get on the internet with GPRS and surf to our server with my laptop for as long as the laptop batteries lasted.

The server is hosted in a colo datacenter which was also in the middle of the affected area. We run a mud on the server, and most of the players are from USA. They never discovered the blackout as the datacenter went on emergency diesel backup and apparently knew to make business with backbone providers that also knew their stuff.

So to the people saying that internet can only route around blackout areas but not _through_ them, this is not true. Seems at least here in denmark all the infrastructure on the backbones got backup power and just keeps working when everyone else is busy lighting candles.

some things to note

[theCat]

The vast majority of the networks that went dark were 24-bit in size. That is generally either small to medium businesses or home office, or a division of a larger business. I think we can all agree that outages at that level, though undesired, are not the end of the world. Small outfits and home office workers can afford the down time in the case of a general crisis (ie the buses aren't running, either, so go have a coffee and read the WSJ) and 4-8 hour outages on their DSL are not uncommon either. I know that is the case where I work, and we have a global presence too.

We invested in a very large portable battery backup system for our server room back when California was having its own blackouts. The stack would probably stay up an hour or so, which we figure is enough to manage most blackouts nicely, and anything longer than that is a "major cockup" that we need to wait out. But if we go down who will care? Just us, and not all that much.

I think that the general expectation regarding the internet is not that it will stay up 100% in a crisis, but that it will continue to operate in cells of functionality during most kinds of disaster, then recover quickly on its own as soon as it can built remote connections again. Compare that to the electric grid, where most or all cells of function were sucked empty and driven into the ground when the grid dried up, and engineers spent days coordinating their recovery so that the first cell to go online didn't feed the entire electric grid on its own. Tricky stuff.

TCP/IP is built to understand rolling outages and uncoordinated recovery. The electric grid still is not. That, I would submit, is the main issue and not that routers on the edge of small networks didn't have generator backup.