Electricity Outage Puts Routing to a Tough Test

infofarmer writes "Today at about 11:30 MSD (GMT+4) a major electricity outage in Moscow, Russia brought new meanings to words like "uninterruptible", "redundant" and "uptime" for network administrators, who haven't experienced such harsh and unexpected power failures since the USSR got its Internet connection. Half of the city is totally out of electricity - including subway and the most important traffic exchange point, half of the top russian sites went down, including www.mail.ru, www.rambler.ru, www.lenta.ru, some of them haven't been brought up yet. IP packets going from ADSL users in Moscow to some local sites got rerouted to somewhere in London and then back to Scandinavia, where they met their "No route to host" deadend. Other routers found themselves in a loopback, which made many packets get dropped with TTL expired. The point is that most of popular servers have got two or three mainline Internet connections, but lack of BGP/RIP2/whatever configuration resulted in packets losing their way to hosts."

Comments

The submitter has to have his priorities checked

[arhar]

Knowing how things are done in Russia, you should be a lot more concerned with things OTHER than Internet.. Everything is such a fucking mess over there, that's I really hope no serious injuries happen. I already read the news that sewer water is being dumped into the Moscow river because of a plant failure. In times like these, who gives a shit about Internet?

Not big headline?

[antdude]

It was interesting that news.google.com, cnn.com, msnbc.com, etc. do not have this story on its front news page. I guess the outage isn't severe like one in New York a few years ago.

Re:The submitter has to have his priorities checke

[josecanuc]

I already read the news that sewer water is being dumped into the Moscow river because of a plant failure.

This is what is supposed to happen. All (nearly all?) sewage treatment plants have a bypass to send the input straight to the output, which is usually a river or lake.

They do it because when a treatment plant cannot accept any more sewage, whether due to excessive water input by rain, or by power loss, the customers are better served by *NOT* letting the sewage back up into their houses. The stuff has to go *somewhere* when all their holding tanks are full. This is the last-resort method of dealing with problems at such plants.

Routing

[EdMcMan]

Are there any technical reports out of what happened to the network? What is the russian equivilant of NANOG?

Re:Luckily,

[fafaforza]

Well, its hosted in Florida.

Re:Internet... works!

[lheal]

both of our peers

That's why.

TCP/IP and the Internet anticipate cooperation among sites. You and your neighbors should all happily route each other's packets.

The trouble is that in many places it doesn't work that way. There are rural "leaf" nodes, of course, but there are many more sites which have only one connection because of what I consider to be petty business decisions.

Two competing ISPs in the same area should share a direct link to each other. If they have different upstream providers, then when one provider goes down the other picks up the slack. In any case local traffic should stay local.

The fear, of course, is that one ISP will choose a bad provider and take advantage of the other. That has an easy fix: if the other one starts to abuse you, pull the plug.

Single points of failure are not supposed to exist.

Re:I'm worse than Russia.

[Nytewynd]

Maybe I'm oversimplifying, but you could have plugged your modem directly into your laptop. No other computers would have worked, but you would have had internet connectivity.

If you had any servers running, though... well... time to get a UPS. :-)

That is obviously an option, but that laptop is set up only with wireless and I really didn't care enough to configure the landline and assign IPs.

My server has gone down because of the power twice in the past 2 days. I would consider UPS if it mattered. Last night even UPS wouldn't have saved me. Power was out for 3-4 hours. For the trivial web pages and teamspeak server it runs, I'm fine with it being offline.

I think this is a "political outage" of some sort

[melted]

There's a Russian politician of Yeltsin era, Anatoly Chubais who is in charge of RAO UES Russia (which is an uber-organization controlling production and distribution of energy in Russia).

While the guy is not as powerful as he was a few years ago, he still poses a significant threat to Putin's third (and fourth, and so on) term presidency, and further concentration of power in Putin's hands.

So within half a few hours of outage, Putin blamed Chubais directly for this, and Russian justice dept opened up a criminal case against him. If you know anything about Russia, you know that Russian DOJ (Prokuratura) doesn't start criminal cases against wealthy and powerfull businessmen and politicians unless instructed to do so by Putin.

So I'd bet dollars against donuts that this outage was caused by folks from Lubyanka (FSB aka KGB) purely to remove Chubais, and if cards play well maybe even give him a lengthy prison term.

CIA running Netwar Wargame This Week

[carcosa30]

This is interesting news coming as it does in the week that the CIA is scheduled to run a 3 day netwar exercise called Silent Horizon.

http://apnews.myway.com/article/20050525/D8AAFUIO2 .html

Am I just blowing smoke here...?