Slashdot Mirror
Electricity Outage Puts Routing to a Tough Test
infofarmer writes "Today at about 11:30 MSD (GMT+4) a major electricity outage in Moscow, Russia brought new meanings to words like "uninterruptible", "redundant" and "uptime" for network administrators, who haven't experienced such harsh and unexpected power failures since the USSR got its Internet connection. Half of the city is totally out of electricity - including subway and the most important traffic exchange point, half of the top russian sites went down, including www.mail.ru, www.rambler.ru, www.lenta.ru, some of them haven't been brought up yet. IP packets going from ADSL users in Moscow to some local sites got rerouted to somewhere in London and then back to Scandinavia, where they met their "No route to host" deadend. Other routers found themselves in a loopback, which made many packets get dropped with TTL expired. The point is that most of popular servers have got two or three mainline Internet connections, but lack of BGP/RIP2/whatever configuration resulted in packets losing their way to hosts."
Sigs cause cancer.
The msk-ix went down, and now that its back up, your going to have it slashdotted?
Can someone give this guy a metal?
You are confusing me with someone who cares.
Oh nevermind...
no more all off mp3 .com
Obviously the MPAA/RIAA are to blame..
must have been males- didn't stop to ask for directions
bride.ru is still up.
Good thing I saved all my russian pr0n.
Pulp Audio Weekly - Geek News and Reviews
Gee, all three quoted sites all seem to work fine...
That's crazy. These sites were down before they hit Slashdot.
Web Design Tips
An alternate headline should be:
Correct router configuration can be difficult!
Online Starcraft RPG? At
Dietary fiber is like asynchronous IO-- Non-blocking!
Last night I lost power for about 3 hours. My laptop worked. My cable modem stayed connected on battery backup. My router is plugged in and died.
Russia's connections at least made a couple of hops before dying. Mine died on 1 hop. It did illustrate the uselessness of a battery-backed up modem on my network, however.
/. ++
for packets to end up in another part of the world before being finally rejected.
many systems are set up to use an upstream transit link if no local routes can be found so packets can get sent quite a long way before they finally hit a brick wall.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
when you don't give in to russian hackers' demands? ;)
Did kremvax stay up?
From article title "cause of blast unclear." I almost read that nuclear. Just need to switch two letters.
For the last three or four weeks my gmail account has been POUNDED by 100-200 cyrillic spam messages every day. The filters catch them, but I have to clean out my spam folder pretty often.
I've gotten none in the last couple hours.
Knowing how things are done in Russia, you should be a lot more concerned with things OTHER than Internet.. Everything is such a fucking mess over there, that's I really hope no serious injuries happen. I already read the news that sewer water is being dumped into the Moscow river because of a plant failure. In times like these, who gives a shit about Internet?
I can't seem to log into my bank account to update my out-of-date account information.
Wonder if these are somehow related.
Beauty is in the eye of the beerholder.
...and now, the /. effect. Hard luck, Russia.
I don't want to read
but lack of BGP/RIP2/whatever configuration resulted in packets loosing their way to hosts."
Those mischevious packets. Unraveling networks where'er they roam.
sewer water is being dumped into the Moscow river
and
who gives a shit about Internet?
What exactly are you trying to say here?
Infuriate left and right
Seriously, I have not received any spam for 4 hours now....
Someone should blackmail the Moscow electrical grid. "If you ever want to send spam again, fork over $200 and send it to this address..."
Its not the United Soviet Socialist Republic anymore duhhhh :) It's Russia.
Allofmp3 is still live and kicking.
:-)
No interuption of service from where I stand
Artificial intelligence is no match for natural stupidity
Apparently the people dumping sewer water do ...
hopefully, this very useful russian website is still up & running
#include "coucou.h"
packets loosing their way to hosts
So by the time the packets got to the hosts, they were really loose? You could probably cram a lot of data in to these packets by the time they got there. I imagine the real problem is that these loose packets are filling up with all kinds of foreign matter as they pass through the Scandinaval regions.
The RIAA's crack anti-mp3 commando teams are rumored to be cutting a bloody swath through street markets and datacenters across the city.
How did the packets loose their way? Was their way hog-tied, fenced, locked in a barn?
Or did you mean "packets losing their way?"
I think you need to check your priorities. How do you think geeks all over the world just found out about the power failure?
It was interesting that news.google.com, cnn.com, msnbc.com, etc. do not have this story on its front news page. I guess the outage isn't severe like one in New York a few years ago.
Ant(Dude) @ Quality Foraged Links (AQFL.net) & The Ant Farm (antfarm.ma.cx / antfarm.home.dhs.org).
I live in Russia, about 1000 km from Moscow. We were hit by network outage, nothing worked (even Slashdot :( ) for about 30 minutes. Number of routes announced by both of our peers was about 700 instead of normal 150000.
:)
But then routes began to appear again! I was amazed, Internet routed itself around damaged segments, packets were routed through Japan (!), Finland and Holland instead of Moscow. The most funny part was when I traced the route to a computer in the next building - it went through Saint-Petersburg
I was able to access Slashdot, and most of Russian sites (http://newsru.com/ , http://ntv.ru/ , http://nbc.ru/ not directly affected by outage.
That explains why there was less spam in my inbox today.
There's more traffic on the 'net than pr0n, wazrez, mpEs and /.
Some of it actually matters.
If the g'vt kept the data on you that google does you'd better believe you'd be calling it "doing evil"
I think any Russian who saw your post would probably say, "Chill out, dude!" They've been dealing with a shoddy infrastructure for decades -- a power outage is not likely to take them unprepared.
This is what is supposed to happen. All (nearly all?) sewage treatment plants have a bypass to send the input straight to the output, which is usually a river or lake.
They do it because when a treatment plant cannot accept any more sewage, whether due to excessive water input by rain, or by power loss, the customers are better served by *NOT* letting the sewage back up into their houses. The stuff has to go *somewhere* when all their holding tanks are full. This is the last-resort method of dealing with problems at such plants.
"Power outages considered harmful"
Are there any technical reports out of what happened to the network? What is the russian equivilant of NANOG?
Yes, but slashdot is concerned with the internet, and so this is an appropriate forum to discuss how an event like this affects the internet. I don't think someone who runs an ISP in Russia should be trying to figure out how to get the sewer working, they should be figuring out how to get the internet up.
during this outage, hacking attempts and spam have declined 95%.
Nobody cares that this was in russia, that people can't get their email, or that it was because of a power outage.
The reason this is on ther front page is because the internet is suposed to be able to handle things like this. People will be watching how the routers automatically deal with the outage (there's one response like that already), and what manual intervention it needed. Hopefully this information will be used for training the next generation of router admins.
Even if this was because of a meteor strike or nuclear bomb, we'd still be interested on how the net took it. We'd be more interested in the everything else about the event, but the response of the network would still be interesting
Perhaps these guys touched live wires ^^
:)
Off-topic, but an interesting read
yes exactly its a method of last resort
you are right nowhere near as bad as letting it back up through the sewer systems into populated areas but that doesn't mean its a good thing just less bad than the availible alternatives.
those sewage treatment plants were installed for a reason you know.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
"Considering that sewage, power and medical processes could all rely on the internet..."
If you have critical processes (any!) relying on the internet? Then you have much bigger problems to deal with?
Damn that's right. I am an American citizen of the United States and I live a squeaky clean life. Hacking is illegal. Spam is illegal and russian sites are illegal to view. I believe in ID cards and plugging my ID card into a special socket on my American made computer to access the American internet to prevent terrorism .Spam is terrorism and russians are communists and terrorists. Soon we will outlaw all non US sites and make our God blessed borders safer for good upright wholesome citizens of the US.
Our proud American fathers like John Wayne and Gary Cooper who made this country great by slaughtering the savage indian invaders would turn in their graves if they thought America's borders were under attack from Russian hackers.
Damn.
Of course Putin will blame this on terrorism and use the opportunity to further consolidate his power.
Which should sound pretty familiar to you folks over there in the US...
Why's that? Because we've all been watching the new Star Wars movie this week?
Oh, I get it... When you say "consolidate power" you mean "narrowly win re-election against a weak candidate." The language barrier is sometimes tough to overcome when interacting with Europeans.
Yes, the fact that we were attacked by terrorists probably was a factor in Bush's re-election. Over here, the destruction of our landmarks and slaughter of our citizens tends to rally people up against the invaders. It's a big deal to us and how the President responded was more important to us than his tax policy, his run-away spending, or even a war which a lot of us disagreed with.
Over there, an attack on the scale of 9/11 barely qualifies as a "busy news day" so I can see where you would be confused by our strong reaction.
--
I am not a script, I am an Anonymous Coward!
line about "In Soviet Union, the internet powers YOU!" (I have friends in Moscow. They gotta be hurting for their /. fix. :-)
MSBPodcast.com The opinions expressed here are my own. If you don't like 'em... Think up your own stuff.
http://mosnews.com/news/2005/05/25/chubaiscriminal case.shtml
From the article:
Russian prosecutors on Wednesday opened a criminal case against the management of power monopoly Unified Energy System (UES) after a major power outage in Moscow, agencies reported Wednesday.
The case was opened to investigate possible negligence, the Interfax agency quoted the Prosecutor General's Office as saying.
half of the top russian sites went down, including www.mail.ru, www.rambler.ru, www.lenta.ru, some of them haven't been brought up yet
Oh, this'll help them...
You should wear a helmet and ride on a special bus, you're just that fucking retarded.
half of the top russian sites went down,
And in other news..phishing sites have decreased by 50%.
aren't datacenters supposed to have backup power?!
or is there more to this story than we are being told right now?
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
there goes 50% of the world's spam. Now if only .kr and hinet.net would go down...
This guy is way out there
Was kremvax affected?
Now on the other hand, this is slashdot, you know about geeky technical stuff, not currentpoliticalaffairsdot.org
There's a Russian politician of Yeltsin era, Anatoly Chubais who is in charge of RAO UES Russia (which is an uber-organization controlling production and distribution of energy in Russia).
While the guy is not as powerful as he was a few years ago, he still poses a significant threat to Putin's third (and fourth, and so on) term presidency, and further concentration of power in Putin's hands.
So within half a few hours of outage, Putin blamed Chubais directly for this, and Russian justice dept opened up a criminal case against him. If you know anything about Russia, you know that Russian DOJ (Prokuratura) doesn't start criminal cases against wealthy and powerfull businessmen and politicians unless instructed to do so by Putin.
So I'd bet dollars against donuts that this outage was caused by folks from Lubyanka (FSB aka KGB) purely to remove Chubais, and if cards play well maybe even give him a lengthy prison term.
Today at about 11:30 MSD (GMT+4)
I suppose that was taken from when all the digit clocks stopped working and just displayed one time...
Damn! now where am i going to get my pirated software from.
... was not touched by this disaster almost at all, by my experiences.
The only sign of trouble was some non-working major sites that time. The trouble located (and is still there for unhappy about 1 mil ppl without electric power) in southern part of Moscow.
I've been trying to go to my banks website since the email said to update my info....wonder why, maybe their head office is in Russia?? lol
There are 10 kinds of people in the world - those who understand binary and those who don't
We had a power outage at a site that hosts one of our major databases.
The biggest issue is the finger pointing between all the parties involved. This is why I said that we have to include all those parties in our disaster readiness planning.
But it seems to fall on deaf ears. At least I'll be able to say "I told you so." when something really bad happens.
PATRIOT Act.
PATRIOT Act II.
RealID.
Open your eyes.
Why, did cars stop working? Did elevators break and drop to the ground? Is that sewage somehow worse than everything else that gets dumped into the rivers in Russia? Biological waste can (and will) be broken down very quickly by the life in the river, but all of the heavy metals and other pollutants that get dumpes aren't the same...
Oh, you're not stuck, you're just unable to let go of the onion rings.
There was a fire in transformer station "Czagino" the following outgage spread like the one in NY.
There have not been any reported casualities...
info from PAP
Now we know why the power went out. Everyone had their spambots cranked up too high.
It's rare that you're presented with a knob whose only two positions are Make History and Flee Your Glorious Destiny.
Soviet Russia.... lololololol!!!
Putin has stated he won't be standing for a concurrant third term. Also, it is doubtful if they wanted to set someone up that they'd use means so destructive to Russia's economy and the personal wealth of many of Putin's supporters. It is more likely this outage was simply an accident or incompetence.
PS Hey, what's with the "To confirm you're not a script" weirdness?
I'm sure Putin will exploit the power outage to weaken and possibly get rid of Chubais.
:)
Whether the FSB caused the outage directly, to prompt an attack on Chubais is another matter. Maybe they were working on a plan but it wasn't ready yet. They have a lot to do
Even Putin sometimes just exploits opportunities.
In any case, the outcome is the same.
So when /. Comments go offline, everything gets rerouted to a lake somewhere? Oo
Eventually we will win over all you terrorists, drug dealers, illegal gamblers, child pornographers, file traders, extortionists and other assorted riff-raff.
The Patriot Act doesn't go nearly far enough.
Just wait until we pass the
Securing America via Federal Enforcement (SAFE) Act and the
National Operations againt Criminal Racketeering on the Internet and in Multi-user Environments (NO CRIME) Act.
It's coming boys and girls, it's coming.
Just because it CAN be done, doesn't mean it should!
One thing is what he "states", and another is what he really does. There's very little in common between Putin's speeches and the stuff he does after giving speeches.
Consider the "dictature of law" doctrine that he so vehemently supported when he came into office. For some reason, dictature of law applies only to his (or his buddy oligrachs') political opponents. Khodorkovsky was an opponent, Chubais is an opponent, Berezovsky, Gusinsky, you can continue this list yourself. Abramovich, Deripaska, Mamut, etc. however didn't get any attention at all from DOJ even though every Russian knows these fellas can be sentenced to at least 10 years each without trial. Just dig a little bit and you'll find some "tax obligations". Please note, I'm not saying Khodorkovsky is not a thief, because he obviously is. However, of all the billionaire Russian thieves he's probably the most humane one. His only mistake was to take on Putin.
So don't judge Putin by what he says. He says one thing and does another.
I doubt it was the lack of RIP2 configuration that caused this. You don't use RIP in the core, you use BGP as the exterior protocol and most likely OSPF or ISIS as the interior protocol.
UPS: at least in one place in MSK-IX they did have proper UPS backups, you can tell from routing tables that some BGP connections have an uptime of 4 weeks plus. They did bounce (or it had a power failure) one of their core routers as all those peering connections only have an uptime of 8.5 hours. I'd rather not provide a link to this as the last thing they need is their core routers slashdotted with BGP table summary requests.
Connectivity: it appears MSK-IX is peered with at least 12 other sites that are also peered with another major IX. For example they are connected to three other sites that are also connected to AMS-IX and four other sites that are also peered with LINX, among a few others with only 1 connection to another Internet Exchange. Many of these were thru Informtelecom XXI, so if they also had power problems everything was running on 50% normal capacity. There should have been enough connections to keep things running (i.e. no single point of failure), but that is assuming everything is working/powered, and assuming these guys in the middle could/would handle all the traffic (unlikely).
BTW, packets don't lose thir way, routers lose their routes to destinations. When all the crap started the routes began to "flap", i.e. go up and down as routers were reset, power came back on, routers went back down under the heavy load, manually trying to route around the problem, etc. When your peer sees your routes flapping, they usually put a holddown on them for a period of time, meaning they won't readvertise your route updates to other routers on the internet (said flaps propogate all over the world, putting undue stress on other routers). So even once you get everything working again, the internet waits for a little bit to accept your routes. Well, some do and some don't or some wait longer. That's why you see routers still forwarding packets to London, apparently London thinks it can still get to Moscow so it's still advertising routes. You don't get the count to infinity problem with BGP, but loops are still possible, especially during major outages and route flapping. And routers get "routing loops," not "found themselves in a loopback."
I provided as much details as I could, it's lacking in a few places because I can't follow russian websites.
Please tell me that somebody is not using RIP to route traffic on the internet... Such actions diserve a flogging!
Which should sound pretty familiar to you folks over there in the US...
Yes, yes it does.
Do you love freedom??? Do you love freedom!!! DO YOU LOVE FREEDOM!!!!!!!!
Indeed, thanks for the link! Many cites have underground networks, the Paris catacombs are legendary. Defo one for the todo list! As the article says, there isn't much for the explorer types to do nowadays!
So this is just regualr life in Russia ?
Cool.
If Putin doesn't get his political shit togeather, he may see Russia revolt because of some of his previous dubious activities in regards to going agains the grain of democrocy. If such a revolt happens, see two possible outcomes.
1.Russia pulls a stunt like what happened in Ukrane and a true democratic leader is in place.
2.Russia falls back to a form of communism as Putin tries to force stability on the country as a tried and true method...in his eyes of a former KGB member.
Life is not for the lazy.
Considering that sewage, power and medical processes could all rely on the internet...
If that's true, whomever designed those systems deserves to be punished severely. Why would you put all your eggs in a basket you don't own?
Sure, I can envision scenarios where a sewage processing plant might work BETTER if it's connected to other plants and to remote sensors via the Internet, but if the plant FAILS CATASTROPHICALLY when the 'net goes out, or even mains power, that is Very Bad Planning.
The same can be said of the electrical grid. And the cellular network. And the water network. And the sewage system. Or the public road infrastructure. Or the food distribution chain. Face it - virtually every aspect of modern life requires you to rely almost completely on infrastructures that you do not own.
One has to remember what the internet actually is - a system to transport data. For me it has proven to be far more reliable than the power grid - when the lights go out the internet connection at my house remains active. Should a system go down the first time a packet is dropped? Absolutely not. But that isn't the case here. What Russia is seeing is a massive widespread power failure that is probably beyond the designed tolerance. And keep in mind what else -could- be happening. Why was the sewage shunted to the river? Was it to keep it out of the basement of the local hospital?
If the g'vt kept the data on you that google does you'd better believe you'd be calling it "doing evil"
Given this new development I bet dollars for peanuts that the assassination attempt against Chubais a month or so ago was initiated by the Russian president.
You can't handle the truth.
I suggest you submit your concerns about Moscow's sewerage problem to Flushdot.
When they came for the communists, I said "He's next door. Take him away. Goddam commies."
Our belowed monopoly net/phone/electricity provider gives us 5 to 30 minute breaks during tv watching, work and other activities regularly.
...
.... usually the net goes as these hits act as if someone triggered an EMP ....
....
....
... i remember when the 3000VA UPSes died .. hmm ... our world and their world ...
... even some of the uplink providers ran outta UPS, and overnight their small generators ran out of fuel in the big overload ...
...
..... though I have to admin that i enjoy candle light dinners with my wife a lot .... ...of course cold food only :) no microwave, or heater :(
Especially in the rainy season. I often bang my head into my office desk as electricity and net goes after the huge explosion-sound of a nearby thunderbolt hit
i can have ups
in the middle of an ssh session, just after saving a bad config, or messing up a production website's dbconnect include
like 3 years ago, when we did not have electricity for more than 20 hours
hahh
on our servers, leaving 10-12 production machines hanging between
backup ? yeah right
and being new here we calculated (and tested that UPSes were enough for 12+ hours and that should be enough anywhere
ps: now as the rainy season is on the doorstep i am thinking about getting a generator for even home
This is interesting news coming as it does in the week that the CIA is scheduled to run a 3 day netwar exercise called Silent Horizon.
2 .html
http://apnews.myway.com/article/20050525/D8AAFUIO
Am I just blowing smoke here...?
Intolerance for ambiguity is the mark of the authoritarian personality.
i wonder what the nuclear missle officers sitting in their silos were thinking when the lights flickered...
(yeah i know, probably no nuke launch sites near moscow. but still, for a system that operates on hair-trigger, you have to wonder about the mental stability of someone whose finger sits on a whole lot of hurt.)
"Single points of failure are not supposed to exist."
In other words. The internet doesn't route well around economic damage.
That is assuming Putin pushes for the constitution to be changed, as constitutionally he cannot stand in the next election.
He has a constitutional majority in the parliament. He can wipe his ass with the Constitution if he wants to.
"Russia falls back to a form of communism"
Totalitarianism more like. Communism doesn't mean what you think it means.
Yes, totalitarianism, and communism. They generally go hand in hand. Not always, but for the most part. Prime examples are N. Korea, former Soviet Union, and Cuba.
Life is not for the lazy.
Chubais poses a threat to Putin? Please, don't make such jokes :)
Chubais is one of the most hated politicians in Russia. He was the leader of criminal privatisation, he is responsible for "fan blackout's" (don't know how to translate it exactly). There's even a semi-political joke: "It's all Chubais's fault, he alone is to blame".
So if you are intent on some conspiracy theory: this blackout was caused by Chubais himself to support his obsession to deregulate energy market in Russia.
No, "communism" is an ideal: "everyone gets what he wants and everyone does what he can". It's not achievable in practice.
You're thinking about "socialism", and that's a completely different thing.
it depends on exactly what you mean by the internet. and on how critical the system is.
buying two links from the same provider or from providers you know have good peering arrangements and making sure there is good infrasturcture in between is very different buying two links from the cheapest commodity isps and hoping they keep routing to each other properly with no comeback if they don't
similarly you may rely on grid power at home or in an office where nothing too critical is done but if you really need power all the time you will put backup systems in place.
before relying on anything you need to ask yourself whats the likely downtime,how pridictable that downtime is,how much that downtime will cost you, what you can do to reduce downtime and at what cost.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
I agree with you, in fact, that's what I've been saying all along (and getting modded down for it).
Socialism takes place at the local level free from government intervention at it's purist form. Communism on the other hand, it socialism imposed on the people by government. As such, regardless of what work you in a communist country, the government regulates the market out of the aspect "equality". And because everyone gets the same quality of life, there is no incentitive to work harder if the fruits of your labor go unrewarded. At this point, it becomes a downward spiral to corruption and poverty...hence my sig.
Life is not for the lazy.
Yup, but he's also one of the wealthiest and most connected men in Russian political elite. He won't be running for president, that's for sure. But he can get someone else elected. Russian people are sheep, they'll vote for whoever does better PR (as demonstrated by Putin).
I live in an ex-USSR state, leave no tracks on russian sites at all. My primary language is not russian.
:-)
But 99% of the spam I get is in russian. This is outrageous!!
In my entire lifetime, I only got 2 [two] spam emails in romanian - which is my native language.
Eeeh.. still... I think spammers have a very good infrastracture.. because the russian spam keeps coming
The saddest poem
I was clumsy enough to forget to miss an opportunity to tell American Language Center to go to hell!
:-)
Those who know the story, will understand me
The saddest poem
No, last year Chubais supported SPS (liberal political party) and it failed spectacularly, it didn't even get 5% of votes required for party to get seats in parliament.
Belive me, PR of SPS was one of the best. But people simply won't vote for ones who robbed them during 'privatisation' of 90s.
Nah. When you don't control the TV you have no PR. Laymen don't read newspapers, especially liberal newspapers. And TV only shows VVP.