Slashdot Mirror
One Broken Router Takes Out Half the Internet?
Silent Stephus writes "I work for a smallish hosting provider, and this morning we experienced a networking event with one of our upstreams. What is interesting about this, is it's being caused by a mis-configured router in Europe — and it appears to be affecting a significant portion of the transit providers across the Internet. In other words, a single mis-configured router is apparently able to cause a DOS for a huge chunk of the Net. And people don't believe me when I tell them all this new-fangled technology is held together by duct-tape and baling wire!"
A couple of Nuclear Subs probably cut an underwater cable...
A router takes out 'half the internet' and I learn this from Slashdot?
Seriously, what is/was the impact? I work for a large e-commerce provider and haven't seen a thing that would indicate a problem today.
The internet's dirty little secret. It's amazing it works at all.
"I'd rather be a lightning rod than a seismometer." -Ken Kesey
There is a post in nanog and on isc.sans.org.
AS 47868 causing AS paths to become too long...
http://www.merit.edu/mail.archives/nanog/msg15472.html
And took out THE _WHOLE_ INTERNET!!!!!
It's true! Ask my wife!
"Speaking the Truth in times of universal deceit is a revolutionary act." -- George Orwell
It must have been the "half the Internet" that I don't use. Which would be an interesting half because many of the sites I visit regularly are based in Europe.
From the thread, it looks like AS 47868 was the route being lost.
http://en.wikipedia.org/wiki/Autonomous_System_Number
Now I really don't know all that much about large-scale networking so maybe someone could explain this to me. What difference does it make if the router is rouge, versus say, green? or black?
Thanks for any insight :)
---- Den ene knappen er powerknapp, den andre er Bender voice knapp "Bite My Shiny Metal Ass"
The AS 47868 decided that they wanted to prepend their ASN about 75 or so times to their BGP announcements. When this got re-populated throughout the rest of the world, a bug in older versions of Cisco IOS still in use on many ISP/NSP networks does not like paths this long. As soon as they saw the prefix with that long of a path, the software terminated the BGP session, resulting in the doorway being closed between the two networks -- So on and so forth throughout the rest of the web.
Since folks on Slashdot seem to like car analogies, I'll just mention that Red Cars Go Faster and assume that the same law applies for routers.
Now I really don't know all that much about large-scale networking so maybe someone could explain this to me. What difference does it make if the router is rouge, versus say, green? or black?
So they announced a route that was, shall we say, malformed. Part of the problem is that due to a Cisco bug (CSCdr54230), some routers choke on it instead of ignoring it. The bug is fixed. It was fixed some time ago. Nonetheless, it's a pretty bad bug, labeled as "1 - catastrophic" by Cisco (in red letters, even). Routers still running affected code versions are having issues.
And it's only at this point in writing my reply that I realize you were taking advantage of a pun by way of misspelling. I'll leave my reply anyway ;-)
"Nature doesn't care how smart you are. You can still be wrong." - Richard Feynman
Can't. It's Monday. No cheezburgers.
They need to replace it with a network that is designed to survive a nuclear attack. Oh wait, hang on....
That's the problem. You shouldn't use rouge on your routers.
I think that a rouged router would possibly be overly promiscuous.
No wonder problems like this can spread like the clap in a port town!
In the free world the media isn't government run; the government is media run.
That's the problem. You shouldn't use rouge on your routers.
They think a rouge router is in vouge, but they're out of their leauge. We should haranuge them! A plauge on them! Rip out their tounges so they cannot aruge! Them and their colleauges. Nothing but demagouges and idealouges I say. There can be no dialouge on this matter. Send them to the moruge!
Are you intriuged by my ideas and want to subscribe to my travelouge?
Oh, say does that Star-Spangled Banner entwine / The myrtle of Venus with Bacchus's vine?
This only broke BGP implementations that are getting pretty long in the tooth now, on a moderately recent version of IOS all we saw is:
Feb 17 05:25:03.731 nzdt: %BGP-6-ASPATH: Long AS path 10026 3356 29113 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 received from xxx.xxx.xxx.xxx: More than configured MAXAS-LIMIT
It was definitely an insane path, our routers were configured to drop anything with an AS path longer than 75, old versions of IOS would often just drop the BGP session ( or even crash with some _really_ old versions ).
I'm sure there will be some red faced network engineers updating IOS or even doing forklift upgrades of old boxes at their edges in the near future.
I am a lawyer and this constitutes legal advice and I shall indemnify you against any losses arising from taking it.
This only took down people running fairly old versions of IOS that didn't patch a known bug.
Did not affect non-cisco.
Did not affect modern versions of IOS
Did not affect old versions of IOS that set the knob to limit the max as-path.
A router takes out 'half the internet' and I learn this from Slashdot?
Seriously, what is/was the impact? I work for a large e-commerce provider and haven't seen a thing that would indicate a problem today.
Well I'm not sure about you.
Personally, I have BIGGER news! A single router in a remote rural US state managed to take down the ENTIRE INTERNETS!!!!
Yes, indeed when I noticed my cat had unplugged the power adapter, I replaced it. Then the ENTIRE internet came back! It was amazing how I single-handedly brought back the whole internets. Al Gore would be proud.
Welcome to Sauronet... One Router to Rule them ALL!!!!
Don't worry, it wasn't a DOS attack. That was just the Internet becoming self-aware.
OK, on second thought, maybe worrying is in order.
Rouge is overpowdered!
The last time I experienced a DOS attack it evolved into Windows. Didn't come out of that one unscathed.
They need to replace it with a network that is designed to survive a nuclear attack. Oh wait, hang on....
Wish I had mod points today. Parent should already be SCORE:5 Funny. Apparently not enough Slashdotters know the history/evolution of the net.
If you're referring to the myth that the Internet was "designed to withstand nuclear attack", perhaps Slashdotters know more than you think.
The Internet was designed to allow distributed control, and to withstand telephone company malice and incompetence. This was a much more useful goal than withstanding nuclear attack.
This "article" is incredibly misleading as nothing has really gone awry. It is just another pointless KDAWSON post. These things are getting REALLY old, KDAWSON.
I work for a tier-3 provider, and if "half the Internet" dies, you are going to hear from a half-brained big media outlet (e.g CNN, ABC) VERY fast.
They need to replace it with a network that is designed to survive a nuclear attack. Oh wait, hang on....
Wish I had mod points today. Parent should already be SCORE:5 Funny. Apparently not enough Slashdotters know the history/evolution of the net.
If you're referring to the myth that the Internet was "designed to withstand nuclear attack", perhaps Slashdotters know more than you think.
The Internet was designed to allow distributed control, and to withstand telephone company malice and incompetence. This was a much more useful goal than withstanding nuclear attack.
One of the early arguments made by DARPA folks to politicians, in order to secure continued federal funding for packet switched network development, was the ability of the network to route around failed or destroyed nodes. They made this argument in the context of the cold war, of nuclear war.
It reality, as you state, this argument had little practical impact on the technical development or evolution of the the network. However, it most certainly did have an impact on the commitment of federal/military funding. This is the origin of the "surviving nuclear attack" lore of the development of DARPANET. It's not a myth. It's real.
Take Obama's current stimulus package as a parallel example. It's not going to solve the recession, but it's being sold as such. And the congress bought into it. Just as this stimulus bill isn't what it's being sold as, most likely DARPANET wouldn't have really given us what it was sold as at one point. Nonetheless, it was sold as such, thus creating the lore that you call myth.
One of the early arguments made by DARPA folks to politicians, in order to secure continued federal funding for packet switched network development, was the ability of the network to route around failed or destroyed nodes. They made this argument in the context of the cold war, of nuclear war.
They made that argument in the context of a widely distributed POTS copper wire network.
The infrastructure of today's internet is fiber based.
And most of that fiber is consolidated in a small number of long backhaul runs.
Remember that grad student whose thesis was classified because he gathered up public documents and mapped out the fiber runs that make up the domestic internet? They classified it (and pulled most of the references he used) because his analysis showed there were a few critical points which, if disrupted, would effectively fracture the domestic internet infrastructure.
The internet isn't nearly as bulletproof as the DoD would like and there isn't much they can do about it short of laying new fiber that skips over the vulnerable points.
[Fuck Beta]
o0t!