Slashdot Mirror
How a Router's Missed Range Check Nearly Crashed the Internet
Barlaam writes "A bug by router vendor A (omitting a range check from a critical field in the configuration interface) tickled a bug from router vendor B (dropping BGP sessions when processing some ASPATH attributes with length very close to 256), causing a ripple effect that caused widespread global routing instability last week. The flaw lay dormant until one of vendor A's systems was deployed in an autonomous system whose ASN, modulo 256, was greater than 250. At that point, the Internet was one typo away from disaster. Other router vendors, who were not affected by the bug, happily propagated the trigger message to every vulnerable system on the planet in about 30 seconds. Few people appreciate how fragile and unsecured the Internet's trust-based critical infrastructure really is — this is just the latest example." Vendor A, in this case, is a Latvian router vendor called MikroTik.
Vendor B is Cisco btw. Dunno why they were being vague.
I'm sure nobody here would argue with me if I suggested that the internet would be a much safer place without routers.
From long experience most people agree... if it isn't broken, don't fix it.
Reminds me of an old "offensive" fortune quote: Working computer hardware is a lot like an erect penis. It stays up as long as you don't fuck with it.
If you have no clue what offensive fortunes are try 'fortune -o'. They are great when you are stoned, drunk or just bored at work. If you don't have fortune installed then you are clearly on the wrong website ;)
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
except in the kdawson style it was a single link to a message board posting about a router "taking out half the internet." Dupe? Correction? I dont care as long as kdawson is kept away from the site for a while.
It just amazes me how differently presented this story is compared with the previous.
Previous story: kdawson. Current story: Timothy. Do you need any more explanation than that?
Few people appreciate how fragile and unsecured the Internet's trust-based critical infrastructure really is - this is just the latest example.
Yeah. Like how everyone is trusted not to google "google".
You have to have a support agreement with Cisco to get the latest IOS. They won't even give you the last version when your support contract ran out. Also, older routers do not always have upgrades available for various reasons, either they do not have enough space or hardware limitations or Cisco End-of-Lifed it and hasn't bothered.
There's also the "if it isn't broke don't fix it" mentality in the networking world. A new version may fix some bugs but it might add some bugs as well. An upgrade, even if minor, generally means a lot of work testing and reconfiguring before you roll it out. Network engineers are expensive and that time isn't free. Sometimes the devil you know is better then the devil you don't.
In an ideal world it wouldn't be an issue, but when it comes to networking it's NEVER an ideal world. There's always too much to do and never enough budget/manpower to do it. Every network admin probably has 10 things on his mental wishlist right now, upgrades he would like to make, redundant hardware he would like to purchase, failover contingencies he needs to test, etc. Upgrading IOS on an old router in a rack somewhere (and hoping it doesn't blow up in your face) can be pretty far down the list.
The critical bug is with the Cisco routers; a Mikrotik router merely nearly triggered the bug.
It would be possible to trigger this bug with any routing software that does not do range checking on the amount of times the ASN is pretended.
The summary is spreading FUD by making Mikrotik, the only named vendor in the summary, look like the vendor at fault.
The next time someone needs you to fix a computer problem and asks what went wrong, simply give them this article's summary as the reason why, replacing "router" and "Internet" with the the defective part in question. You're also guarenteed to look a bit sharper, too.
"A bug by power supply vendor A (omitting a range check from a critical field in the configuration interface) tickled a bug from power supply vendor B (dropping BGP sessions when processing some ASPATH attributes with length very close to 256), causing a ripple effect that caused widespread global routing instability last week. The flaw lay dormant until one of vendor A's systems was deployed in an autonomous system whose ASN, modulo 256, was greater than 250. At that point, the power supply was one typo away from disaster. Other power supply vendors, who were not affected by the bug, happily propagated the trigger message to every vulnerable system on the planet in about 30 seconds. Few people appreciate how fragile and unsecured the power supply's trust-based critical infrastructure really is â" this is just the latest example."
Mikrotik are known GPL violators, that use a modified Linux (they re-branded that as "RouterOS") and a terribly bad implementation of the BGP protocol..
In some custom community network, where MikroTik has been deployed internally, that stolen-Linux is being hacked to use the Quagga instead of MikroTik's BGP.
In short: that "RouterOS" has been higly unsuitable for the Internet. I can't believe somebody was so stupid to trust it.
Reminds me of a story that Keith Marzullo told our class in a graduate level reliability class. This was back in the days of using UUCP to send email, and the vendor that he worked for had just released a "failsafe" product they were very proud of -- essentially, it was a mail router that could detect if a path went down, and would try an alternate router instead. The company touted it as a bulletproof solution.
So they go to a conference, and set up some routers, unplug some of them, etc., and everything is going fine until they ask an audience member for his UUCP address. UUCP addresses are in the form of host1!host2!host3!username, with the routing for the username explicitly specified... the addresses could thus get quite long. In this case, the guy's email address was over the buffer limit the company's routers used.
Guess what happened?
The mail server tried sending an email to the next router in the chain. The router buffer overflowed and crashed. The reliable server than tried another router... and crashed it. It then went through the entire network, and crashed every single one of the nodes, turning a bug that would have been a single point of failure into a total network collapse.
=)
Yeah, one of my favorite stories from UCSD.
Maybe if they updated their IOS back in 2003 when Cisco came out with the fix they wouldn't have these problems. You wouldn't give an XP user a pass on not updating for 6 years and having a problem, don't give these upstreams any.
-zifr
Untrue. Cisco TAC wil give you the latest firmware for free, provided you tell then n\you need it due to security flaws discovered in your current version. Yoy may need to point to their blletin about the bug, but that should be trivial (http://www.cisco.com/en/US/products/products_security_advisories_listing.html)
Since Cisco almost exclusivly patches current versions due to security bugs, all their IOS are belong to us for free.