Slashdot Mirror
Cisco Routers to Blame for Japan Net Outtage
An anonymous reader passed us a link to a Network World article filling in the details behind the massive internet outage Japanese web users experienced earlier this week. According to the site faulty Cisco routers were to blame for the lapse, which left millions of customers without service from late evening Tuesday until early in the morning on Wednesday. "NTT East and NTT West, both group companies of Japanese telecom giant Nippon Telegraph and Telephone (NTT), are in the process of finalizing their decisions on a core router upgrade, according to the report. The routing table rewrite overflowed the routing tables and caused the routers' forwarding process to fail, the CIBC report states."
Japanese police have put out an APB for some guy named "Dijkstra."
...They will not be anymore the dot in .jp
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
For those that have used JunOS before, im sure are all saying.
"A Juniper router is like my girlfriend.. It will never go down on me."
I think it's funny. Usually, when you open Cisco TAC about a "faulty" router not forwarding traffic anymore, Cisco will tell you it's your config's fault if it's not working properly.
Usually what happens is that the router doesn't have enough memory to store all the CEF (Cisco Express Forwarding) info, causing the router to not forward packets for certain subnets. I've seen it happen often enough to know. While Cisco is right, the problem is caused by a lack of memory for the config, I think it shouldn't stop forwarding the packets all together (as in stop using CEF if the table gets out of hand).
While I think Cisco is not completely to blame (badly scaled networks, not upgrading routers in time), it sucks that this will hit them. There are better solutions out there, but I have to say that Cisco's support is quite good and they're pretty fast. I work in an all-Cisco environment (for the routers) and they've been fast whenever we needed a router analyzed.
-- www.globaltics.net
Political discussion for a new world
"The routing table rewrite overflowed the routing tables and caused the routers' forwarding process to fail, the CIBC report states"
Ok.. That says to me that their routing tables got really big, the routers ran out of memory... Or.. they Had a prefix limit set, and it kept dropping the BGP session(s)...
If either of the above is true, properly designed filtering of the prefixes they send/receive to their BGP neighbors would have resolved this outage... It sounds like someone may have been incompetent, and they are trying to pawn off the "ownership" of this outage on Cisco.
Either that, or its a major IOS bug, and the article's author just sucks and didn't mention that..
Being a current CCIE, and having extensive experience with both vendors boxes, I wouldn't use anything other than a Juniper for core infrastructure, and I'm never going back to cisco kit..
To be fair Cisco is untouchable in the enterprise class with their CPE's..
- Did a manual change exceed a design limit? If so, why wasn't the manual change rejected? (If it was rejected, that's not a fauilt, it's user error)
- Did an automatic change (like fail-over) applied to a valid configuration produce an invalid one? If so, did the routers report this, generate some kind of trap or alarm? If so, I guess the problem is a bit nebulous; maybe a monitoring failure, but maybe the system could have issued warnings that certain kinds of possible failover could exceed implementation limits. Hard to know without more detailed information.
- Did an automatic change silently produce the wrong result (like forwarding some traffic and not other traffic) *without* generating a trap or alert? If so, I would certainly call this a fault (bug). But the article doesn't contain enough information to point conclusively in this direction.
The event is big news, so I guess NW felt they had to say *something*. But while I'm no big fan of Cisco gear, it looks to me that the explanation is as likely to be human error as equipment faults or bugs. One potential cause of problems in big routers is that the high-level software's view of the state of the routing engine gets out of sync with the actual state of the ASICs. I wonder if that happened here. My guess is that once more details of the incident emerge it will turn into a not-news story.I think that it's great when a company is blamed on having poor products when it's really the company using them (in this case, NTT). The way the article is presented seems unfairly biased. The problem isn't with cisco products here but the lack of knowledge on scaling them properly. The headline is similar to saying something like "Ford Motor company cars involved in most car accidents historically". A properly designed network with just about any vendor, especially cisco, would have avoided this issue.
Hacked by Chinese kekeke ^_^
Cisco routers to blame for most of the rest of the internet's non-outage.
ccalam - acoustic versions of new songs.
This was certainly a problem with slightly older Cisco kit, such as 6500s with Sup720a cards. Their TCAM memory (that holds prefix+destination tuples in a form of cache) overflowed as the internet approaches 245,000 routes. Once there is no more space in TCAM, many Cisco architectures fall back to processor routing. That means that when traffic that was switched in hardware starts hitting the CPU, the box falls over whimpering for mercy.
If NTT had been following Cisco mailing lists, or keeping up to date on what their salesmen had been telling them for several years, they would have seen this problem looming and changed their routing structure or at least upgraded the processors for something with slightly more TCAM. The size of the internet is not going to stop growing because many companies chose to go with underpowered Cisco kit. The internet will continue to grow by 12,000 to 17,000 routes per month, accelerating over the next few years as IPv4 space becomes exhausted and de-aggregation becomes the norm.
This is one of my long standing grudges about Cisco design. They always are designing their core routers to be just slightly ahead of the size of the internet, forcing people to upgrade within a few years. Designed obsolescence is the term. Even their new CRS1 platform will fail over to CPU near 512,000 routes (0x80000), or sometime around the end of 2008 to mid 2009. By then, they'll probably have an expensive upgrade path for customers that will hold for just another year or two.
It's not just Cisco kit that is going to have problems over the next few months. By the end of June the internet will be at 256,000 routes (really 262,144 or 0x40000), which will be a problem for some other manufacturers. Some are starting to fail at 0x3C000 (245,000) routes, some already failed at 0x30000 last year.
On the plus side, the OpenBGPd crowd doesn't suffer from this, since their code is all CPU switched (but using very clever and efficiently coded routing tables) so their routing table is limited only by memory. But an OpenBGPd machine will never have the raw efficiency of a VLSI based hardware solution.
A quick look at my local looking glass shows 233,979 routes on the internet this morning.
the AC
Hemos is like...sci-fi fans;he thinks technology is cool, but he hasn't bothered to understand the science it's based on
The problem was that the internet had grown beyond the capacity of their core routers, hence the core router upgrade that was "in progress". The headline should actually read:
OLD Cisco Routers to Blame for Japan Net Outage (with only one 't' in "Outage', just as in TFA!)
Hey folks, don't stop CIDR'ing routes just because there seems to be enough routing table space "right now"!
#naabhaprzrag, #sverubfr-000, #agi-fcbafberq, negvpyr[pynff*=' negvpyr-ary-'] { qvfcynl: abar !vzcbegnag; }
I read this as human error and lack of planning that some admins have successfully blamed on hardware.
Ford to blame for all car crashes involving Fords...
Smith & Wesson to blame for misuse of firearms...
Yawn... Nothing to see here... move along...
I have a friend who works for Cisco, setting up and configuring routers remotely via telepresence, all over the world. I know that lately they have been working on some routers in Japan.
:)
*IF* (and I don't know for sure that this is the case, so it's a big IF) the project this person has been working on relates to the problem in this story, then I would say that your guess of 'human error' is likely a very large part of what happened.
I say this because my friend has filled me in on some of the stories relating to language barriers during the work they are doing. While the stories are very humorous (and that was their sole reason for telling me said stories), I can see where they would easily contribute to a situation where mistakes could be made.
From the stories, it is extremely evident that English is *very* much a second language on the Japan-side of things. To the point that it would be smart to have a person there whose first language and culture would be that of the company employees who were doing the install.
I'd guess that the lesson here is that globalization has a lot of bugs to be worked out still...
250K is quite lame. I just tested a bit over 1 million installed routes on a Juniper in my lab. ... a 5 year old m-series at that.
This is no shock: Juniper's first innovation was the use of high speed ram rather than tcam for tcam lookup so route table scaling has never been a problem for them...
Marketing on the other hand... geesh those cartoons still give me nightmares.
...to ficks mye badd speling. Cant seam two finde "outtage" (orr "expirey") inn mye dikshunairy...
'routers went down .. after a switchover to backup routes triggered the routers to rewrite routing tables'
"At this time, Cisco and NTT have not determined the specific cause of the problem"
davecb5620@gmail.com
Did you mean: "FUCK DSLAM"
No sig for now.
Why would it be neccisary to run full BGP routes in all of your routers anyways? Couldnt you rull partial routes in the edge routers and save your full routes for your core and peering points? 2 -4 thousand routers is a lot to go down at once.
"When they invent bitch slaps that can go through a monitor you better f'ing duck" --deft (253558)
You're doing an Apples and Oranges comparision. Cisco's IOS is far more dedicated to a specific set of tasks than the other notable OS's. So yes, one would expect far less bugs to be visable. That doesn't mean they aren't there; just that they haven't been discovered.
Having worked at many of the companies which supply OS's, Cisco is, IMHO, the worst. They go for lots of cheap talent. The common theme is to hire lots of low paid talent rather than focusing on getting the best and the brightest. And it shows. Things which shouldn't happen, do. And the general level of code quality is below average.
The general development infrastructure sucks badly as well. So much so, that they've actually developed bandaids to make it semi-palitable.
This isn't to say that they don't have some good talent there. They do. But they are a minority, and are hindered by the general red-tape which keeps those folks from having a greater impact.
Sun, on the otherhand, had the best development environment, talent and infrastructure that I've ever seen, back in the 90's. I've heard that things have fallen off a bit since then, but I really can't say.
Anyway, the bottom line here is that I wouldn't at all be surprised if Cisco screwed up on the basics. The cheap talent is biting them daily in ways the top management can't see, and it all adds up eventually. Things like this are to be expected, and I also expect it to get worse over time, not better.
1. Local status (Am I alive)
2. Path (can I get from me to you, what is the quality of the path?)
3. End point (are you there?)
If at any time you let the number of paths and interconnects overwhelms you. Get a new job. You've lost control. Draw pictures of the network. When you have an outage start looking immediately at what you have connectivity with and what you don't. Large data centers can get complex in their interconnects. Divide it up into "blocks" verify a block and move on.
The biggest problem in a situation like this is that I'm willing to bet the techs were wasting their time trying to figure out why the network went down. Who cares why. You need to quickly assess what is down. What you can do. What you can't do. You need to know what is normal and what is not. If you don't a situation like this can happen.
The worst thing that can happen is if the network is divided into "territories." Usually in a case like this people spend more time trying to blame the other guy then they do finding the cause of the problem. Finally design. Somewhere along the line some pencil pusher decided that a single point of failure was economically feasible. The techs were willing to sheep right along, the Sr Admin was played politics and didn't rock the boat.
In the end. The techs blew it. The after action report and follow up will tell the final tale.
I'm sorry, I'm to tired to be witty at the moment so this message will have to do.
WTF did parent get modded as a troll? Remember the 30% decrease in spams back when the earthquake earlier this year knocked out most of southeast Asia? I agree; keep them offline.
Why do people still buy Cisco's junk anyway? Ugh. Grow up, they're dinosaurs in the market place now! force10networks.com By happy there's a new networking player in town that trumps its predecessors.
Someone wrote "Cisco" in their Death Note!