Slashdot Mirror
UUNET/WorldCom Backbone Diffiiculties
FearlessFritz writes "UUnet seems to be having a bad time recently. Several sites in the SouthEast of the US have been slow or down. Here is Worldcom's quote from their web page: 'WorldCom is currently experiencing an interruption of service in various hubs in the U.S. We are working to restore a routing anomaly, and making necessary progress toward resolving this disruption in service.' There are several rumors abounding, but the best is that they performed a hardware upgrade that failed. Is anyone outside of the Southeastern U.S. experiencing the effects of this outage? (I am peered to several providers so I can post!)"
Maybe the recent hurricanes knocked over the trailer containing the routers.
"Would it kill you to put down the toilet seat?" -- Maya Angelou
If you're not here, raise your hand! If you can't get online, send an e-mail to the network admin!
Good judgment comes from experience.
Experience comes from bad judgment.
Damn. Make that 2 network techs. The 3rd one is busy moding of /.
Yeah, you look like you're in the center of the problem, beingin metro DC...here's a line out of my traceroute between work and home...
9 0.so-1-2-0.TL2.DCA6.ALTER.NET (152.63.3.194) 1234.111 ms 1194.558 ms 1206.814 ms
The times just get worse from there on out...
This space for rent. Call 1-800-STEAK4U
Hub: Normal
Outages: Normal
this is from their network status page, i try to abstain from being a smart-ass but outages are normal?
-tid242
With a few exceptions, secrecy is deeply incompatible with democracy and with science. --Carl Sagan
a whole lot of red over at the InternetTrafficReport any other good informative sites?
May this post be indexed by spiders, and archived for all to see as my Internet epitaph.
I think his petswarehouse.com site's had so much traffic over the last couple of hours it's exploded and caused a huge chain reaction ;-)
Code, Hardware, stuff like that.
. when our backbones fail... what do we do?
Slither around on the floor?
I've had enough abrasive sigs. Kittens are cute and fuzzy.
There's been discussion of this on the NANOG list, and my DS3 in Chicago was taken down hard by this. Physical layer okay, but traffic died once it was two or three hops into UUnet/Worldcom's core. First outage was from 2am to 8am, second outage from approx. 10:45am (CST) to 2pm. The master tickets for this outage are 651744 (DS1 and below) and 651751 (DS3, OC3 and above). I just got off the phone with Worldcom's NOC and the story I got is that all the border routers that took a dive are back up save a few that they're bringing back up here in Chicago. Worldcom has provided confirmation that the Reason For Outage was a wildly unsuccessful BGP config propagation.
. We've got computers, we're tapping phone lines, you know that ain't allowed - Talking Heads, "Life During Wartime"
"Diffiiculties?"
Oh, man, it's affecting data transmission quality now.
-Waldo Jaquith
Following is WorldCom's maintenance announcement about today's work, which I recieved because WorldCom is my company's broadband ISP.
During the Normal operations window on Oct 3, 2002
WorldCom will be performing the following scheduled maintenance
activities.
This activity is scheduled to take place from 3:00 a.m. to 6:00 a.m.
(local hub time) in the contiguous US and elsewhere from 3:00 a.m. to
7:00 a.m. (local hub time) and may affect your connectivity. The
following
customer ID will be impacted: XXXXXXXXX.
If you have any questions, please contact our local Customer Network
Support Center. Please reference the internal ticket number 645346.
Quality System Management-Global Maintenance Planning
Worldcom (http://www.uu.net)
1(800) 900-0241 / +1(703) 886-5440
WorldCom United States 1-800-900-0241 (select the following options in
order: 2, then 4, then 1)
WorldCom Denmark (45) 80.30.50.50
WorldCom Italy (39) 02.3600.1887
WorldCom Sweden (46) 8.750.88.50
WorldCom Switzerland (41) 1.580.86.11
Anyone who's done any kind of IOS upgrading on some of the upper-end Cisco routers and Juniper routers knows that the upgraded images aren't always the most stable items around.
At one point, there was a severe outage at Genuity referred to as "Black Tuesday", when an IOS upgrade sunk a majority of the network and caused a ripple that made for a really shitty morning.
That was a few years ago, though. I can't go into the specifics of the RFO...but the failure was a very visible issue which resulted in modifications to the testing and change management processes.
Unfortunately, sometimes testing production software doesn't sufficiently break until actually put into production.
// Agent Green (Ian / IU7 / KB1JQO)
// IEEE 802.3: All 10base Are Belong To Us
The worst part of an outage like this is the users always blame you for any connectivity problems. "I can't get to the D&B website, when are you going to have it fixed?", you patiently explain the circuits to your provider are fine, your provider's circuts are fine, and the problem is either with D&B's network or their provider. "Yeah, whatever, when are you going to have it fixed?", lusers are utterly hopeless, unfortunately you have to at least humor them when they sign your paychecks.
Happy Fun Ball is for external use only.