Slashdot Mirror
UUNET/WorldCom Backbone Diffiiculties
FearlessFritz writes "UUnet seems to be having a bad time recently. Several sites in the SouthEast of the US have been slow or down. Here is Worldcom's quote from their web page: 'WorldCom is currently experiencing an interruption of service in various hubs in the U.S. We are working to restore a routing anomaly, and making necessary progress toward resolving this disruption in service.' There are several rumors abounding, but the best is that they performed a hardware upgrade that failed. Is anyone outside of the Southeastern U.S. experiencing the effects of this outage? (I am peered to several providers so I can post!)"
From the reports from one of our managed networking providers the trouble started with the DC peering center and moved outwards. They lost a couple OC12's that still haven't come back and have had other lines up and down all day.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
We have a T1 Worldcom line here in Toronto, it's fine for the most part, but we have some servers hosted (?somewhere?) in the US with Level3, and we've had a horrible time connecting to them today. Through my home cable Internet connection, the connections to our servers are fine.
I'll have something intelligent to add one of these days...
You want http://www1.worldcom.com/us/tools/noc/status.xml
News Performance: Normal
DNS Service: Normal
Backbone: WorldCom is currently experiencing an interruption of service in various hubs in the U.S. We are working to restore service as quickly as possible.
Dialup: Normal
Hub: Normal
Outages: Normal
One of the big problems here is that Woldcom still operates various units as separate entities, virtually no integration has been done to get UUnet working with MFS working with MCI. It's a lot of fun troubleshooting a circuit and having techs tell you "the problem is with MCI, I work for MFS." !!!!! They all work for Worldcom!
Okay, rant mode off.
. We've got computers, we're tapping phone lines, you know that ain't allowed - Talking Heads, "Life During Wartime"
Anyone who's done any kind of IOS upgrading on some of the upper-end Cisco routers and Juniper routers knows that the upgraded images aren't always the most stable items around.
At one point, there was a severe outage at Genuity referred to as "Black Tuesday", when an IOS upgrade sunk a majority of the network and caused a ripple that made for a really shitty morning.
That was a few years ago, though. I can't go into the specifics of the RFO...but the failure was a very visible issue which resulted in modifications to the testing and change management processes.
Unfortunately, sometimes testing production software doesn't sufficiently break until actually put into production.
// Agent Green (Ian / IU7 / KB1JQO)
// IEEE 802.3: All 10base Are Belong To Us
I run a small ISP in Portland, OR who's been down for two days because my network got deleted from the RADB from which the backbone ISP builds their routing tables. It's been working fine since I started using it almost a year ago, and magically stopped working the evening of Oct 1 (first of the month, in the evening when the backbone updates their tables), so I think a policy change topside is the "routing anomaly" that has barfed up everything. At least I'm supposed to be back online later this evening...