Slashdot Mirror
UUNET/WorldCom Backbone Diffiiculties
FearlessFritz writes "UUnet seems to be having a bad time recently. Several sites in the SouthEast of the US have been slow or down. Here is Worldcom's quote from their web page: 'WorldCom is currently experiencing an interruption of service in various hubs in the U.S. We are working to restore a routing anomaly, and making necessary progress toward resolving this disruption in service.' There are several rumors abounding, but the best is that they performed a hardware upgrade that failed. Is anyone outside of the Southeastern U.S. experiencing the effects of this outage? (I am peered to several providers so I can post!)"
It will take a while for the 3 network techs that havent' been fired yet some time to fix this.
(the voice from the movie WarGame after the first wave of Russian missile launches hit:) "We're still here!"
I have had very slow internet access most of the day in the upper midwest. The problem started at about 8:15 CT this morning.
Im in Alexandria, VA, and I've had horrible service all day long. I could get local sites, but nothing far away. Pinging slashdot wont even work half the time :(. And no google....
Maybe the recent hurricanes knocked over the trailer containing the routers.
"Would it kill you to put down the toilet seat?" -- Maya Angelou
I can verify this issue. I work for a company out of the DC area and our data lines are provided by UUNet. I live in Kansas though and do all my work via SSH to the servers in DC. This morning my SSH sessions would randomly seem to hang. After a bit of investigation I started noticing it was taking like 600ms per hop once I got to the UUNet network. So that is North-East there.
Unstable Apps: Our Android Apps Don't Suck
If you're not here, raise your hand! If you can't get online, send an e-mail to the network admin!
Good judgment comes from experience.
Experience comes from bad judgment.
wasn't MCI/Worldcom/C&W involved in lawsuit a few yrs ago for similar problems, but they went on for 3 days ?.. back then they got sued for loss of business, imagine whats gonna happen now.
You would think they would have learn't. But they're still hiding information from people.. great. Maybe it's gone with the 6 billion dollars...
Hub: Normal
Outages: Normal
this is from their network status page, i try to abstain from being a smart-ass but outages are normal?
-tid242
With a few exceptions, secrecy is deeply incompatible with democracy and with science. --Carl Sagan
a whole lot of red over at the InternetTrafficReport any other good informative sites?
May this post be indexed by spiders, and archived for all to see as my Internet epitaph.
I think his petswarehouse.com site's had so much traffic over the last couple of hours it's exploded and caused a huge chain reaction ;-)
Code, Hardware, stuff like that.
I work at a small hosting company and our UUNet connectivity (Central California via Anaheim hub) has been screwed since around 6am pacific time. Up and down all morning with latency between 500 and 2000 ms when it is up. Yay worldcom.
...seems to be fine.
Oh, come on. Laugh! You know you want to.
Check it out here
http://www.merit.edu/mail.archives/nanog/msg040
nothing concrete and MIDS doesn't show anything on the weather reports (not that it means anything).
From the reports from one of our managed networking providers the trouble started with the DC peering center and moved outwards. They lost a couple OC12's that still haven't come back and have had other lines up and down all day.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
We have a T1 Worldcom line here in Toronto, it's fine for the most part, but we have some servers hosted (?somewhere?) in the US with Level3, and we've had a horrible time connecting to them today. Through my home cable Internet connection, the connections to our servers are fine.
I'll have something intelligent to add one of these days...
So they are having routing problems and you put a link to their web page up. Nice of you.
Conceptually, the logic states that there should be multiple backbones through multiple geographic areas, such that a failure of one provider could be dealt with by routing traffic through the alternate backbone. Realistically this is difficult and expensive, and the primary reason that there are very few top tier connections running across the united states.
If you look at the map from 1992 (NSF Net | XO OC192 Network), you'll notice that there really are only 2 main paths from east coast to west coast. The southern path is probably at least slightly affected by the incoming hurricaine, and the northern path seems to be overloaded or failing for some other reason.
Precautions? Make sure the hardware is sound and easily replaced, and that alternate routes are available in case of failure. The problem is finding alternate routes that aren't completely congested due to the failure.
Mooniacs for iOS and Android
. when our backbones fail... what do we do?
Slither around on the floor?
I've had enough abrasive sigs. Kittens are cute and fuzzy.
There's been discussion of this on the NANOG list, and my DS3 in Chicago was taken down hard by this. Physical layer okay, but traffic died once it was two or three hops into UUnet/Worldcom's core. First outage was from 2am to 8am, second outage from approx. 10:45am (CST) to 2pm. The master tickets for this outage are 651744 (DS1 and below) and 651751 (DS3, OC3 and above). I just got off the phone with Worldcom's NOC and the story I got is that all the border routers that took a dive are back up save a few that they're bringing back up here in Chicago. Worldcom has provided confirmation that the Reason For Outage was a wildly unsuccessful BGP config propagation.
. We've got computers, we're tapping phone lines, you know that ain't allowed - Talking Heads, "Life During Wartime"
The problem is pretty clear - they are working to restore a routing anomaly rather than correcting the ones they still have. I would tell them that if they continue to restore anomalies things will only get wrose, but I can't get through to them.
I'm an American. I love this country and the freedoms that we used to have.
"Diffiiculties?"
Oh, man, it's affecting data transmission quality now.
-Waldo Jaquith
You want http://www1.worldcom.com/us/tools/noc/status.xml
News Performance: Normal
DNS Service: Normal
Backbone: WorldCom is currently experiencing an interruption of service in various hubs in the U.S. We are working to restore service as quickly as possible.
Dialup: Normal
Hub: Normal
Outages: Normal
One of the big problems here is that Woldcom still operates various units as separate entities, virtually no integration has been done to get UUnet working with MFS working with MCI. It's a lot of fun troubleshooting a circuit and having techs tell you "the problem is with MCI, I work for MFS." !!!!! They all work for Worldcom!
Okay, rant mode off.
. We've got computers, we're tapping phone lines, you know that ain't allowed - Talking Heads, "Life During Wartime"
The network outage was unrelated to WorldCom's bankruptcy, and the cause is unknown, Burns said.
I have this image that in order to save money, the are routing all of the Southeast's traffic through and AOL dialup using Windows internet sharing.
Do not taunt Happy Fun Ball(TM)
I can't even get to the Pets Warehouse site!
echo 656472616c73746f6e406d61632e636f6d0a|xxd -r -p
I will give you your so called "Internet" back as soon as you declare me King of the World! Pinky, gnaw on that wire some more.
But Brain, it hurts my teef.
Pinky! Destiny awaits us!
Narf!
Brain taps foot, frowns at Pinky standing alone covered in electrical char with a wire in his mouth.
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
Following is WorldCom's maintenance announcement about today's work, which I recieved because WorldCom is my company's broadband ISP.
During the Normal operations window on Oct 3, 2002
WorldCom will be performing the following scheduled maintenance
activities.
This activity is scheduled to take place from 3:00 a.m. to 6:00 a.m.
(local hub time) in the contiguous US and elsewhere from 3:00 a.m. to
7:00 a.m. (local hub time) and may affect your connectivity. The
following
customer ID will be impacted: XXXXXXXXX.
If you have any questions, please contact our local Customer Network
Support Center. Please reference the internal ticket number 645346.
Quality System Management-Global Maintenance Planning
Worldcom (http://www.uu.net)
1(800) 900-0241 / +1(703) 886-5440
WorldCom United States 1-800-900-0241 (select the following options in
order: 2, then 4, then 1)
WorldCom Denmark (45) 80.30.50.50
WorldCom Italy (39) 02.3600.1887
WorldCom Sweden (46) 8.750.88.50
WorldCom Switzerland (41) 1.580.86.11
Anyone who's done any kind of IOS upgrading on some of the upper-end Cisco routers and Juniper routers knows that the upgraded images aren't always the most stable items around.
At one point, there was a severe outage at Genuity referred to as "Black Tuesday", when an IOS upgrade sunk a majority of the network and caused a ripple that made for a really shitty morning.
That was a few years ago, though. I can't go into the specifics of the RFO...but the failure was a very visible issue which resulted in modifications to the testing and change management processes.
Unfortunately, sometimes testing production software doesn't sufficiently break until actually put into production.
// Agent Green (Ian / IU7 / KB1JQO)
// IEEE 802.3: All 10base Are Belong To Us
" Is anyone outside of the Southeastern U.S. experiencing the effects of this outage? "
So let me get this straight: You want the users of Slashdot to report internet outages? And how are we supposed to rule out that Slashdot didn't cause them?
Significant increase in demand.
Redhat 8.0 ISO's
Mandrake 9.0 ISO's
UT 2003
Its obviously NOT a backbone problem....Redhat 8.0 just came out a few days ago...of coarse traffic is going to be nuts, and slower than hell, ITS 5 CD'S NOW!(including source, mind you)
In college, really poor, need a flatscreen.
The worst part of an outage like this is the users always blame you for any connectivity problems. "I can't get to the D&B website, when are you going to have it fixed?", you patiently explain the circuits to your provider are fine, your provider's circuts are fine, and the problem is either with D&B's network or their provider. "Yeah, whatever, when are you going to have it fixed?", lusers are utterly hopeless, unfortunately you have to at least humor them when they sign your paychecks.
Happy Fun Ball is for external use only.
While the Wired News article says that some people were speculating that it was the Slapper worm, other people were speculating that it was a fiber cut, but it first quotes the UUnet page which says they're having a routing anomaly and that it was affecting multiple gateways. That means it's not likely to be a cable cut, because that would be more localized, and it's also not likely to be the Linux worm because the routing stuff isn't happening on Linux boxes - it'd be either Cisco or Juniper, and I'm not aware of any reports that the worm affects those platforms.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
I run a small ISP in Portland, OR who's been down for two days because my network got deleted from the RADB from which the backbone ISP builds their routing tables. It's been working fine since I started using it almost a year ago, and magically stopped working the evening of Oct 1 (first of the month, in the evening when the backbone updates their tables), so I think a policy change topside is the "routing anomaly" that has barfed up everything. At least I'm supposed to be back online later this evening...