Slashdot Mirror
AT&T vs MCI on Network Outages
James Ensor writes "
Cnn.com is running an interesting article comparing MCI Worldlnet's recent frame relay outage to an outage that AT&T had last year, comparing their method's of handling the situation. Good Reading. " It's fun to watch major corporations blunder about. Unless it's your connection that's down.
It finally happened. I knew one day I'd browse on over to Slashdot and find a naughty story about the company I work for. Well, I can't say they weren't asking for it.
Yeah, the point about all the companies within MCI Worldcom rings true. Like the MCI local network, MCI long distance network, Worldcom long distnace network, and the Worldcom local network (which is actually the MFS local network and the Brooks fiber local network). Then you've got a layer on top of that, like UUNet. Not all that straightforward. They are working on these issues, though.
Responsibility issues? They are there. You should see the hoops I have to go through just to get someone in the same building to investigate a problem within our own LAN. MCI highly emphasises "the procedure".
But the story that Slashdot should be covering isn't the frame network outage. It is MCI's agreement to sell off their IT department to EDS. Most everyone assumed it has gone through. Guess what? They made the announcement 6+ months ago, and the deal (along with IT) is hanging in limbo.
Will we work for EDS? MCI? Do we get our current stock options? Benefits? Policies? Management is saying absolutely nothing. Rumors are flying that the negotiations are going bad and neither side wants to walk away because of a hefty $$ penalty. Even an arbitrator hasn't helped. (That's right, an arbitrator just to reach an agreement. Heaven help both sides over the next five years.)
The entire IT department of a major corporation that can't even say what conditions they will be working under next month. How smart is that?
The main problem is that EDS is a very bad fit for a company like MCI Worldcom. EDS is great for an organization that is low-growth, low-innovation, and reliability situations aren't critical. Say, a local government. But a phone company? Oh boy.
I can't say I know an employee that is enthusiastic about the merger. The only management that claims to love the idea is from the VP level and up.
it is short for people. u know just leave some letters out. learned that from the jews.
We switched our ISP's from MCI to Telwest, MCI left our T1 on for 18 months before disconnecting it. (Then tried to bill us!)
Then do tons of documentation so it never happens again!
I work daily with the technical support team. Their job is to solve problems, not pass the buck. And if they cant fix it, it goes up the chain of support, And yes all the way to Frank Ianna.
Our goal is no more than 5 minutes of unplanned outage a year. Thats our motto at the office. (Damn good one also)
AT&T. We were using MCI for our T1 services and it seemed like everyweek they had some router in Atlanta that was on fire, had grown legs and walked off or had been somehow run over by a bus. Honest to god. They said a bus hit their relay station. We have been using AT&T now for almost 5 months and haven't had a hiccup, they aren't the fastest to get out here and install the stuff but they definately have the better of the two services.
--"Karma is justice without the satisfaction"
If the CBT can't plan ahead and shell out a little extra cash so they can be multihomed, they deserve whatever crap they get. I wonder what happens to them when NT crashes. Does the whole place go belly up or do they actually have some kind of HA setup.
-matt
i too have seen this at several companies and it is very sad to see adults act like 5 year olds around a broken cookie jar.
bottom line: lazy people that can't prevent disasters can only find scape goats. stay away from them or they'll pick YOU!
get your dirty sig off me, you filthy APE!
I agree MCI Worldcom should take the blame. They should have focused more on the customer's needs and backed out the software to get the network back up. It was clearly the fault of upper management for listening to the vendor and his "assumptions" and "recommendations". The customer should be faulted for blind faith, not the vendor.
If the sales reps had information and decided not to contact their customers, I'm not sure if that's a management problem or a weasely sales sales person problem. Either way, there wasn't very much information available because not very much was known or understood about the problem. For the longest time (the first week or so)Lucent even claimed it wasn't a result of the software upgrade!
Same here. I work(ed) (still part time when they need me) for a small ISP (no T3's there anytime soon) that used Sprintlink for backbone and Sprint for local service. First, their local guys kick ass. After all the local telco jokes, these guys were a pleasant surprise. They knew how to do it and could deliver a channelized T1 or PRI a week within ordering, 2 weeks in the worse case. Another ISP I have had contact with in GTE land has to order 3-4 months ahead and the usual reaction is T-what?
Sprintlink has been very reliable. We had a little trouble with not enough bandwidth leaving the local POP we connect to, but they got it figured out eventually. In fact, we were going to replace them with someone cheaper, signed the contracts, got the new feed in, and after testing it, are trying to get out of the contracts. They sucked. Sprintlink is well worth the extra money.
Just like the company I used to work for. Customer has a problem, x'fer them to someone else. Even though our department most likely could finger the solution out faster than the "correct person". Kind of like "The buck stops over there, besides it's not my responsibiltiy."
It seems like in the race to save a dollar companies have forgotten that most folks will pay a little extra for good service. Perhaps they should teach that concept in business school. Wait, I graduated from one and they didn't teach that. To damn busy doing SWOT analysis, Gannet charts and critical path analysis to worry about User Experience. IBM and AT&T haven't been around forever and a day letting their customers go away mad.
My man, you are a much braver person than I could ever be.
I thought working for a large companies would insure stability in your career but with the woes of MCI (does our frame relay and one T-1 from UUNet) and SGI (go back to Silicon Graphics) who needs that type of instability?
On the other hand, my current small company is trying to write up policies and procedures so I guess I will be looking for another job soon.
ChozSun [e-mail]
ChozSun
ChozSun.com
Where does Sprint fit in to all of this? We've been buying several T1s and ISDN lines from them for a couple of years and have had almost no downtime. Even when we do, a quick call to their TAC either gets it fixed or gets me an explanation (talking to the actual techs who work on things .. not some brain dead support rep). In fact, I have a full DS3 feed on order from them right now. I don't know everyone else's experience, but we've been very happy with their service.
"FORMAT C:" - Kills bugs dead!
I have a lot of friends that are AT&T employees in R&D. On day 8 of the MCI outage, I mentioned the MCI outage to them. They hadn't heard about it and were damned pissed about it also. After April 1998, when they saw AT&T outage on the front page, etc. and all the heads rolling around them, they couldn't believe that MCI wasn't being eaten alive. 8 days! Then MCI goes and blames it on Lucent. That really got these guys going since Lucent all but shares buildings with them (actually they did up until 3 months ago).
Thank god one of the journalist crowd has a clue about this and published. Very impressive.
I was reading an article about the outage on news.com on day 9 - when I got one of those "Hi, are you happy with your current long distance service?" calls. It was from Qwest, a company that I haven't heard much dirt about yet. Yet.
My old housemate had moved to Europe and they offered decent international rates, so for once I said 'yes' to one of those telemarketers that I usually hate being bothered by.
I did tell them that the only real reason that I was switching was because MCI really messed up. Of course, I was talking to a 3rd party telemarketer who probably has 1000 resumes our trying to get a real job and really doesn't care about telecommunications in the least bit.
But it made me feel better.
.
And besides, there's even a beta of a driver for the thing, if you happen to be running OS/2
--Matthew
Chris
Chris
So Buddha walks into a pizza parlor and says: "Hey, make me one with everything."
AT&T Worldnet is AT&T's Internet Service (One of the top ranked national ISPs, too). Perhaps there's where the misinformation is coming from.
,'s in e-mail address with .'s.
j-a-w-a-d------------------------------
replace
i dont display scores, and my threshhold is -1. post accordingly.
Discuss
After a company grows beyond a certain size, it gets increasingly tough to just "wing it"...
Maybe you mean something more sinister by "policies and procedures"?
Anyone notice, that while talking about frame relay, they have a Centronics (SCSI/Printer) cable
picture on the article?
Just because you disagree doesn't make it offtopic or flamebait.
Did AT&T have a monopoly at one time and learned from it (the hard way)? PPL and Companies should learn again that "The Customer is the King", as we Germans sometimes say.
This rang somewhat true, and in the years since, I watched AT&T live up to this accusation. Until now...
I'm wondering if I was a bit too eager to write them off. Maybe AT&T actually does have a future...
I can see the fnords!
...this sort of thing is likely to continue in this day and age of mega-mergers. AT&T did the Right Thing(tm) and I'm pleasantly impressed. I guess it is possible for a 600 pound gorilla to handle a situation intelligently...
censorship is a form of noise, which actively seeks to drown out content with silence - Crash Culligan
They have a "just enough" attitude to their services. Just give your customers as little as possible and continue to overcharge them. Outages are one thing, how come they refuse to upgrade overloaded networks?
Putting blame on others is also quite common.
I get that impression from many American companies. An embedded company called TERN gave me similar BS at a previous job.
Their development tools were antiquated, the board was defective and they refused to replace it. Transferring new code kept giving CRC errors, and sometimes it would get through, but with errors. They gave me a similar runaround, claiming that all of our computers, including my Dell Dimension XPS 200, had "nonstandard" serial ports that couldn't keep up with 57.6k. Yeah right.
I finally got to the "CEO" (small company) who told me that GM had the very same problem with 100 different new computers, and that all of those computers were bad.
Yeah, right.
We hauled the boards out of both of the projects we were working on and replaced them with cool, reliable ZWorld boards.
Development time shrunk immensely, and reliability was great. Their tech support was also extremely good, they would email me code snippets and everything. They also encouraged messing around with their libraries, I thought that was excellent and found them well written and easy to hack.
Hmmm... Parallels the referenced article quite well. Goes to show, if you're not satisfied, walk. It makes all the difference.
Myself and a few thousand other people were completely left in the lurch when MCI sold its dial up service to Cable and Wireless...Neither company had a full accounting of their subscribers, but that didn't keep both sides from debiting money from my checking account for services never rendered! I had to close my bank account to stop it!
What's really sad is the freakin' Chicago Board of Trade got the same clueless response I did! It was always some other group's problem when I called. My account number started with a J and whoever was on the phone didn't handle J accounts. I must have talked to 10 different finance groups between MCI and C&W before I found somebody who knew what was going on...
The business weasels in charge probably wouldn't have heard about the outage if the stock hadn't tanked that day.
MCI sucks, pass it on...
Situtation is being handled by blameing everybody.
Yes indeedy.. the quickest way out of a situation is to make sure you don't get the flak for it..
Of course, that means that the "smoking gun" left the gunpowder traces on your hand(s).. so, in the end [if not before] we all know you did it anyways.. so why bother hiding it?
Insert mind here.
AT&T's way is defintely the way a company should handle problems and you know that next time AT&T will be able to solve the problem quickly. No one likes to be blamed for a problem and blaming someone that you are in buisness with will only alienate them from. Unfortunately I have encounted too many companies with this attitude here in the valley and I was very suprised. After a while it became clear to me that in most companies blame needs to go to someone and woe to the person who gets up and says "sorry that was my fault I f%@ed up". You either get fired or get a bad performance review because no one around you admits to any wrongdoing then you must be only incompentant monkey working there.
just my 0.02$
Vidi, vici, veni. (I saw, I conquered, I came)
that the quote at the bottom of the page is:
"One reason why George Washington Is held in such veneration: He never blamed his problems On the former Administration." -- George O. Ludcke
Serendipitous or nice touch?
Note that this is the same company who still handles payroll via paper and Federal Express.
* As is generally the case, my opinions do not reflect those of my employer.
How the hell would a sales force successfully deliver info. to the customers by email if the whole problem was that their network connections were down?
One thing that this article points out, indirectly, is that honesty and responsibility pay. AT&T, by taking its share of the blame and being open about its problems, probably gained quite a few customers; MCI, by keeping quiet and shrugging off the blame, looks like it could lose some major clients.
We can draw a lesson from this. Most of the source of anger against Microsoft is the fact that they close the ranks and engage in finger-pointing when there's a problem, and attack the competition with FUD instead of concentrating on a better product. The Linux world, which is for the most part inherently honest (thanks to the Open Source paradigm), earns the reputation of a better product.
Another lesson to draw from this: it is better to be open about the strengths and weaknesses of Linux than to just plain Microsoft bash. But then again, this is already in the Linux Advocacy HOWTO.
Finding God in a Dog
I know that QWest provides LD service for a lot of schools and colleges here in the Northeast. I guess ECCI gets contracts with individual instituions and then bundles the stuff together to get good QWest rates. From what I hear, they're pretty reliable (at least better than CampusLink, who just does individual school stuff and where you have to just *hope* stuff works).
This sort of thing just makes me feel better about trusting the big bad former monopolies with their specialties :)
:)
Just goes to show one reason why AT&T and IBM have stuck around this long after having their monopolies broken up -- they realized that they could no longer compete with the same old "kill the competition" practices, and went ahead and started providing service. Of course, I also like IBM and (what used to be Bell Labs) Lucent for their significant outlay for R&D. That sort of thing can't be over-estimated in the value it gives a company that can afford it.
What is the meaning of this ramble? Not much, except to maybe provide some food for thought towards what Microsoft might be like it gets its ass kicked by the DoJ
--Matthew
They've been complaining about MCI for quite a while now, and viewing the situation from afar, their bitching is justified.
Redundancy and QoS? Cripes, the half-assed (or full assed) ISP I belong to is dual-homed, and probably has no problem keeping things running because of it. MCI seems to have no QoS policy.
I think the Chicago Board of Trade should send a wake-up call to MCI and change providers, possibly to AT&T. THAT would get their attention. If it doesn't, it should signal the other big subscribers to MCI to find other sources of bandwitdth.
1) It took them two months to get it setup and installed! All the while, we were chewing on Sprint, trying to get them to keep us on for a little while longer (we'd already dumped and reactivated the Ts twice).
2) When we finally did get them installed and working (which, BTW, we had to do ourselves. MCI/WC swore that the tech they sent would be able to configure a Definity G3 switch. he didn't. I learned a lot about phone switch administration that week), the two voice T1s would lose channels at complete random intervals, sometimes dropping to as few as 2 channels per line.
3) The third T1 had all 12 voice channels......and no data. It took over 3 months before we had an Internet connection on that line. Well, 2 months, 27 days, over 400' of ethernet wire (212' run installed and replaced), and one bridge later.
Fortunately for the ADSL users around my home, we have GTE running the lines (yes, I know, most of you hate GTE, but they've been wonderful for me) and only 2 providers subscribe to MCI's backbone. One (mine) colocated their Cisco with GTE at the frame relay network interconnection point, so my ADSL pipe is only 1 hop away from either Level3 or UUNet (I know UUNet is part of MCI, so we just used Level3 for those 10 days).