Slashdot Mirror
AT&T vs MCI on Network Outages
James Ensor writes "
Cnn.com is running an interesting article comparing MCI Worldlnet's recent frame relay outage to an outage that AT&T had last year, comparing their method's of handling the situation. Good Reading. " It's fun to watch major corporations blunder about. Unless it's your connection that's down.
What's PPL, another monopolist telco?
It finally happened. I knew one day I'd browse on over to Slashdot and find a naughty story about the company I work for. Well, I can't say they weren't asking for it.
Yeah, the point about all the companies within MCI Worldcom rings true. Like the MCI local network, MCI long distance network, Worldcom long distnace network, and the Worldcom local network (which is actually the MFS local network and the Brooks fiber local network). Then you've got a layer on top of that, like UUNet. Not all that straightforward. They are working on these issues, though.
Responsibility issues? They are there. You should see the hoops I have to go through just to get someone in the same building to investigate a problem within our own LAN. MCI highly emphasises "the procedure".
But the story that Slashdot should be covering isn't the frame network outage. It is MCI's agreement to sell off their IT department to EDS. Most everyone assumed it has gone through. Guess what? They made the announcement 6+ months ago, and the deal (along with IT) is hanging in limbo.
Will we work for EDS? MCI? Do we get our current stock options? Benefits? Policies? Management is saying absolutely nothing. Rumors are flying that the negotiations are going bad and neither side wants to walk away because of a hefty $$ penalty. Even an arbitrator hasn't helped. (That's right, an arbitrator just to reach an agreement. Heaven help both sides over the next five years.)
The entire IT department of a major corporation that can't even say what conditions they will be working under next month. How smart is that?
The main problem is that EDS is a very bad fit for a company like MCI Worldcom. EDS is great for an organization that is low-growth, low-innovation, and reliability situations aren't critical. Say, a local government. But a phone company? Oh boy.
I can't say I know an employee that is enthusiastic about the merger. The only management that claims to love the idea is from the VP level and up.
it is short for people. u know just leave some letters out. learned that from the jews.
We switched our ISP's from MCI to Telwest, MCI left our T1 on for 18 months before disconnecting it. (Then tried to bill us!)
Then do tons of documentation so it never happens again!
I work daily with the technical support team. Their job is to solve problems, not pass the buck. And if they cant fix it, it goes up the chain of support, And yes all the way to Frank Ianna.
Our goal is no more than 5 minutes of unplanned outage a year. Thats our motto at the office. (Damn good one also)
Maybe just the ones you work at, moron.
This is not an OS issue, it's a networking issue. At a shop the size of CBT it would belong to the networking group. Linux,NT, Solaris, HP-UX , or any other OS would be downstream of the problem and not relevnt to the solution.
Saw a couple of informative posts about MCI now and rating is as bad as ever. Or they haven't been rated at all. There goes Slashdot quality... yuck
i.c.wiener
You have to figure that all of this cost-cutting has to reach down and have some effect on the people that actually do the work. When benefits get trimmed and the company won't pay competitive salaries to new hires, your top talent gets cheesed off and heads for greener pastures. That leaves your less-than-top talent minding the store when things like this outage happens. ...Not to mention that they're having to compete aginst what used to be the company's best and brightest, many of whom have probably landed new jobs with your competition.
-- Phil McRevis
AT&T. We were using MCI for our T1 services and it seemed like everyweek they had some router in Atlanta that was on fire, had grown legs and walked off or had been somehow run over by a bus. Honest to god. They said a bus hit their relay station. We have been using AT&T now for almost 5 months and haven't had a hiccup, they aren't the fastest to get out here and install the stuff but they definately have the better of the two services.
--"Karma is justice without the satisfaction"
If the CBT can't plan ahead and shell out a little extra cash so they can be multihomed, they deserve whatever crap they get. I wonder what happens to them when NT crashes. Does the whole place go belly up or do they actually have some kind of HA setup.
-matt
i too have seen this at several companies and it is very sad to see adults act like 5 year olds around a broken cookie jar.
bottom line: lazy people that can't prevent disasters can only find scape goats. stay away from them or they'll pick YOU!
get your dirty sig off me, you filthy APE!
Okay i don't use MCI but I wish i would've had some of their stock! After the outages were fixed it gained 18 points in 1 day! O wait this is a linux board i'm not supposed to care about money So long live free information or whatever
I agree MCI Worldcom should take the blame. They should have focused more on the customer's needs and backed out the software to get the network back up. It was clearly the fault of upper management for listening to the vendor and his "assumptions" and "recommendations". The customer should be faulted for blind faith, not the vendor.
If the sales reps had information and decided not to contact their customers, I'm not sure if that's a management problem or a weasely sales sales person problem. Either way, there wasn't very much information available because not very much was known or understood about the problem. For the longest time (the first week or so)Lucent even claimed it wasn't a result of the software upgrade!
Anyone remember the Remote Explorer Virus/Worm? I do
Remember how MCI downplayed the situation and blamed Network Associates for releasing privilged information? I do
The situation was MUCH more serious than MCI admitted to. Network Associates didn't leak the news, MCI employees did!
Same here. I work(ed) (still part time when they need me) for a small ISP (no T3's there anytime soon) that used Sprintlink for backbone and Sprint for local service. First, their local guys kick ass. After all the local telco jokes, these guys were a pleasant surprise. They knew how to do it and could deliver a channelized T1 or PRI a week within ordering, 2 weeks in the worse case. Another ISP I have had contact with in GTE land has to order 3-4 months ahead and the usual reaction is T-what?
Sprintlink has been very reliable. We had a little trouble with not enough bandwidth leaving the local POP we connect to, but they got it figured out eventually. In fact, we were going to replace them with someone cheaper, signed the contracts, got the new feed in, and after testing it, are trying to get out of the contracts. They sucked. Sprintlink is well worth the extra money.
Of course, I also like IBM and (what used to be Bell Labs) Lucent for their significant outlay for R&D.
Yeah, "significant outlay for R&D", that would explain the LT Winmodem.
Just like the company I used to work for. Customer has a problem, x'fer them to someone else. Even though our department most likely could finger the solution out faster than the "correct person". Kind of like "The buck stops over there, besides it's not my responsibiltiy."
It seems like in the race to save a dollar companies have forgotten that most folks will pay a little extra for good service. Perhaps they should teach that concept in business school. Wait, I graduated from one and they didn't teach that. To damn busy doing SWOT analysis, Gannet charts and critical path analysis to worry about User Experience. IBM and AT&T haven't been around forever and a day letting their customers go away mad.
...is planning to fail.
-Einstein
oops forgot. It takes a genious to understand that and they do not have _any_ there.
My man, you are a much braver person than I could ever be.
I thought working for a large companies would insure stability in your career but with the woes of MCI (does our frame relay and one T-1 from UUNet) and SGI (go back to Silicon Graphics) who needs that type of instability?
On the other hand, my current small company is trying to write up policies and procedures so I guess I will be looking for another job soon.
ChozSun [e-mail]
ChozSun
ChozSun.com
Where does Sprint fit in to all of this? We've been buying several T1s and ISDN lines from them for a couple of years and have had almost no downtime. Even when we do, a quick call to their TAC either gets it fixed or gets me an explanation (talking to the actual techs who work on things .. not some brain dead support rep). In fact, I have a full DS3 feed on order from them right now. I don't know everyone else's experience, but we've been very happy with their service.
"FORMAT C:" - Kills bugs dead!
I have a lot of friends that are AT&T employees in R&D. On day 8 of the MCI outage, I mentioned the MCI outage to them. They hadn't heard about it and were damned pissed about it also. After April 1998, when they saw AT&T outage on the front page, etc. and all the heads rolling around them, they couldn't believe that MCI wasn't being eaten alive. 8 days! Then MCI goes and blames it on Lucent. That really got these guys going since Lucent all but shares buildings with them (actually they did up until 3 months ago).
Thank god one of the journalist crowd has a clue about this and published. Very impressive.
I was reading an article about the outage on news.com on day 9 - when I got one of those "Hi, are you happy with your current long distance service?" calls. It was from Qwest, a company that I haven't heard much dirt about yet. Yet.
My old housemate had moved to Europe and they offered decent international rates, so for once I said 'yes' to one of those telemarketers that I usually hate being bothered by.
I did tell them that the only real reason that I was switching was because MCI really messed up. Of course, I was talking to a 3rd party telemarketer who probably has 1000 resumes our trying to get a real job and really doesn't care about telecommunications in the least bit.
But it made me feel better.
.
And besides, there's even a beta of a driver for the thing, if you happen to be running OS/2
--Matthew
Chris
Chris
So Buddha walks into a pizza parlor and says: "Hey, make me one with everything."
AT&T Worldnet is AT&T's Internet Service (One of the top ranked national ISPs, too). Perhaps there's where the misinformation is coming from.
,'s in e-mail address with .'s.
j-a-w-a-d------------------------------
replace
i dont display scores, and my threshhold is -1. post accordingly.
Discuss
Lighting rods, anyone?
After a company grows beyond a certain size, it gets increasingly tough to just "wing it"...
Maybe you mean something more sinister by "policies and procedures"?
Due to the ancient mainframe software that was geared toward provisioning phone lines, it was quite easy to lose a customer's IP address and two or three times a week we'd get a call because some customer's network had stopped talking to the internet. In the majority of those cases, the reason the network had lost connectivity was because we'd provisioned another customer with the IP addresses and that other customer had already assigned his entire network and gone on-line, leaving us in the awkward position of having to tell one of the two that they'd have to change. And lets not even get started on the lost circuits themselves (Which was completely outside the domain of the software I had to deal with.)
Also frequently we'd lose customer orders in our system. That doesn't sound too bad until you consider that on a regular basis we'd freeze all orders for either T1's or fractional circuits because our routers were all at capacity and then some. So if a customer managed to squeak in under the wire and then his order was lost, he was SOL, sometimes for months. Combine that with some lame-ass company like US West which could take upward of a YEAR to run a local loop and it was all just a recipe for disaster. Another engineer I used to know termed his job "Going from one fire to the next, trying to get some work done on the hottest one."
Frankly I'm surprised no one ever sued them while I was there. I bet you could float a malpractise suit against them, as much as they suck.
Anyone notice, that while talking about frame relay, they have a Centronics (SCSI/Printer) cable
picture on the article?
Just because you disagree doesn't make it offtopic or flamebait.
Chicago Board of Trade is a Windows NT shop, and doesn't have the expertise to properly manage a network. Heck, their CIO is in all the Microsoft ads. Thinks NT is great. Has all the right buzzwords. Barf.
I blame the government I blame society I blame their parents
howdy.
as an employee of Qwest, i can tell you that we've ALL had our share of minor outages, but most were handled quite gratiously from my vantage point.
EVERYONE will have these problems some time. in a very non-ideal world, software and hardware is simply too complicated to truly test each permutation to prove your solutions are perfect. (unless, maybe, you have some really really good test engineers (remember, they're grossly expensive), a great testing process (either steal it, or "acquire" it at great cost), and superb ss7 & other test equipment (also not cheap by any stretch). even then, i believe it's mathematically impossible to test all the permutations faster than many years. thus, you consciously choose to test the most probable permutations.
HOWEVER, NOT everyone will handle this the way MCI did. i can guarantee that were it me, i'm a shoot-from-the-hip kind of guy and would have been as up-front about it as possible about it. there's nothing more damning than poor communication in such trying times, and finger-pointing (read, lying) is for those that prefer an early stress-induced death. (one's conscience will EVENTUALLY catch up with ones self, probably in one's child-rearing years)
now, i'm quite positive that our upper management would never agree to being that honest. image (err. i mean the *** short term *** value of their stock options) is far too important. (m... ust.... cut.... off.... nose......)
you obviously see the phrase most prominently highlighted, and with GOOD reason.
i SINCERELY believe that the one-quarter-out mentality has greatly exacerbated, if not produced all of these problems.
;)
Did AT&T have a monopoly at one time and learned from it (the hard way)? PPL and Companies should learn again that "The Customer is the King", as we Germans sometimes say.
no offense intended, my friend.
simply this:
policies and procedures are NECESSARY for existence. that's just something we all need to get used to. i defintely don't LIKE them, but i get used to them, for they tend to "serve the greater good" when executed correctly.
NOTHING truly operates a company better than policies and procedures enforced by a body filled with common-sense that can articulate their arguments well, and admit their errors.
a little common sense and decency goes a LONG way in these things...
process for process' sake is simply folly.
(what i'm saying is, GET INVOLVED. don't let "them" rule the world)
;)
MCI is such a horrible company
JediLuke
JediLuke
-Do or Do Not, There is no Try
This rang somewhat true, and in the years since, I watched AT&T live up to this accusation. Until now...
I'm wondering if I was a bit too eager to write them off. Maybe AT&T actually does have a future...
I can see the fnords!
...this sort of thing is likely to continue in this day and age of mega-mergers. AT&T did the Right Thing(tm) and I'm pleasantly impressed. I guess it is possible for a 600 pound gorilla to handle a situation intelligently...
censorship is a form of noise, which actively seeks to drown out content with silence - Crash Culligan
They have a "just enough" attitude to their services. Just give your customers as little as possible and continue to overcharge them. Outages are one thing, how come they refuse to upgrade overloaded networks?
Putting blame on others is also quite common.
I get that impression from many American companies. An embedded company called TERN gave me similar BS at a previous job.
Their development tools were antiquated, the board was defective and they refused to replace it. Transferring new code kept giving CRC errors, and sometimes it would get through, but with errors. They gave me a similar runaround, claiming that all of our computers, including my Dell Dimension XPS 200, had "nonstandard" serial ports that couldn't keep up with 57.6k. Yeah right.
I finally got to the "CEO" (small company) who told me that GM had the very same problem with 100 different new computers, and that all of those computers were bad.
Yeah, right.
We hauled the boards out of both of the projects we were working on and replaced them with cool, reliable ZWorld boards.
Development time shrunk immensely, and reliability was great. Their tech support was also extremely good, they would email me code snippets and everything. They also encouraged messing around with their libraries, I thought that was excellent and found them well written and easy to hack.
Hmmm... Parallels the referenced article quite well. Goes to show, if you're not satisfied, walk. It makes all the difference.
They're both useless companies.
-Dyl6
Myself and a few thousand other people were completely left in the lurch when MCI sold its dial up service to Cable and Wireless...Neither company had a full accounting of their subscribers, but that didn't keep both sides from debiting money from my checking account for services never rendered! I had to close my bank account to stop it!
What's really sad is the freakin' Chicago Board of Trade got the same clueless response I did! It was always some other group's problem when I called. My account number started with a J and whoever was on the phone didn't handle J accounts. I must have talked to 10 different finance groups between MCI and C&W before I found somebody who knew what was going on...
The business weasels in charge probably wouldn't have heard about the outage if the stock hadn't tanked that day.
MCI sucks, pass it on...
Situtation is being handled by blameing everybody.
and there are peoples who trying to get 1st post.
Situtation is being handled by blameing everybody.
Yes indeedy.. the quickest way out of a situation is to make sure you don't get the flak for it..
Of course, that means that the "smoking gun" left the gunpowder traces on your hand(s).. so, in the end [if not before] we all know you did it anyways.. so why bother hiding it?
Insert mind here.
AT&T's way is defintely the way a company should handle problems and you know that next time AT&T will be able to solve the problem quickly. No one likes to be blamed for a problem and blaming someone that you are in buisness with will only alienate them from. Unfortunately I have encounted too many companies with this attitude here in the valley and I was very suprised. After a while it became clear to me that in most companies blame needs to go to someone and woe to the person who gets up and says "sorry that was my fault I f%@ed up". You either get fired or get a bad performance review because no one around you admits to any wrongdoing then you must be only incompentant monkey working there.
just my 0.02$
Vidi, vici, veni. (I saw, I conquered, I came)
that the quote at the bottom of the page is:
"One reason why George Washington Is held in such veneration: He never blamed his problems On the former Administration." -- George O. Ludcke
Serendipitous or nice touch?
Jeez gimme a break MCI. If our network with a couple thousand users would be down for more than say six hours in a row, I'd have to keep my phone offhook and a dozen people would come into the office every minute asking what's wrong.
Sounds like a story worth posting on comp.risks:
- many hands will eventually break anything
- outsourcing may cut costs, but also cuts brainpower (which is bad)
- MCI ain't got no stinking QoS
- their network will be the first to fail when some drunk idiot drives his car into Mae-East
(remember where those routers REALLY are?)
i.c.wiener
Note that this is the same company who still handles payroll via paper and Federal Express.
* As is generally the case, my opinions do not reflect those of my employer.
How the hell would a sales force successfully deliver info. to the customers by email if the whole problem was that their network connections were down?
One thing that this article points out, indirectly, is that honesty and responsibility pay. AT&T, by taking its share of the blame and being open about its problems, probably gained quite a few customers; MCI, by keeping quiet and shrugging off the blame, looks like it could lose some major clients.
We can draw a lesson from this. Most of the source of anger against Microsoft is the fact that they close the ranks and engage in finger-pointing when there's a problem, and attack the competition with FUD instead of concentrating on a better product. The Linux world, which is for the most part inherently honest (thanks to the Open Source paradigm), earns the reputation of a better product.
Another lesson to draw from this: it is better to be open about the strengths and weaknesses of Linux than to just plain Microsoft bash. But then again, this is already in the Linux Advocacy HOWTO.
Finding God in a Dog
I know that QWest provides LD service for a lot of schools and colleges here in the Northeast. I guess ECCI gets contracts with individual instituions and then bundles the stuff together to get good QWest rates. From what I hear, they're pretty reliable (at least better than CampusLink, who just does individual school stuff and where you have to just *hope* stuff works).
This sort of thing just makes me feel better about trusting the big bad former monopolies with their specialties :)
:)
Just goes to show one reason why AT&T and IBM have stuck around this long after having their monopolies broken up -- they realized that they could no longer compete with the same old "kill the competition" practices, and went ahead and started providing service. Of course, I also like IBM and (what used to be Bell Labs) Lucent for their significant outlay for R&D. That sort of thing can't be over-estimated in the value it gives a company that can afford it.
What is the meaning of this ramble? Not much, except to maybe provide some food for thought towards what Microsoft might be like it gets its ass kicked by the DoJ
--Matthew
"Method's?" For the love of God!
They've been complaining about MCI for quite a while now, and viewing the situation from afar, their bitching is justified.
Redundancy and QoS? Cripes, the half-assed (or full assed) ISP I belong to is dual-homed, and probably has no problem keeping things running because of it. MCI seems to have no QoS policy.
I think the Chicago Board of Trade should send a wake-up call to MCI and change providers, possibly to AT&T. THAT would get their attention. If it doesn't, it should signal the other big subscribers to MCI to find other sources of bandwitdth.
All that matters is who has the lowest rates. The Chicago Board of Trade isn't going anywhere. They will use this to negotiate cheaper service and then shut up.
1) It took them two months to get it setup and installed! All the while, we were chewing on Sprint, trying to get them to keep us on for a little while longer (we'd already dumped and reactivated the Ts twice).
2) When we finally did get them installed and working (which, BTW, we had to do ourselves. MCI/WC swore that the tech they sent would be able to configure a Definity G3 switch. he didn't. I learned a lot about phone switch administration that week), the two voice T1s would lose channels at complete random intervals, sometimes dropping to as few as 2 channels per line.
3) The third T1 had all 12 voice channels......and no data. It took over 3 months before we had an Internet connection on that line. Well, 2 months, 27 days, over 400' of ethernet wire (212' run installed and replaced), and one bridge later.
Fortunately for the ADSL users around my home, we have GTE running the lines (yes, I know, most of you hate GTE, but they've been wonderful for me) and only 2 providers subscribe to MCI's backbone. One (mine) colocated their Cisco with GTE at the frame relay network interconnection point, so my ADSL pipe is only 1 hop away from either Level3 or UUNet (I know UUNet is part of MCI, so we just used Level3 for those 10 days).