Slashdot Mirror
More on AT&T Wireless's Bungled System Upgrade
An anonymous reader writes "CIO.com has posted a very in-depth article on the recent failings of AT&T Wireless that resulted in the state of the company today. What's fascinating about this article is the sheer amount of accurate information gleaned from former and current employees on the company's bungled attempts to follow FCC mandates on local number portability last November, the inside story on outsourcing efforts, and terrible executive management decisions that ultimately led to its demise. Ironically, the scathing and sometimes highly sarcastic commentary at the end of the article from former employees makes this read even better."
Yes, AT&T Wireless (now part of Cingular) had its share of problems. There's no doubt about that. But at they made the right decision to choose GSM. Verizon and Sprint PCS chose wrongly, and so they are destined to fail.
There is only room for one mobile phone technology in this world, and it's not CDMA. I know the US government is behind it, but they cannot force us all to use it.
Siebel is horrible and this isnt the only company that this has happened too.
Take a look at Telus Communications.. When they implemented a Siebel based system customer complains skyrocketed.. The system was unstable and basically useless.. You couldent get any information to people on what was happening..
Papers had a field day on how much customer service sucked.
God have mercy on anyone who has to implement Siebel 7 in a large enterprise enviroment.
As the go-live date neared, former employees say that Deloitte and Touche project managers relaxed testing requirements for various pieces of the system.
The Big 5 (or however many there are now - I mean Arthur Andersen, Ernst & Young, Price Waterhouse, Deloitte & Touche, etc) charge hundreds of dollars an hour for "experts" that aren't experts at all. They're usually just one page ahead of the client. They even charge over $100/hr for wet-behind-the-ears college grads.
We've all dealt with them before, they are usually intelligent people but have no expertise or experience in the task they are being paid to complete.
Yet again and again, despite all their failings, they are being hired by big corporations for major projects.
I'd like to know why.
I wonder how many man-years of outsourcing it will take to make back that 100 mil AT&T lost?
Haven't RTFA, but last week my bro mentioned that when he moved to a Virginia town an hour outside of Washington, D.C., ATT wouldn't even offer him cell service, finally saying when pressed that they had a computer meltdown that resulted in an at least month-long, nationwide freeze on signing up new cellular customers. Ouch, says the bottom line. This was in the September-November timeframe. At the time I wondered if their selling themselves to the highester bidder a few months later was related.
Lots of people here in the SF Bay Area got so fed up with AT&T that they all changed over to Verizon. Unfortunately now Verizon's network is overloaded with all the new customers and it's almost impossible to get a call through around 9pm when the switch to off-peak hours occurs. Of course I didn't find this out until I already told AT&T to get lost and changed to Verizon. However, I'm gonna stick with Verizon anyway - at least they seem to be a bit more competent than AT&T so they will get these problems fixed sometime soon. The grass isn't quite as green as it looked from the other side.
personal development. it takes training to become good in your field. they have in-the-house training.
Spoken like someone who has either (1) never hired or dealt with a Big 5 consulting firm, or (2) someone who works for one.
The partners in these firms will sell anything. They will claim expertise in any thing they must in order to start billing. They send out green college recruits and charge over $100/hr for them.
As for "in-the-house training", the only experience most of them have is a series of failed PeopleSoft deployments (or Ariba, insert your own "enterprise" software here). Many of them have degrees in something completely unrelated to the project. Yes, the English major from Duke is very intelligent, but she doesn't know anything about the project at hand.
-they pay hard on their mistakes. Cash. When you buy outside your organization your excpectations are higher
Well, they may pay for their mistakes, but they certainly don't pay cash. These guys will absolutely bleed a company through continued billing regardless of how successful the project is.
I call bullshit. Working at Customer Care on the front lines, we knew a long time ago about LNP. AWE drug its feet trying to get LNP extended or demolished, and when it was apparent that it wasn't going to happen, they outsourced to whoever does their vendor calls centers, I think, instead of using what EVERYONE else was using, it was almost like a deliberate thing. Then when Siebel went down they drug their feet with that too. While I am by no means a sys admin or have dealt with the computer users that AWE does for their customer care and sales, but how hard would it have been to ROLL back from Siebel 7 so it didn't piss off every person who wanted to use technology that customer care was enforced to shove down peoples throats? Feh, since my karma is teh sux now I won't be able to reply to hardly anyone who posts back to this, but this is the grave they dug to make themselves more attractive to Cingular.
"Do you suppose that's why God lives in the Heavens? Because he lives in fear of His creations?" - Steve Buscemi
(Rant Mode On)
After going through 5 SonyEricsson T68i's in 3.5 mo, only to have each successive one start refusing to make outgoing calls, and juggling many hours and dropped calls with AT&T customer service only willing to send me another T68i, I just decided to ditch AT&T and go to T-Mobile. I found a Nokia 3650 on Amazon for $250 w/ 2-$150 rebates - one from T-Mobile and the other from Amazon.
The new phone, and T-Mobile service have been perfect here in Pittsburgh, and I'm enjoying the nationwide unlimited GPRS with bluetooth from my iBook, but AT&T screwups still continue. Three weeks after I switched I receive a spankin' new T226 in the mail. Phone works much better than the T68i, but no bluetooth, and a crappy screen. But hey, free phone I'll never use.
Unfortunately I still had 5 mo on my contract. Canceling stiffs me with a $180 charge, so I called AT&T 6 weeks ago to switch to the $20 plan. (a $100 loss, cheaper than $180). Well, yesterday I get my bill and find out that they never processed my request. Call customer service and finally get a rep who tries to be helpful, but can't figure out why the logged plan switch wasn't carried out. Supposedly I'll be reimbursed, but I'm not holding my breath. The rep didn't seem surprised when I explained I'd switched because of all the previous cockups.
I could go on and on with AT&T screwup stories, but you get the idea. I think the biggest pain is that those still in contracts have few or no options other than biting the bullet and switching.
(Rant Mode Off)
While this is a sad story--especially about the poor guys with Indian "consultants" following them around asking a zillion questions about how to do their jobs--it's worthwhile to remember where the article appears: CIO magazine. CIO is focused on the needs/wants/interests of the guys in ties in a corporate IT environment--and in general a lot of CIOs think that outsourcing/offshoring is a hell of a good idea. The general tone of this article is "look at how these yobbos bungled the implementation of Siebel CRM." What they didn't mention at all is, "look at how these geniuses totally misunderstood their business, and pissed away roughly $40 billion in stock capitalization in just three years. And therefore died the death that they so richly deserved."
It's the technology, stupid...
There are companies, even in the 21st century, that can ignore cutting-edge technology. You don't need to be e-commerce enabled to be a plumber. But if you're in the wireless telephony business, in the midst of a headlong rush into a blizzard of new technologies, the core focus of your business isn't marketing or sales or re-carpeting the executive suite. Your core focus MUST be on the technology--and as soon as you lose sight of that focus, your competitors will consume you.
And these geniuses decided to offshore 3,000 jobs. And were doubtless shocked--shocked, I tell you!--to hear that employee morale about the developers was down.
I'm no techno-protectionist
I remember discussing the inevitable introduction of competition from overseas back in the late 1980s, and debating the possibility endlessly while working in Japan in the mid-90s. There will be companies that decide that, in their businesses, in their business models, IT work is a cost, not an investment. They will decide that they want to minimize that cost. They will focus on maintaining existing systems (with marginal, incremental improvements) and eschew major new developments. They will find that that approach may make it feasible to hire developers in the Third World. But those businesses that do so are making a conscious, deliberate decision: we're not going to focus the company on technology. We're going to try to minimize the company's dependence on technology. IT is a cost--it does not contribute to revenue.
For a wireless telephone company to take this position is simply insane: they are in the technology business. They are smack in the middle of a global technology race--one of the few technology races with competitors from practically every part of the northern hemisphere. They need to be faster to market with new products; the new products must be faster, better, more efficient, and more effective; and they have to have a world-beating customer service experience. Instead of fleeing from technology, they should be driven by it. They should be absolutely focused on it. They should be actively recruiting talent to build their strengths....
Because that's what every other company that's focused on technology is doing. Subcontracting out your technology--in a technology business--is sort of like farming, but buying all your crops at the supermarket.
I am not a lawyer...
But I am an engineering team leader at a U.S. electronics company that leads the world in our industry: lighting controls. We export electrical and electronic equipment to countries around the world--including Japan, South Korea, China, Taiwan, Singapore, Australia, and every country in Europe--because we focus on five core principles. And Principle #4 is "Innovate with high-quality products." In other words, we're in the technology business, so we focus--relentlessly--on the technology.
Once upon a time, AT&T did too...
AT&T Wireless was spun off from AT&T--but the corporate heritage is obviously there. And AT&T, once upon a time, ruled the world--literally chan
To quote... "Don't know which world you are in when you say number portability was nothing new and they should have been able to get the job done. " And I don't know what world your in? Number portability is DEFINATELY nothing new. You mean to say that every time an AT&T customer decided they wanted a newer, better cell phone, they had to get a whole new number because they couldnt have the number switched over to the new phone? Never gotten a new phone and been asked if you want to check if the same 4 digits as your home phone are open? Gee, that's news to me. And its also pretty pathetic if you didnt know companies could set the number to anything the felt like in 5 minutes time and a couple phone calls. The only thing stopping the companies from doing this from the beginning is that it might mean they lose a few customers. All the hardware/software had to already be in place for keeping numbers working for their current customers.
~~ Please keep your arms, legs, and outright stupidity inside the ride at all times. Thank You ~~
I am the CTO for a large enterprise software company (>$1B).
I spend about 30% of my time in front of the IT departments of the largest companies in the world, all of which are household names. They almost all tell me two things about our software:
1. It is heavily modified (they all have source)
2. They wish it was not
The fact is that these large customization projects, particularly ones which involve the Big 5, are over budget and late by factors that would boggle the minds of most mortals. It is not uncommon for these companies to spend >$100M for a software upgrade ON A SINGLE SITE. These companies have hundreds of sites.
As a contrast, another $9B electronics company I met with a few weeks ago can install a complete factory, including financials, manufacturing, logistics, scheduling, human resources, and reporting, all in less than 6 weeks. They have done it over 100 times. How do they do it? They have the entire cookie-cutter system burned on a DVD. Literally no customization is allowed at the plant level.
The only way to be successful at these kind of projects is to use an axe, not a scalpel. AT&T Wireless tried to use a scalpel. They should have thrown out all that junk and started over.
I would also point out that if you read the CIO's biography, he is an advisor to HP. Notice that they also chose HP as their outsourcing partner!
Can you say "conflict of interest"?
About six years ago, I was a contractor at ATTWS in Paramus, NJ, working on the deployment of their point-of-sale cell phone activation app.
Without a doubt, it was the most dysfunctional office of a fortune 500 company I had ever seen, and I've seen a few. There were about six absolutely brilliant people there, who I would be glad to work with anywhere else, and a few hundred that I wouldn't trust to flip the proverbial burgers without putting someone's eye out.
Just one example: there was a pointy-haired middle manager there who liked to gather about $2k/hour worth of consultants into a conference room twice a week and just expound at length upon his management philosophy.
I was required to attend meetings on "planning my career path at ATTWS". I was a *contractor*, and I had work to do that didn't include making busy work for HR drones who didn't grok that ATT was a CUSTOMER, not a CAREER for me.
I even went to a meeting (again, mandatory), to hear some blithering bureaucrat tell us about ATTWS's process for developing processes. (I swear, I'm not making this up.)
Thank Judge Bell for opening those clowns up to competition. Somebody had to eat their lunch, and I wish every other cell phone company the best of luck.
-jcr
The only title of honor that a tyrant can grant is "Enemy of the State."
Are ther any good Siebel stories? I'm curious!
Ditto. Although in my case, I wasn't trying to get away from any other company, I was consolidating and getting national coverage. I'm just sorry I wasn't aware of this in January. I got some song-and-dance when I said I wanted to switch my phone number. I decided, to hell with it, and just dropped the old number.
In any case, the quality of signal is abominable. I think that's because it's GSM. I can use the phone OK in Ohio, or Wisconsin, or Georgia, but I get unbelievably bad reception in MY OWN HOUSE here in Florida, and it doesn't work at all in my mother-in-law's house in Mississippi (oh, wait. Is that a bad thing?).
I think enough time has passed for me to drop this junk for Verizon (or even Sprint) - I _like_ CDMA.
I have to totally agree. I'm in the northeast and signed up for Suncom about 6 months before they got bought by AT&T wireless.
The service with Suncom was quite good. There was good coverage over the entire area and the voicemail and missed call alerts worked perfectly with almost no lag between the call and the notification.
AT&T wireless took over and service deteriorated rapidly. Coverage became spotty in the area and varied widely from day to day and the voicemail and call notification system became abysmal not noting calls that came in or voicemails for days after the original call was made.
6 months later the plan was up and so was my patience. Now on Verizon wireless for the last year or two and they've been quite good overall.
-------- This space intentionally left blank --------
namely, denial. the only piece that works is the old long lines department, now ATT business data. everything else with the "death star" logo is useless. outsourcing the people who are supposed to save you is the latest ATT lunacy, capping a string of them all the way back to divestiture.
thank god the baby bells got freed from that mess. all the folks vying to lead ATT in the 90s -- joe nacchio, mike annunziata, leo hindery, c. michael armstrong -- turned out to be a shitspread at their respective next stop in employment at the top of the tower. "little mikey" in particular broke up and sold his company down the river in several stages, then left it to hide out at comcast and count his money. "joey nachos" almost killed qwest, a fiber startup, and USWest together after he merged them to bleed the treasury at USWest. annunziata and hindery rode Global Crossing into the toilet, and hindery got into another telco startup and crashed with it.
moral: if you want to invest, check for former ATT execs on the board of a company. if you find any, flee in terror.
if this is supposed to be a new economy, how come they still want my old fashioned money?
I agree. I was at AT&T in 1984, just after divestiture, and most of the company was wandering around in a daze trying to figure out how to operate as anything but a regulated monopoly. It took many years for the old "phone company" monoculture to fade, but there was never anything dynamic to take its place. Every time they'd get into some new business, it was always to grab a piece of an existing pie, never to actually create something new that peple would pay money for.
Bellcore-later-Telcordia was even worse. Because of legacy contracts with the Baby Bells, they were able to hold on to their biggest cash cows longer than AT&T was. Think about it: Which came first, intense competition in the long-distance market, or intense competition in local service billing software? They were as dead as AT&T, but it took them several more years to fall over. And I won't even get into the cultural train wreck that occurred when SAIC bought Telcordia.
And people wonder why I took a job with a Big Evil Defense Contractor.
It's funny, AT&TW was so inept and screwed-up, yet I never had ANY problems with them when I had the service. I had them for about 4 years until last November when I moved to Verizon. Their coverage was great, reception was clean, customer service was good, I was pretty happy with them. Then all this stuff happened and all I hear is horror stories of mistreated customers and network problems. It's strange. I live in Texas, so maybe they were just more efficient down here for some reason.
As for Verizon, I am VERY happy with them. They have a nice selection of phones, their Texas coverage is excellent, and their customer service reps are incredibly friendly and helpful. I once was very late on a payment due to financial problems, and after I talked to the rep about it, she removed the late fees without resistance. So, if you're on AT&TW and trying to escape, check them out.
AT&T has had really bad executives and really bad leadership. When Ma Bell was broken up the current CEOs showed great ineptitude by capitulating to MCI and Bill McGowan. Whereas IBM and Microsoft have shown that you can fight Antitrust cases and settle them on your own terms Ma Bell capitulated and decided to split up on bad terms. This was the first example of bad leadership.
AT&T bought a stake in Sun Microsystems. After which Sun mysteriously changed religions from BSD to System or SunOS to beastly Solaris.
Robert Allen is bad executive number two. In 1991 AT&T under Robert Allen decided to buy NCR for $7.5 billion NCR and renamed it as AT&T GIS. AT&T also sold of Unix System Labs to Novell.
NCR was later spun off at a loss of billions due to the inability to manage it. It came down to management.
Michael Armstrong is another bad CEO. Under Armstrong TCI was purchased and so was Media one.
Armstrong spent 100 billion buying different companies. One mistake was getting into a bidding war over Media One. Another gaffe was Excit@Home whereas Timewarner had roadrunner Excite@Home was allowed to fail under AT&Ts watch.
After merging the company decided to demerge and spin off AT&T wireless and AT&T broadband. A stake was sold to NTT Docomo in AT&T wireless. The paradox is why would you spend billions of dollars to buy companies and then decide your corporate strategy is wrong and spin off the companies?
AT&T wireless was a big mess. NTT Docomo purchased a stake in AT&T with strings attached. AT&T Wireless was pushed to used Docomos version of GSM e-mode. They also erred in switching wireless protocols from TDMA to GSM instead of CDMA. CDMA is better and more established in the US. Sprint activelly promotes their version of CDMA as better than AT&Ts GSM. The migration to GSM was very complex and had lots of glitches. However, when they brought in NTT Docomo as a stockholder it went with GSM. It isn't just GSM it's a variant of NTT's. NTT Docomo wouldn't even bid for AT&T wireless afterward.
AT&T also erred in spinning off AT&T wireless as a tracking stock (A mistake Sprint has soon learned) and hiring Michael Armstrong as CEO. The company basically gave away 40 billion dollars in value including the Cable and Wireless divisions (speaking of which is another misnomer for a badly managed company). Jack Welch the CEO of GE felt that spinoffs were bad and spinoffs just spinoff cash and assets to shareholders with vary little return for the parent. It proves the old adage that your only as good as the leader. A very bad leader can destroy a company as can be seen at MCI err MCI-Worldcom.
I work for a partner of AT&T, and I can say that they are so disorganized the left foot doesn't know wht the right foot is doing. There are good people there, but more often than not, those are the people that get screwed by upper management, which ends up screwing the service they are offering.
A slip of the foot you may soon recover, but a slip of the tongue you may never get over. -Benjamin Franklin
Say all you want, that this was mismanaged, bureaucratic, a case of piss-poor consulting decisions, etc... but in my opinion, this is a textbook case of simply not following any sort of software development lifecycle method.... and yeah, I suppose management is part of that, but I simply don't think looking at this as merely a "poorly managed project" gets at the heart of it.
Any technology company needs to adopt and follow some sort of SDLC, and this is an obvious case where this has never been done. Criticising them for bungling this is all well and good, but I feel this article would've been better off talking about the real meat of "why" it failed.
...would seem to be a major survival skill at many corporate offices.
There is a reason that the 456 was called the exchange, the older term for a phone switch. 456 referred to a particular set of hardware. Routing was sequential. You could route the connection as the number was dialed. The number sequence was mapped to routing sequence.
With the new LNP, you can't do that anymore.
Like the Y2K effort, it was a lot work to find those 'you can't do this anymore' bits of code.
Or, at least that's my understanding, not being a 5E coder.
> -they pay hard on their mistakes. Cash. When you buy outside your organization your excpectations are higher
:) It's rather depressing really, but when you have a lot of code put together by a lot of people under a deadline, you don't necessarily know WHO wrote something. And identifiying an issue is a much different political exercise than extracting cash from a vendor for said piece of code with a dubious audit trail (the "Who really wrote this?" question would keep you in court for years).
:)
Ummmm.... I'm going to have to disagree with these statements when talking about large projects. The first step in "pay hard on (sic) their mistakes" is labelling a "mistake". You don't usually find out about a botched piece of code until after the dust has settled and something happens.
As an example, a large multi-year project that had the vendor's professional services firm involved (ie "the experts") wound up more or less six months ago. We recently found out that some of the code that creates output files (ie billing information) DOES NOT HAVE ERROR CHECKING!!!! Yep, if for whatever reason the files can't be created, you cannot tell from the output of the functions that an error occurred. Sweet, huh?
Did I mention that we found this out during a period where we had some filesystem issues?
Keep in mind that the politics are MUCH more important than what you can prove, or what you think happened. There might be absolutely no doubt in your mind about to whom responsibility for an issue belongs, but getting the other party to accept that responsibility can turn into a greek tragedy. "Truth" is important, but sometimes irrelevant in contracts.
That beats the heck out of the all-hands meeting I attended at my former employer, HP.
The head-honcho-manager-person at my location told everyone that they were going to be doing some layoffs in the next couple of months. She then proceeded to talk down to everyone and tried to make us all feel like it was a GOOD thing!?!?
This all came ONE WEEK (I kid you not) after we were all forced to attend a mandatory meeting with an efficiency consultant, whose job was basically to make everyone believe that customer service was top priority. We were not to adhere to the contract, but rather to do what the customer requested, no matter how ridiculous it was. This included us doing things that would directly harm the customer (if they asked us to, of course).
The layoff meeting was preceeded by another meeting, a day before, telling everyone to basically not do "everything the customer says" and to stick to the contract as it's written.
I'm glad I'm doing honest tech work again.
This is in response to the OBSCENE wait times for customer service. Yesterday, I had a 2:11:03 call in to customer service (my 611 records will prove the call times), and the issue is still not resolved.
AT&T is a voluntary signatory on September 9, 2003 to the Cellular Telecommunications & Internet Association's Consumer Code for Wireless Service.
Section 8 of the Consumer Code for Wireless Services states that the signatories will: "Ready Access" to solve account issues is not being provided by AT&T customer service.
Though AT&T is clearly in breach of at least one section of the Consumer Code for Wireless Service, they are still using the seal to advertise their adherence to the contract. I believe falsely advertising adherence to the Consumer Code warrants service agreements signed under this false pretense null and void.
-extremely dissatisfied customer
As a former member of the Odyssey II Environment Support team I had to live through this hell. As the article mentioned each "environment" consisted of somewhere between 12-16 individual systems. The team consisted of a dozen employees/contractors that had to install and maintain each system. Each employee was assigned two or three systems which they had to become the "experts." Because of the complicity and vast number of changes to each system it was difficult to become proficient in more then two or three. This approach sounds okay until you factor in all the development and testing environments required and the long hours of testing.
At the time I there were 18 different environments that were up and running during the day. By day, I mean expected to be up and running between the hours of 7:00am until 10:00pm. The major enviorments (Siebel Dev, System Test, Integration, etc...) ran until midnight. Any changes to the environment had to take place after hours. With the average "kit" install taking between four to six hours it meant the we were running a 24-hour shop.
They tried to split the 12 members of the team into three 8-hour shifts. With each member only trained three systems that meant we could only cover 12 of the 16 systems with four employees. Multiply that by the 18 environments and you can see where the troubles begun. Those 8-hour days turn to 10 and then 12. None of the environment were stable and consistently were down during testing due to bad code. Emergency kits were commonplace and since installation were so long (due to Siebel's shitty product) testing was always behind requiring weekends as well. All this added up to 70+ hours of work, 0 sleep (had a newborn at the time), and one VERY pissed off wife.
I was lucky enough to have left to get another job just before the system went live. It was obvious that it was going to fail and I had a huge shit-eating grin on my face when I heard of all their troubles.