Slashdot Mirror
More on AT&T Wireless's Bungled System Upgrade
An anonymous reader writes "CIO.com has posted a very in-depth article on the recent failings of AT&T Wireless that resulted in the state of the company today. What's fascinating about this article is the sheer amount of accurate information gleaned from former and current employees on the company's bungled attempts to follow FCC mandates on local number portability last November, the inside story on outsourcing efforts, and terrible executive management decisions that ultimately led to its demise. Ironically, the scathing and sometimes highly sarcastic commentary at the end of the article from former employees makes this read even better."
[huge grin] wonder how he got *that* nickname
Seriously, someone who has new/wonderful management process *can* help a company tread a dangerous path, but you *need* the domain expertise to be well-represented in a solution to a technical problem. Imho, at least. From the reports, it seems the process wasn't that wonderful, either
You can expect a certain level of bitterness in ex-employees, especially after a disaster, but there ain't no smoke without fire, and when there's lots of smoke, start looking for the towering inferno...
Simon.
Physicists get Hadrons!
Including wireless internet access to write this on Slashdot, and I've had no pro
For some reason I just cant feel sorry for a company not being able to rally its workers and threaten the workers of off shoring their work. I believe that AWE got exactly what it deserved - number portability was nothing new and they should have been able to get the job done. Yet AWE insisted on moving towards outsourcing instead of figuring out what needed to be done. I have seen similarly situations where no matter how much cheaper labor you look for, if you can't devise the plan, no one will be able to follow it. Good riddance to AWE and I wonder if Cingular is going forward with the outsourcing.
Yes, AT&T Wireless (now part of Cingular) had its share of problems. There's no doubt about that. But at they made the right decision to choose GSM. Verizon and Sprint PCS chose wrongly, and so they are destined to fail.
There is only room for one mobile phone technology in this world, and it's not CDMA. I know the US government is behind it, but they cannot force us all to use it.
- a young blogger
Siebel is horrible and this isnt the only company that this has happened too.
Take a look at Telus Communications.. When they implemented a Siebel based system customer complains skyrocketed.. The system was unstable and basically useless.. You couldent get any information to people on what was happening..
Papers had a field day on how much customer service sucked.
God have mercy on anyone who has to implement Siebel 7 in a large enterprise enviroment.
As the go-live date neared, former employees say that Deloitte and Touche project managers relaxed testing requirements for various pieces of the system.
The Big 5 (or however many there are now - I mean Arthur Andersen, Ernst & Young, Price Waterhouse, Deloitte & Touche, etc) charge hundreds of dollars an hour for "experts" that aren't experts at all. They're usually just one page ahead of the client. They even charge over $100/hr for wet-behind-the-ears college grads.
We've all dealt with them before, they are usually intelligent people but have no expertise or experience in the task they are being paid to complete.
Yet again and again, despite all their failings, they are being hired by big corporations for major projects.
I'd like to know why.
I don't give AT&T Wireless any credit whatsoever. For God's sake! They can't even figure out how to properl set the time on their towers.
In the Boston area, they reset the time for their towers by setting the clocks forward one hour at Daylight Savings Time (as opposed to properly setting the "Daylight Savings Time flag). Now whenever you use Cingular's network, you get the proper settings, but as soon as you go bact to AT&T, it puts you an hour ahead on wintertime hours.
I wonder how many man-years of outsourcing it will take to make back that 100 mil AT&T lost?
having worked for attws for a year now, and having been on the receving end of all the angry bitter customers when seibel 7.5 f-ed up, I can tell you that this was the very worst transition I have ever seen. 100 million dollars? between 50 and 200 thousand customers lost to curn? WTF? and it's still a joke on the inside.
Posted ac for my job...
I had AT&T wireless 5 years ago in college and they were no better then. Bad coverage areas, customer service people without a clue, and screwed up billing.
Say what you like about Verizon's CDMA technology. There is more to a phone company than the technology they choose. If the company can provide robust coverage, provide wireless broadband, and treat me right - they will succeed.
As for the CDMA detractors: Try getting 500k/sec. on ANY GSM system now....Verizon's testing it in DC and NY and will soon roll out nationwide.
AT&T wireless didn't fall apart because of their technology choices...they fell apart because they treated their customers badly.
-ted
Haven't RTFA, but last week my bro mentioned that when he moved to a Virginia town an hour outside of Washington, D.C., ATT wouldn't even offer him cell service, finally saying when pressed that they had a computer meltdown that resulted in an at least month-long, nationwide freeze on signing up new cellular customers. Ouch, says the bottom line. This was in the September-November timeframe. At the time I wondered if their selling themselves to the highester bidder a few months later was related.
Lots of people here in the SF Bay Area got so fed up with AT&T that they all changed over to Verizon. Unfortunately now Verizon's network is overloaded with all the new customers and it's almost impossible to get a call through around 9pm when the switch to off-peak hours occurs. Of course I didn't find this out until I already told AT&T to get lost and changed to Verizon. However, I'm gonna stick with Verizon anyway - at least they seem to be a bit more competent than AT&T so they will get these problems fixed sometime soon. The grass isn't quite as green as it looked from the other side.
I did infer the saboutaging that was probably going on from the article but there were also the comments that were "we told the project managers what didn't work before and what would work and some possibilities to look into and they didn't listen". THAT is arrogance, and regardless of the silos, ignoring advice like this is dooming your project to failure.
[insert witty comment here]
She started out by developing the industry standards, and then learned all to quickly to play Government Fat Cat. When we look back at the contributions to science that came out of Bell Labs, both in communications and computer science; it is easy to see that this was once truly an industry giant.
But like all giants, when you get used to playing 800 pound gorilla, you stop thinking and just keep throwing your weight around.
Even after it became public knowledge that Ma Bell was holding back technological advancement for their own purposes and profit, as long as the lobby on the hill kept a few important palms crossed, the tyranny continued. Finally, after a couple rounds of public humilliations and rebukes, the government was forced to order the split-up.
But very deeply imbedded in each and every part of the baby Bells was the crippling notion that they were the best and only company and that the thought of changing their behavior neven even had the slightest possibility of beginning to cross their tiny little corporate brains.
To make a long story short, their corporate egos never evolved back to being lean mean compedetive machines. If there ever was a company that should get back to it's roots of research and innovation this would have been it; but the chance is gone.
My local baby Bell, for example, relies on their internet customers to have their error checking turned off, when they visit the customer service website. As a developer I keep mine turned on and get about a half-dozen errors when each page loads, and a few more with each and every control encountered. Why is it that they still behave like the customer doesn't matter? Because in each division there is at least 1 fat cat who is more concerned with their own well being than anything else; and someone who profits by their actions does their level headed best to keep them there.
Whatever happened to quality of service?
"Can there be a Klein bottle that is an efficient and effective beer pitcher?"
personal development. it takes training to become good in your field. they have in-the-house training.
Spoken like someone who has either (1) never hired or dealt with a Big 5 consulting firm, or (2) someone who works for one.
The partners in these firms will sell anything. They will claim expertise in any thing they must in order to start billing. They send out green college recruits and charge over $100/hr for them.
As for "in-the-house training", the only experience most of them have is a series of failed PeopleSoft deployments (or Ariba, insert your own "enterprise" software here). Many of them have degrees in something completely unrelated to the project. Yes, the English major from Duke is very intelligent, but she doesn't know anything about the project at hand.
-they pay hard on their mistakes. Cash. When you buy outside your organization your excpectations are higher
Well, they may pay for their mistakes, but they certainly don't pay cash. These guys will absolutely bleed a company through continued billing regardless of how successful the project is.
(Rant Mode On)
After going through 5 SonyEricsson T68i's in 3.5 mo, only to have each successive one start refusing to make outgoing calls, and juggling many hours and dropped calls with AT&T customer service only willing to send me another T68i, I just decided to ditch AT&T and go to T-Mobile. I found a Nokia 3650 on Amazon for $250 w/ 2-$150 rebates - one from T-Mobile and the other from Amazon.
The new phone, and T-Mobile service have been perfect here in Pittsburgh, and I'm enjoying the nationwide unlimited GPRS with bluetooth from my iBook, but AT&T screwups still continue. Three weeks after I switched I receive a spankin' new T226 in the mail. Phone works much better than the T68i, but no bluetooth, and a crappy screen. But hey, free phone I'll never use.
Unfortunately I still had 5 mo on my contract. Canceling stiffs me with a $180 charge, so I called AT&T 6 weeks ago to switch to the $20 plan. (a $100 loss, cheaper than $180). Well, yesterday I get my bill and find out that they never processed my request. Call customer service and finally get a rep who tries to be helpful, but can't figure out why the logged plan switch wasn't carried out. Supposedly I'll be reimbursed, but I'm not holding my breath. The rep didn't seem surprised when I explained I'd switched because of all the previous cockups.
I could go on and on with AT&T screwup stories, but you get the idea. I think the biggest pain is that those still in contracts have few or no options other than biting the bullet and switching.
(Rant Mode Off)
While this is a sad story--especially about the poor guys with Indian "consultants" following them around asking a zillion questions about how to do their jobs--it's worthwhile to remember where the article appears: CIO magazine. CIO is focused on the needs/wants/interests of the guys in ties in a corporate IT environment--and in general a lot of CIOs think that outsourcing/offshoring is a hell of a good idea. The general tone of this article is "look at how these yobbos bungled the implementation of Siebel CRM." What they didn't mention at all is, "look at how these geniuses totally misunderstood their business, and pissed away roughly $40 billion in stock capitalization in just three years. And therefore died the death that they so richly deserved."
It's the technology, stupid...
There are companies, even in the 21st century, that can ignore cutting-edge technology. You don't need to be e-commerce enabled to be a plumber. But if you're in the wireless telephony business, in the midst of a headlong rush into a blizzard of new technologies, the core focus of your business isn't marketing or sales or re-carpeting the executive suite. Your core focus MUST be on the technology--and as soon as you lose sight of that focus, your competitors will consume you.
And these geniuses decided to offshore 3,000 jobs. And were doubtless shocked--shocked, I tell you!--to hear that employee morale about the developers was down.
I'm no techno-protectionist
I remember discussing the inevitable introduction of competition from overseas back in the late 1980s, and debating the possibility endlessly while working in Japan in the mid-90s. There will be companies that decide that, in their businesses, in their business models, IT work is a cost, not an investment. They will decide that they want to minimize that cost. They will focus on maintaining existing systems (with marginal, incremental improvements) and eschew major new developments. They will find that that approach may make it feasible to hire developers in the Third World. But those businesses that do so are making a conscious, deliberate decision: we're not going to focus the company on technology. We're going to try to minimize the company's dependence on technology. IT is a cost--it does not contribute to revenue.
For a wireless telephone company to take this position is simply insane: they are in the technology business. They are smack in the middle of a global technology race--one of the few technology races with competitors from practically every part of the northern hemisphere. They need to be faster to market with new products; the new products must be faster, better, more efficient, and more effective; and they have to have a world-beating customer service experience. Instead of fleeing from technology, they should be driven by it. They should be absolutely focused on it. They should be actively recruiting talent to build their strengths....
Because that's what every other company that's focused on technology is doing. Subcontracting out your technology--in a technology business--is sort of like farming, but buying all your crops at the supermarket.
I am not a lawyer...
But I am an engineering team leader at a U.S. electronics company that leads the world in our industry: lighting controls. We export electrical and electronic equipment to countries around the world--including Japan, South Korea, China, Taiwan, Singapore, Australia, and every country in Europe--because we focus on five core principles. And Principle #4 is "Innovate with high-quality products." In other words, we're in the technology business, so we focus--relentlessly--on the technology.
Once upon a time, AT&T did too...
AT&T Wireless was spun off from AT&T--but the corporate heritage is obviously there. And AT&T, once upon a time, ruled the world--literally chan
I am the CTO for a large enterprise software company (>$1B).
I spend about 30% of my time in front of the IT departments of the largest companies in the world, all of which are household names. They almost all tell me two things about our software:
1. It is heavily modified (they all have source)
2. They wish it was not
The fact is that these large customization projects, particularly ones which involve the Big 5, are over budget and late by factors that would boggle the minds of most mortals. It is not uncommon for these companies to spend >$100M for a software upgrade ON A SINGLE SITE. These companies have hundreds of sites.
As a contrast, another $9B electronics company I met with a few weeks ago can install a complete factory, including financials, manufacturing, logistics, scheduling, human resources, and reporting, all in less than 6 weeks. They have done it over 100 times. How do they do it? They have the entire cookie-cutter system burned on a DVD. Literally no customization is allowed at the plant level.
The only way to be successful at these kind of projects is to use an axe, not a scalpel. AT&T Wireless tried to use a scalpel. They should have thrown out all that junk and started over.
I would also point out that if you read the CIO's biography, he is an advisor to HP. Notice that they also chose HP as their outsourcing partner!
Can you say "conflict of interest"?
"Ironically, the scathing and sometimes highly sarcastic commentary at the end of the article from former employees makes this read even better."
Speak for yourself. This is not good news for me, as I'm a long time AT&T Wireless Services customer. I chose them because they offered the best service, and now they're being bought out by the company I was trying to get away from.
Fred
"A fool and his freedom are soon parted"
-RMS
About six years ago, I was a contractor at ATTWS in Paramus, NJ, working on the deployment of their point-of-sale cell phone activation app.
Without a doubt, it was the most dysfunctional office of a fortune 500 company I had ever seen, and I've seen a few. There were about six absolutely brilliant people there, who I would be glad to work with anywhere else, and a few hundred that I wouldn't trust to flip the proverbial burgers without putting someone's eye out.
Just one example: there was a pointy-haired middle manager there who liked to gather about $2k/hour worth of consultants into a conference room twice a week and just expound at length upon his management philosophy.
I was required to attend meetings on "planning my career path at ATTWS". I was a *contractor*, and I had work to do that didn't include making busy work for HR drones who didn't grok that ATT was a CUSTOMER, not a CAREER for me.
I even went to a meeting (again, mandatory), to hear some blithering bureaucrat tell us about ATTWS's process for developing processes. (I swear, I'm not making this up.)
Thank Judge Bell for opening those clowns up to competition. Somebody had to eat their lunch, and I wish every other cell phone company the best of luck.
-jcr
The only title of honor that a tyrant can grant is "Enemy of the State."
Are ther any good Siebel stories? I'm curious!
Has anyone ever checked out the AT&T wireless coverage maps? Your 'local calling area' (the places where you don't roam') are like a shade of orange lighter than the 'roaming' area. So to many males and people with less than good eyesight, it all looks the same.
Check it out on your own at http://www.mlife.com
I'm convinced that some companies just have a dysfunctional corporate culture that's immune to real reform. Their only hope is that things get so bad that all the top idiots lose their jobs -- and they're very, very lucky in choosing their new management. (That's basically what saved IBM.) But AT&T's so far gone, not even a total shakeout can save them.
Ditto. Although in my case, I wasn't trying to get away from any other company, I was consolidating and getting national coverage. I'm just sorry I wasn't aware of this in January. I got some song-and-dance when I said I wanted to switch my phone number. I decided, to hell with it, and just dropped the old number.
In any case, the quality of signal is abominable. I think that's because it's GSM. I can use the phone OK in Ohio, or Wisconsin, or Georgia, but I get unbelievably bad reception in MY OWN HOUSE here in Florida, and it doesn't work at all in my mother-in-law's house in Mississippi (oh, wait. Is that a bad thing?).
I think enough time has passed for me to drop this junk for Verizon (or even Sprint) - I _like_ CDMA.
namely, denial. the only piece that works is the old long lines department, now ATT business data. everything else with the "death star" logo is useless. outsourcing the people who are supposed to save you is the latest ATT lunacy, capping a string of them all the way back to divestiture.
thank god the baby bells got freed from that mess. all the folks vying to lead ATT in the 90s -- joe nacchio, mike annunziata, leo hindery, c. michael armstrong -- turned out to be a shitspread at their respective next stop in employment at the top of the tower. "little mikey" in particular broke up and sold his company down the river in several stages, then left it to hide out at comcast and count his money. "joey nachos" almost killed qwest, a fiber startup, and USWest together after he merged them to bleed the treasury at USWest. annunziata and hindery rode Global Crossing into the toilet, and hindery got into another telco startup and crashed with it.
moral: if you want to invest, check for former ATT execs on the board of a company. if you find any, flee in terror.
if this is supposed to be a new economy, how come they still want my old fashioned money?
It seems rather apt actually since the "Odyssey" is about Ulysses spending 20 years going every direction but the correct one to get to his destination.
Say all you want, that this was mismanaged, bureaucratic, a case of piss-poor consulting decisions, etc... but in my opinion, this is a textbook case of simply not following any sort of software development lifecycle method.... and yeah, I suppose management is part of that, but I simply don't think looking at this as merely a "poorly managed project" gets at the heart of it.
Any technology company needs to adopt and follow some sort of SDLC, and this is an obvious case where this has never been done. Criticising them for bungling this is all well and good, but I feel this article would've been better off talking about the real meat of "why" it failed.
I'm waiting for a new book along the lines of "The Japan That Can Say No" (sans the jingoism) -- about a new breed of CIO and IT manager that can push back when told to accomplish the impossible within an unrealistic timeframe.
Part of the problem is, I think, the New Success Story psychology. CEOs are so brainwashed into thinking that accomplishing the impossible is what defines you as a successful CEO, that they push their people to do absurdly difficult things in the most miniscume timespans. It's not doing the impossible that's a hallmark of a good CEO, it's doing the possible well and doing the impossible when you HAVE to -- not because it'll win you bragging rights.
(Of course, the whole question of what constitutes a "have to" in this case is probably open-ended.)
Honorary Member of Jackie Chan's Kung Fu Process Servers
...would seem to be a major survival skill at many corporate offices.
"Big government" is one of the political cliches I get really tired of. Anything you dislike about what the government does you can conveniently label as "big government". If the government won't let you burn your leaves, and you think that's dumb, it's "big government". But if you care about air pollution, it's government doing it's job. Your necessary program is my "big government".
You're entitled to criticize what the government does (indeed, it's more or less your obligation as a citizen!). But if you hope to actually accomplish anything, try to make your criticisms based on specifics, not vague, subjective terms that mean whatever you chose them to mean.
I've been in the situation before where you are fairly sure that things are going downhill in the company and that layoffs are already starting to sprout up.
It is an absolutly horrible environment to work in, to the point where you feel physically sick when you wake up in the morning to get ready to go to work. That being said i've seen some people in this situation fight to the last breath to try and prop the company back up. The difference in the AT&T wireless situation is that these employees knew that even if they did get the system up, they were going to get the shaft. I couldnt even imagine how horrible it would be to have people tag around with you to be your replacements.
In general I stick with small companies that cant afford the logistics of outsourcing. The apparent security of working for a big company is just an illusion if you have any morals and dont step on other people to keep your job.
The Ro Factor - Jeep/Linux Weblog
This is in response to the OBSCENE wait times for customer service. Yesterday, I had a 2:11:03 call in to customer service (my 611 records will prove the call times), and the issue is still not resolved.
AT&T is a voluntary signatory on September 9, 2003 to the Cellular Telecommunications & Internet Association's Consumer Code for Wireless Service.
Section 8 of the Consumer Code for Wireless Services states that the signatories will: "Ready Access" to solve account issues is not being provided by AT&T customer service.
Though AT&T is clearly in breach of at least one section of the Consumer Code for Wireless Service, they are still using the seal to advertise their adherence to the contract. I believe falsely advertising adherence to the Consumer Code warrants service agreements signed under this false pretense null and void.
-extremely dissatisfied customer
As a former member of the Odyssey II Environment Support team I had to live through this hell. As the article mentioned each "environment" consisted of somewhere between 12-16 individual systems. The team consisted of a dozen employees/contractors that had to install and maintain each system. Each employee was assigned two or three systems which they had to become the "experts." Because of the complicity and vast number of changes to each system it was difficult to become proficient in more then two or three. This approach sounds okay until you factor in all the development and testing environments required and the long hours of testing.
At the time I there were 18 different environments that were up and running during the day. By day, I mean expected to be up and running between the hours of 7:00am until 10:00pm. The major enviorments (Siebel Dev, System Test, Integration, etc...) ran until midnight. Any changes to the environment had to take place after hours. With the average "kit" install taking between four to six hours it meant the we were running a 24-hour shop.
They tried to split the 12 members of the team into three 8-hour shifts. With each member only trained three systems that meant we could only cover 12 of the 16 systems with four employees. Multiply that by the 18 environments and you can see where the troubles begun. Those 8-hour days turn to 10 and then 12. None of the environment were stable and consistently were down during testing due to bad code. Emergency kits were commonplace and since installation were so long (due to Siebel's shitty product) testing was always behind requiring weekends as well. All this added up to 70+ hours of work, 0 sleep (had a newborn at the time), and one VERY pissed off wife.
I was lucky enough to have left to get another job just before the system went live. It was obvious that it was going to fail and I had a huge shit-eating grin on my face when I heard of all their troubles.