Slashdot Mirror


More on AT&T Wireless's Bungled System Upgrade

An anonymous reader writes "CIO.com has posted a very in-depth article on the recent failings of AT&T Wireless that resulted in the state of the company today. What's fascinating about this article is the sheer amount of accurate information gleaned from former and current employees on the company's bungled attempts to follow FCC mandates on local number portability last November, the inside story on outsourcing efforts, and terrible executive management decisions that ultimately led to its demise. Ironically, the scathing and sometimes highly sarcastic commentary at the end of the article from former employees makes this read even better."

28 of 285 comments (clear)

  1. Best quote ever by Space+cowboy · · Score: 4, Insightful

    '... many of us spent 3-4 days awake, with minimal sleep, listening to "Mr. Jazz Hands" (Deloitte leader), lie his way through the explanations as to why it was deploying so poorly ... "We greatly underestimated the complexity of the integration of Siebel 7". What a load of crap! "We" = Upper management and Deloitte.'


    [huge grin] wonder how he got *that* nickname :-)

    Seriously, someone who has new/wonderful management process *can* help a company tread a dangerous path, but you *need* the domain expertise to be well-represented in a solution to a technical problem. Imho, at least. From the reports, it seems the process wasn't that wonderful, either :-)

    His 'new operating model' was such a joke that one new VP hired to help implement it resigned after less than a month on the job.

    You can expect a certain level of bitterness in ex-employees, especially after a disaster, but there ain't no smoke without fire, and when there's lots of smoke, start looking for the towering inferno...

    Simon.
    --
    Physicists get Hadrons!
  2. AWE did it to themselves by stecoop · · Score: 5, Insightful

    For some reason I just cant feel sorry for a company not being able to rally its workers and threaten the workers of off shoring their work. I believe that AWE got exactly what it deserved - number portability was nothing new and they should have been able to get the job done. Yet AWE insisted on moving towards outsourcing instead of figuring out what needed to be done. I have seen similarly situations where no matter how much cheaper labor you look for, if you can't devise the plan, no one will be able to follow it. Good riddance to AWE and I wonder if Cingular is going forward with the outsourcing.

    1. Re:AWE did it to themselves by vk2 · · Score: 4, Insightful
      Don't know which world you are in when you say number portability was nothing new and they should have been able to get the job done. LNP was implemented and enforced for the first time with US carriers during the times AT & T had problems.

      I have a first hand experience of working on the LNP issues and no way its as easy as it was said in plain english. Agreed that you might think it as changing some sort of dns information - but the way its implemented and the patchy design as you go approach for the whole thing is really insane.

      The whole thing would have been better designed if the war lords at Verizon and other had not banked on getting the mandatory deadline extended. None of the carriers actually believed that the nov dead line would be final - hence none of them actually cared to the design and implementation. Resourceful and gutsy teams at big companies like Verizon and Cingular pulled it through and companies like AWE and sprint paid heavily with customer dissatisfaction.

      --
      No Sig for you.!
    2. Re:AWE did it to themselves by LaCosaNostradamus · · Score: 4, Insightful

      Certainly. Here's a quote from the article:

      Former employees say morale wasn't helped by Corrado's first presentation to the IT group, in which they say he proclaimed, "Come in every day and expect to be fired." Intended to inspire the troops to greater effort, the talk backfired, says another former employee.

      Although the quote is probably out of context, telling people they should expect to be fired at any time is probably a motivational technique learned in today's MBA environment. Force and fear doesn't even work well with prisoners. So why do MBAs and other assorted managerial parasites think they work for tech?

      Newsflash for Managers {tm}: People expecting to be fired, will make their own plans for their futures instead of working 100% at your projects. Like, duh, eh?

      --
      [You have a stable society when some nut guns down a schoolyard and the law doesn't change.]
    3. Re:AWE did it to themselves by Ironica · · Score: 4, Insightful

      Newsflash for Managers {tm}: People expecting to be fired, will make their own plans for their futures instead of working 100% at your projects. Like, duh, eh?

      That was pretty much my response. It's one thing to let people know that if they don't meet your (clearly defined and within the realm of the possible) standards, they won't be kept around for sentimental reasons. But generally, if you want loyalty you have to give it.

      I think maybe these people are operating under the false belief that, if their employees think they might be replaced, they'll work harder to try to prove themselves more valuable. But they don't realize that people are much more pessimistic these days than they were during the dot-com boom. If outsourcing is being considered, it comes across as an inevitable death-knell, not a spur to do better.

      --
      Don't you wish your girlfriend was a geek like me?
    4. Re:AWE did it to themselves by sphealey · · Score: 3, Insightful
      AT&T also erred in spinning off AT&T wireless as a tracking stock (A mistake Sprint has soon learned) and hiring Michael Armstrong as CEO
      Keep in mind though that during that entire time period Worldcom was lying about its financials to Wall Street, to the tune of $40 billion. AT&T was under tremendous pressure from the financial world to match Worldcom's performance, which of course they could never have done since Worldcom wasn't really doing it either.

      Under Armstrong, AT&T had the only large-scale telecom strategy that I thought would work: a comprehensive menu of business and high-value consumer services, all under one roof. Two problems: they never got the divisions to work together. And they way overpaid for the cable assets.

      sPh

  3. We Geeks Are Humans Too! by osewa77 · · Score: 5, Insightful
    Meanwhile, rumors of layoffs and offshore outsourcing began swirling around Odyssey. "[The rumors] slowed things down," says a former employee. "When stuff like that happens, people start looking for other work. I know I was looking for other work when I should have been testing."
    It is a basic fact that most people need to be happy, content, secure to produce great work. It is another basic fact that managers will never stop lookiing for ways around that 'limitation'!
    - a young blogger
    1. Re:We Geeks Are Humans Too! by Anonymous Coward · · Score: 1, Insightful

      It is another basic fact that managers will never stop lookiing for ways around that 'limitation'!

      Hmm, bad managers for sure, but I've worked with many managers who were well aware of the rules. They were always concerned with the team's comfort and happiness, sometimes sacrificing their own for the team's. I've worked for some PHBs, but their attitude is not universal in any way.

    2. Re:We Geeks Are Humans Too! by gurustu · · Score: 5, Insightful
      To be fair, not all managers are like this. Some managers do their best to create bubbles of sanity and stability, to allow their teams to do the very best work possible. The best of managers will recognize that the largest part of their job is to be a drama sponge ... absorb the drama radiating down, dampen the drama radiating up, and shield their team to let them get work done.

      However, what can a manager do when a project or organization starts to underperform? Too frequently, managers will attempt to improve performance by "shaking things up". They think "Okay, after one more reorganization, then things will be right." or "Well, if we put this process into place, it will be perfect."

      The problem is really that management is as much an art as writing code or designing systems. There are a lot of people out there who can do it when the problems are normal, when the future is well understood. Unfortunately, most of them respond in counterproductive ways when confronted with crisis and uncertainty.

      In short (too late!), it isn't that managers look for ways to stop people from being "happy, content, secure" out of spite. It's just that they're trying to do something, anything to make things better. They flail, and things get worse.

    3. Re:We Geeks Are Humans Too! by thisgooroo · · Score: 2, Insightful
      To be fair, not all managers are like this.

      in my experience, the majority of managers (at least those i have worked with) are ok. it's just that the jerks who give you the impression that you work under them can destroy any working relationship with a company

      The best of managers will recognize that the largest part of their job is to be a drama sponge ... absorb the drama radiating down, dampen the drama radiating up, and shield their team to let them get work done.

      you'll see that mostly with the managers you work with directly. the upper levels have more contact with those people who are only worried about the next quarterly result

      However, what can a manager do when a project or organization starts to underperform? Too frequently, managers will attempt to improve performance by "shaking things up". They think "Okay, after one more reorganization, then things will be right." or "Well, if we put this process into place, it will be perfect."

      a real solution would require an analysis of the reasons and careful planning what to do about them. in our corporate culture, this would require too much time. so even decent managers are forced to try to fix things using the "hope this will work" method

  4. The problems go back at least 5 years. by zerofoo · · Score: 4, Insightful

    I had AT&T wireless 5 years ago in college and they were no better then. Bad coverage areas, customer service people without a clue, and screwed up billing.

    Say what you like about Verizon's CDMA technology. There is more to a phone company than the technology they choose. If the company can provide robust coverage, provide wireless broadband, and treat me right - they will succeed.

    As for the CDMA detractors: Try getting 500k/sec. on ANY GSM system now....Verizon's testing it in DC and NY and will soon roll out nationwide.

    AT&T wireless didn't fall apart because of their technology choices...they fell apart because they treated their customers badly.

    -ted

  5. Re:SILOS by Snover · · Score: 3, Insightful

    I did infer the saboutaging that was probably going on from the article but there were also the comments that were "we told the project managers what didn't work before and what would work and some possibilities to look into and they didn't listen". THAT is arrogance, and regardless of the silos, ignoring advice like this is dooming your project to failure.

    --

    [insert witty comment here]
  6. Re:How long to make back the 100 Mil? by fbform · · Score: 2, Insightful

    I wonder how many man-years of outsourcing it will take to make back that 100 mil AT&T lost?

    I see your point, but I'd like to remind you that $100M is not as much as you think it is. Another proponent of outsourcing - Carly Fiorina of HP - nearly got a $115M bonus deal (to have been shared with Michael Cappelas of Compaq) for the HP-Compaq merger.

    My point is that if some companies are prepared to throw money like this at their CEOs, they probably don't really care about losing some customers.

    Then again, I could be wrong. In this particular case, AT&T Wireless lost customer goodwill more than money.

    --
    Time flies like an arrow. Fruit flies like a banana.
  7. RTFA? by andy1307 · · Score: 3, Insightful
    Meanwhile, rumors of layoffs and offshore outsourcing began swirling around Odyssey.

    We'd see whiteboards that had questions like, 'What opportunity do we have to offshore/outsource?'"

    On Nov. 19, The Wall Street Journal ran a story on planned layoffs and outsourcing at AT&T Wireless

    Did you RTFA? I did...WLNP went live on Nov 24th. Offshoring hadn't begun by then. Blame this on the 200$/hr consultants from D&T.
    1. Re:RTFA? by Ironica · · Score: 4, Insightful
      For some reason I just cant feel sorry for a company not being able to rally its workers and threaten the workers of off shoring their work. ... AWE insisted on moving towards outsourcing instead of figuring out what needed to be done. I have seen similarly situations where no matter how much cheaper labor you look for, if you can't devise the plan, no one will be able to follow it.
      Did you RTFA? I did...WLNP went live on Nov 24th. Offshoring hadn't begun by then. Blame this on the 200$/hr consultants from D&T.

      Did you read the quote you posted? The planning had begun, and people saw the signs... while they were up against an immovable deadline on number portability. No doubt D&T can share the blame generously, but frankly, when you're counting on your employees to complete a very difficult job is not the time to be plotting to fire them all and replace them with cheap knock-offs.
      --
      Don't you wish your girlfriend was a geek like me?
  8. Re:Big 5 consultants by Anonymous Coward · · Score: 2, Insightful

    Christ! Is this the only view of managers? That all we do is try to cover our own asses, take credit from our workers, and have tiny shrivelled balls when it comes to dealing with our seniors?

    While some managers may cover their asses with a consulting firm or the refusal of another department to pay for the firm when a project fails, most will take credit for the failure because it was *their* project. If a consulting firm isn't going to help, they won't recommend it. The reasons that consulting firms are used aren't because they shake up the workers. Its because they take away the tunnel vision that anyone who has worked on a long project faces. Thats it, nothing else. Sure they have smart guys, but they aren't going to finish the project for you by themselves.

  9. That's your opinion by ThisIsFred · · Score: 2, Insightful

    "Ironically, the scathing and sometimes highly sarcastic commentary at the end of the article from former employees makes this read even better."

    Speak for yourself. This is not good news for me, as I'm a long time AT&T Wireless Services customer. I chose them because they offered the best service, and now they're being bought out by the company I was trying to get away from.

    --
    Fred

    "A fool and his freedom are soon parted"
    -RMS
  10. Re:The Continuing Saga of the Death of Ma Bell by fatman22 · · Score: 5, Insightful

    What ever happened to quality of service? It's dead my friend. As the AT&T CIO put it so clearly "we work to achieve best-in-class margins." Quality be damned, he's going for maximum profit. That attitude is epidemic these days and I blame its existence on the CEO/CIO/C-whatever management model. Their pay and bonuses depend more on happy shareholders than happy customers and when they finish running off all the customers and employees at one place they just move on.

  11. Re:How long to make back the 100 Mil? by DAldredge · · Score: 2, Insightful

    Yes, and HP paid for that fucking bonus by firing 1000's of employees and gutting their R&D staff.

    By the time this hurts HP she will be gone and someone else will have to clean up her mess.

  12. The End of AT&T by fm6 · · Score: 4, Insightful
    Everything, and I mean everything AT&T has done since they spun off their operating companies has turned to shit. Computers. exchange equipment, long distance service, broadband, and of course cellular service. The final humiliation was when they were booted from the DJIA to make a place for one of their own spinoffs!

    I'm convinced that some companies just have a dysfunctional corporate culture that's immune to real reform. Their only hope is that things get so bad that all the top idiots lose their jobs -- and they're very, very lucky in choosing their new management. (That's basically what saved IBM.) But AT&T's so far gone, not even a total shakeout can save them.

  13. Re:Big 5 consultants by nathanh · · Score: 4, Insightful
    I'd like to know why.

    I can think of two reasons.

    First, I think there are sufficiently talented and experienced people to implement perhaps 5% of technology projects being built today. There is simply such a high demand and such a low supply of IT professionals that the market is completely off kilter. That's why salaries are so high and the average skill level is so low. The worst part about the high salaries is that they attracted terribly unskilled people who don't care about IT but only care about the money.

    Second, the IT field is so freaking complex it defies imagination. There is simply too much to know. So you have these specialists who know only their narrow field, but inevitably those fields go out of fashion and the former specialist joins another field they have no experience in. It's a vicious cycle caused by (I think) the fact that IT isn't truly a mature industry. It's a research field that has been adopted too early by other industries. So there's lots of change which leads to regular retraining and inexperienced workers.

  14. Re:Big 5 consultants by Anonymous Coward · · Score: 1, Insightful

    dude, I'm a programmer I work in the IT department of a large bank and i really don't think that most IT work is really that complex. the hardest part of IT work is getting the business people to decide what they want, and keeping what they want the same for the rest of the project. Don't get me wrong, there is plenty of stuff in computers that is interesting/hard/whatever, but most IT work can and is done by monkeys.

    I like my job, but I don't think most IT work is as complex as you think it is.

  15. Re:Big 5 consultants by PCM2 · · Score: 4, Insightful
    We've all dealt with them before, they are usually intelligent people but have no expertise or experience in the task they are being paid to complete.

    Yet again and again, despite all their failings, they are being hired by big corporations for major projects.

    I'd like to know why.

    I can think of one legitimate (if sad) reason: All too often, companies bring in consultants because it would be absolutely impossible to get anything done any other way. Their own corporate cultures are so rife with political infighting, bureacracy, and years of inertia driving legacy processes that no decisions can be made and no actions can be taken, except one: Bring in the consultants.

    Then begins the months of meetings that turn into screaming matches once the emasculated junior management at the company have a scapegoat (the consultants) upon whom to lay the blame for their own impotence. Eventually the consultants figure out who's got the most signing authority for their checks and they start telling those people what they want to hear.

    So you can only really half blame the consultants. Every few years they come up with a new little portfolio of tricks to flash around (outsourcing, for example), but your corporate execs still have to sign off on all this stuff. When a company goes down the tubes, you really can't blame anybody but its own senior management.

    Yeah, the corporate world sucks, don't it?

    --
    Breakfast served all day!
  16. Re:Big 5 consultants by ChilyWily · · Score: 4, Insightful
    Yet again and again, despite all their failings, they are being hired by big corporations for major projects. I'd like to know why.
    One big reason (that I've observed a 2 places where I've worked) is that upper management makes these decisions unilaterally - the subordinates (including middle management) rarely are involved at all. Upper management also tends to be clueless - especially when it comes to technical issues (which require true foresight) that these consultants boast of. I've read the comments about tunnel vision etc. but I think it is highly demotivating to ignore the opinions of the people who work for you and understand the real issues closely. If they say that the consultant is a sham, then there is good reason to depend on their judgement. Sadly, this doesn't happen very often. Why? politics :)
  17. Re:How long to make back the 100 Mil? by MillionthMonkey · · Score: 4, Insightful

    HP used to make extremely well-made products. Then they shifted their focus, from serving customers to serving stockholders. Now they produce garbage. I was in Fry's today and overheard a lady returning a cheap HP printer she had foolishly spent money on. "Their customer support is horrible and they're all from India!" I almost burst out laughing. Sucker! NEVER buy anything from HP. They've been coasting on their name for years and they're slowly grinding to a halt. Soon "HP" will be synonymous with crap and they'll have to change their name to something like "Claria" the way Gator did.

    If it weren't for the printer ink racket they're running, they'd have gone under long ago. What a sad end to what used to be a great American company.

  18. Re:Big 5 consultants by Knetzar · · Score: 2, Insightful

    When working as a consultant you should never admit you don't know. As one of my bosses once told me, if you seem like you know everything they'll continue to hire you, but if you're honest and ask for help from the place that hired you they'll realize that they don't need you as much as they thought they did.

  19. The CIO That Can Say No by The+Gline · · Score: 4, Insightful

    I'm waiting for a new book along the lines of "The Japan That Can Say No" (sans the jingoism) -- about a new breed of CIO and IT manager that can push back when told to accomplish the impossible within an unrealistic timeframe.

    Part of the problem is, I think, the New Success Story psychology. CEOs are so brainwashed into thinking that accomplishing the impossible is what defines you as a successful CEO, that they push their people to do absurdly difficult things in the most miniscume timespans. It's not doing the impossible that's a hallmark of a good CEO, it's doing the possible well and doing the impossible when you HAVE to -- not because it'll win you bragging rights.

    (Of course, the whole question of what constitutes a "have to" in this case is probably open-ended.)

    --
    Honorary Member of Jackie Chan's Kung Fu Process Servers
  20. Political Cliche time by fm6 · · Score: 4, Insightful
    This is why it is better for corporations to fail than for governments to fail.
    That only works if, when it fails, the company goes out of business. Doesn't always work that way. AT&T has been failing, over and over, for 20 years! Not only is it still there, but nobody seems to have lost their job over it. I don't mean the peons who can downsized because of other people's decisions -- I mean the people who made the decisions.
    This is why November 2004 is going to be so hard a choice. Who do vote for? Oh dear oh dear, two choices and two paths to bigger government.
    Oops, didn't realize you were simply setting us up for an anti-beltway rant. Which means we're already way offtopic. But before the moderators attack, let me respond.

    "Big government" is one of the political cliches I get really tired of. Anything you dislike about what the government does you can conveniently label as "big government". If the government won't let you burn your leaves, and you think that's dumb, it's "big government". But if you care about air pollution, it's government doing it's job. Your necessary program is my "big government".

    You're entitled to criticize what the government does (indeed, it's more or less your obligation as a citizen!). But if you hope to actually accomplish anything, try to make your criticisms based on specifics, not vague, subjective terms that mean whatever you chose them to mean.