More on AT&T Wireless's Bungled System Upgrade

An anonymous reader writes "CIO.com has posted a very in-depth article on the recent failings of AT&T Wireless that resulted in the state of the company today. What's fascinating about this article is the sheer amount of accurate information gleaned from former and current employees on the company's bungled attempts to follow FCC mandates on local number portability last November, the inside story on outsourcing efforts, and terrible executive management decisions that ultimately led to its demise. Ironically, the scathing and sometimes highly sarcastic commentary at the end of the article from former employees makes this read even better."

Comments

Best quote ever

[Space cowboy]

'... many of us spent 3-4 days awake, with minimal sleep, listening to "Mr. Jazz Hands" (Deloitte leader), lie his way through the explanations as to why it was deploying so poorly ... "We greatly underestimated the complexity of the integration of Siebel 7". What a load of crap! "We" = Upper management and Deloitte.'

[huge grin] wonder how he got *that* nickname :-)

Seriously, someone who has new/wonderful management process *can* help a company tread a dangerous path, but you *need* the domain expertise to be well-represented in a solution to a technical problem. Imho, at least. From the reports, it seems the process wasn't that wonderful, either :-)

His 'new operating model' was such a joke that one new VP hired to help implement it resigned after less than a month on the job.

You can expect a certain level of bitterness in ex-employees, especially after a disaster, but there ain't no smoke without fire, and when there's lots of smoke, start looking for the towering inferno...

Simon.

AWE did it to themselves

[stecoop]

For some reason I just cant feel sorry for a company not being able to rally its workers and threaten the workers of off shoring their work. I believe that AWE got exactly what it deserved - number portability was nothing new and they should have been able to get the job done. Yet AWE insisted on moving towards outsourcing instead of figuring out what needed to be done. I have seen similarly situations where no matter how much cheaper labor you look for, if you can't devise the plan, no one will be able to follow it. Good riddance to AWE and I wonder if Cingular is going forward with the outsourcing.

Re:AWE did it to themselves

[vk2]

Don't know which world you are in when you say number portability was nothing new and they should have been able to get the job done. LNP was implemented and enforced for the first time with US carriers during the times AT & T had problems.

I have a first hand experience of working on the LNP issues and no way its as easy as it was said in plain english. Agreed that you might think it as changing some sort of dns information - but the way its implemented and the patchy design as you go approach for the whole thing is really insane.

The whole thing would have been better designed if the war lords at Verizon and other had not banked on getting the mandatory deadline extended. None of the carriers actually believed that the nov dead line would be final - hence none of them actually cared to the design and implementation. Resourceful and gutsy teams at big companies like Verizon and Cingular pulled it through and companies like AWE and sprint paid heavily with customer dissatisfaction.

Re:AWE did it to themselves

[LaCosaNostradamus]

Certainly. Here's a quote from the article:

Former employees say morale wasn't helped by Corrado's first presentation to the IT group, in which they say he proclaimed, "Come in every day and expect to be fired." Intended to inspire the troops to greater effort, the talk backfired, says another former employee.

Although the quote is probably out of context, telling people they should expect to be fired at any time is probably a motivational technique learned in today's MBA environment. Force and fear doesn't even work well with prisoners. So why do MBAs and other assorted managerial parasites think they work for tech?

Newsflash for Managers {tm}: People expecting to be fired, will make their own plans for their futures instead of working 100% at your projects. Like, duh, eh?

Re:AWE did it to themselves

[Ironica]

Newsflash for Managers {tm}: People expecting to be fired, will make their own plans for their futures instead of working 100% at your projects. Like, duh, eh?

That was pretty much my response. It's one thing to let people know that if they don't meet your (clearly defined and within the realm of the possible) standards, they won't be kept around for sentimental reasons. But generally, if you want loyalty you have to give it.

I think maybe these people are operating under the false belief that, if their employees think they might be replaced, they'll work harder to try to prove themselves more valuable. But they don't realize that people are much more pessimistic these days than they were during the dot-com boom. If outsourcing is being considered, it comes across as an inevitable death-knell, not a spur to do better.

We Geeks Are Humans Too!

[osewa77]

Meanwhile, rumors of layoffs and offshore outsourcing began swirling around Odyssey. "[The rumors] slowed things down," says a former employee. "When stuff like that happens, people start looking for other work. I know I was looking for other work when I should have been testing."

It is a basic fact that most people need to be happy, content, secure to produce great work. It is another basic fact that managers will never stop lookiing for ways around that 'limitation'!
- a young blogger

Re:We Geeks Are Humans Too!

[gurustu]

To be fair, not all managers are like this. Some managers do their best to create bubbles of sanity and stability, to allow their teams to do the very best work possible. The best of managers will recognize that the largest part of their job is to be a drama sponge ... absorb the drama radiating down, dampen the drama radiating up, and shield their team to let them get work done.

However, what can a manager do when a project or organization starts to underperform? Too frequently, managers will attempt to improve performance by "shaking things up". They think "Okay, after one more reorganization, then things will be right." or "Well, if we put this process into place, it will be perfect."

The problem is really that management is as much an art as writing code or designing systems. There are a lot of people out there who can do it when the problems are normal, when the future is well understood. Unfortunately, most of them respond in counterproductive ways when confronted with crisis and uncertainty.

In short (too late!), it isn't that managers look for ways to stop people from being "happy, content, secure" out of spite. It's just that they're trying to do something, anything to make things better. They flail, and things get worse.

The problems go back at least 5 years.

[zerofoo]

I had AT&T wireless 5 years ago in college and they were no better then. Bad coverage areas, customer service people without a clue, and screwed up billing.

Say what you like about Verizon's CDMA technology. There is more to a phone company than the technology they choose. If the company can provide robust coverage, provide wireless broadband, and treat me right - they will succeed.

As for the CDMA detractors: Try getting 500k/sec. on ANY GSM system now....Verizon's testing it in DC and NY and will soon roll out nationwide.

AT&T wireless didn't fall apart because of their technology choices...they fell apart because they treated their customers badly.

-ted

Re:The Continuing Saga of the Death of Ma Bell

[fatman22]

What ever happened to quality of service? It's dead my friend. As the AT&T CIO put it so clearly "we work to achieve best-in-class margins." Quality be damned, he's going for maximum profit. That attitude is epidemic these days and I blame its existence on the CEO/CIO/C-whatever management model. Their pay and bonuses depend more on happy shareholders than happy customers and when they finish running off all the customers and employees at one place they just move on.

The End of AT&T

[fm6]

Everything, and I mean everything AT&T has done since they spun off their operating companies has turned to shit. Computers. exchange equipment, long distance service, broadband, and of course cellular service. The final humiliation was when they were booted from the DJIA to make a place for one of their own spinoffs!

I'm convinced that some companies just have a dysfunctional corporate culture that's immune to real reform. Their only hope is that things get so bad that all the top idiots lose their jobs -- and they're very, very lucky in choosing their new management. (That's basically what saved IBM.) But AT&T's so far gone, not even a total shakeout can save them.

Re:Big 5 consultants

[nathanh]

I'd like to know why.

I can think of two reasons.

First, I think there are sufficiently talented and experienced people to implement perhaps 5% of technology projects being built today. There is simply such a high demand and such a low supply of IT professionals that the market is completely off kilter. That's why salaries are so high and the average skill level is so low. The worst part about the high salaries is that they attracted terribly unskilled people who don't care about IT but only care about the money.

Second, the IT field is so freaking complex it defies imagination. There is simply too much to know. So you have these specialists who know only their narrow field, but inevitably those fields go out of fashion and the former specialist joins another field they have no experience in. It's a vicious cycle caused by (I think) the fact that IT isn't truly a mature industry. It's a research field that has been adopted too early by other industries. So there's lots of change which leads to regular retraining and inexperienced workers.

Re:Big 5 consultants

[PCM2]

We've all dealt with them before, they are usually intelligent people but have no expertise or experience in the task they are being paid to complete.

Yet again and again, despite all their failings, they are being hired by big corporations for major projects.

I'd like to know why.

I can think of one legitimate (if sad) reason: All too often, companies bring in consultants because it would be absolutely impossible to get anything done any other way. Their own corporate cultures are so rife with political infighting, bureacracy, and years of inertia driving legacy processes that no decisions can be made and no actions can be taken, except one: Bring in the consultants.

Then begins the months of meetings that turn into screaming matches once the emasculated junior management at the company have a scapegoat (the consultants) upon whom to lay the blame for their own impotence. Eventually the consultants figure out who's got the most signing authority for their checks and they start telling those people what they want to hear.

So you can only really half blame the consultants. Every few years they come up with a new little portfolio of tricks to flash around (outsourcing, for example), but your corporate execs still have to sign off on all this stuff. When a company goes down the tubes, you really can't blame anybody but its own senior management.

Yeah, the corporate world sucks, don't it?

Re:Big 5 consultants

[ChilyWily]

Yet again and again, despite all their failings, they are being hired by big corporations for major projects. I'd like to know why.

One big reason (that I've observed a 2 places where I've worked) is that upper management makes these decisions unilaterally - the subordinates (including middle management) rarely are involved at all. Upper management also tends to be clueless - especially when it comes to technical issues (which require true foresight) that these consultants boast of. I've read the comments about tunnel vision etc. but I think it is highly demotivating to ignore the opinions of the people who work for you and understand the real issues closely. If they say that the consultant is a sham, then there is good reason to depend on their judgement. Sadly, this doesn't happen very often. Why? politics :)

Re:How long to make back the 100 Mil?

[MillionthMonkey]

HP used to make extremely well-made products. Then they shifted their focus, from serving customers to serving stockholders. Now they produce garbage. I was in Fry's today and overheard a lady returning a cheap HP printer she had foolishly spent money on. "Their customer support is horrible and they're all from India!" I almost burst out laughing. Sucker! NEVER buy anything from HP. They've been coasting on their name for years and they're slowly grinding to a halt. Soon "HP" will be synonymous with crap and they'll have to change their name to something like "Claria" the way Gator did.

If it weren't for the printer ink racket they're running, they'd have gone under long ago. What a sad end to what used to be a great American company.

Re:RTFA?

[Ironica]

For some reason I just cant feel sorry for a company not being able to rally its workers and threaten the workers of off shoring their work. ... AWE insisted on moving towards outsourcing instead of figuring out what needed to be done. I have seen similarly situations where no matter how much cheaper labor you look for, if you can't devise the plan, no one will be able to follow it.

Did you RTFA? I did...WLNP went live on Nov 24th. Offshoring hadn't begun by then. Blame this on the 200$/hr consultants from D&T.

Did you read the quote you posted? The planning had begun, and people saw the signs... while they were up against an immovable deadline on number portability. No doubt D&T can share the blame generously, but frankly, when you're counting on your employees to complete a very difficult job is not the time to be plotting to fire them all and replace them with cheap knock-offs.

The CIO That Can Say No

[The Gline]

I'm waiting for a new book along the lines of "The Japan That Can Say No" (sans the jingoism) -- about a new breed of CIO and IT manager that can push back when told to accomplish the impossible within an unrealistic timeframe.

Part of the problem is, I think, the New Success Story psychology. CEOs are so brainwashed into thinking that accomplishing the impossible is what defines you as a successful CEO, that they push their people to do absurdly difficult things in the most miniscume timespans. It's not doing the impossible that's a hallmark of a good CEO, it's doing the possible well and doing the impossible when you HAVE to -- not because it'll win you bragging rights.

(Of course, the whole question of what constitutes a "have to" in this case is probably open-ended.)

Political Cliche time

[fm6]

This is why it is better for corporations to fail than for governments to fail.

That only works if, when it fails, the company goes out of business. Doesn't always work that way. AT&T has been failing, over and over, for 20 years! Not only is it still there, but nobody seems to have lost their job over it. I don't mean the peons who can downsized because of other people's decisions -- I mean the people who made the decisions.

This is why November 2004 is going to be so hard a choice. Who do vote for? Oh dear oh dear, two choices and two paths to bigger government.

Oops, didn't realize you were simply setting us up for an anti-beltway rant. Which means we're already way offtopic. But before the moderators attack, let me respond.

"Big government" is one of the political cliches I get really tired of. Anything you dislike about what the government does you can conveniently label as "big government". If the government won't let you burn your leaves, and you think that's dumb, it's "big government". But if you care about air pollution, it's government doing it's job. Your necessary program is my "big government".

You're entitled to criticize what the government does (indeed, it's more or less your obligation as a citizen!). But if you hope to actually accomplish anything, try to make your criticisms based on specifics, not vague, subjective terms that mean whatever you chose them to mean.