Slashdot Mirror
A Possible Cause of AT&T's Wireless Clog — Configuration Errors
AT&T customers (iPhone users notably among them) have seen some wireless congestion in recent months; Brough Turner thinks the trouble might be self-inflicted. According to Turner, the poor throughput and connection errors can be chalked up to "configuration errors specifically, congestion collapse induced by misconfigured buffers in their mobile core network." His explanation makes an interesting read.
This is not really news at all. They spend little to nothing to keep their network up to the devices they have on it. This misconfiguration of buffers (if that is really a cause at all) is probably because they might not hire people with any knowledge of what they are doing to keep costs low.
Anything can be found funny, from a certain point of view.
His explanation makes an interesting read.
I'd like to think that's a given, considering it's a news story. At any rate, from TFA:
The bottleneck link is the over-the-air link, i.e. the connection from radio access network or UTRAN to the Mobile Statation (MS) in the above diagram, therefore the critical buffers are those at the UTRAN. In practice the UTRAN includes both the basestations (called Node-Bs) and the Radio Network Controllers (RNCs) which coordinate handovers between basestations (among other things). Because of hand-overs, the amount of data buffered at the Node-B is relatively small. It's the buffers at the RNC that must be large enough to deal with the delay variations in the radio network and yet small enough to induce packet loss when the network gets congested.
I am not a network engineer, but how exactly could 8 second ping time be not noticed by the AT&T engineers who set up, configured, and monitored their OTA link? I would think that we're not talking about some dude's set of bridged dd-wrt linksys routers, but some serious heavy-duty RF equipment. I'm thinking on the order of several zeros...
Whoever modded that down is a humorless dick.
One CPU cycle wasted on digital restrictions management is ONE TOO MANY.
How can you expect to have zero packet loss on a wireless network? That's just stupid.
Please fix this ATT!
(I'm not holding my breath... )
Every time I deal with AT&T I am amazed that anything works at all over there. My phone almost always shows five bars at home, yet frequently calls don't cause the phone to ring - they go to voicemail after pretending to ring. The jaded amongst us could suspect a deliberate misconfiguration of phones and signal strength monitoring. Similarly, it would not surprise me if AT&T data networks weren't about as reliable as the signal strength indicator on my phone. The recent alleged blurb from an Apple "genius" in NYC that 1/3 of all iPhone calls get dropped seems to point in that direction.
That a cell-phone won't work everywhere and perfectly every time is a given. However, wouldn't it be nice if the companies that stood behind these networks would actually be held accountable for some of the advertising statements they make? What it comes down to is that we're dealing with an oligopolistic market, where only a few carriers can achieve the scale and the coverage to satisfy most mobile customers most of the time. On the flipside, that also means that said carriers can be truly dismal when it comes to customer service, back-end efficiency, etc. since consumers don't have many choices. Considering the ongoing consolidation in the industry, the only way out seems to be a trust-busting activity on the part of the DoJ to regulate the industry.
Not sure that is the better alternative... nor what the best structure for a regulated market would be.
The public has shown repeatedly that it will value cost above quality
Right, that is why Apple laptop sales have tanked in the downturn. Oh, wait.
Why should AT&T attempt to improve the quality of its network if people are a) willing to pay for what they currently have, and b) won't pay for any attempts AT&T will make to improve quality.
Because those people if they dislike the network enough, will leave eventually. That is the motivation to improve on what they have now, never mind they want to stop the customers bitching who are losing them new customers right now. They have plenty of reasons, they even have plenty of money from the influx of iPhone people. There's more than enough motivation, it's more a question of execution now.
People will pay for quality. For some the cost is financial. For others, the "cost" is that they will not buy an iPhone while the AT&T network has issues.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Wouldn't be the first time, except maybe for AT&T.
I don't think that it's limited to just AT&T - I am in Australia, so have never even had to deal with them, but I am finding that in the vast majority of Australian companies as well, simple back to basics work quality is plummeting. Everything seems to be about making everything as cheap as possible - whether or not it even functions the way it is supposed to. That also goes for the majority of customer service dealings as well.
It seems that the "Do it once but do it properly" mentality is limited to very few people and businesses. I work as a business analyst and the amount of arguing I have to do with each project to get extra money spent to do things properly (the majority of the time it saves money in the long run anyhow for other projects - I am not even taking into account the maintenance and support savings into that equation) yet I seem to always have to fight the same battles over and over.
Moved to http://soylentnews.org/. You are invited to join us too!
That's why to the greatest degree possible, libraries, programs, and algorithms should be auto-tuning. You can provide all the knobs you want, but people won't actually touch them. They'll choose which library, application, or operating system they use based on the default settings, so you'd better damn well make sure the default settings are good --- or even better, that you don't need settings at all.
quality, and fashion, one not having to be the other...
Indeed, and that is why many companies built atop the foundations of showy fashion are gone now. Fashion is transient and fickle. Apple however delivers a quality product that delivers new customers through loyalty and word of mouth. If this were not so Apple would not be a tenth of what it is now.
It doesn't hurt that it is fashionable, too. But that is not why I and so many other people buy Apple products.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
This is the problem. Thanks to the competitive barriers (such as the inability to move phones between all but two of the top four networks, and none of the top 3) moving can take a long time (2 year contract must expire) before someone can move networks unless they want to pay a large fee.
You say that like it does not matter because the period of switchover is two years. But it matters a great deal still, because many people will still leave then (or if they are mad enough pay the fee). A company like AT&T must be forward thinking in what problems now do for subscribers in future years, never mind the day when T-Mobile is also carrying iPhones (we have seen Verizon will not happen).
It's rather amazing how many people go to AT&T for the iPhone. I think they said about 1/3 of their iPhone customers are coming from other networks. I wonder how many more people would get iPhones if it wasn't for their current contract?
Probably quite a few, as I said that was a factor too. Which is why the two year thing doesn't really matter for motivation because they are losing new customers in addition to the ones potentially lost in two years.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Right, that is why Apple laptop sales have tanked in the downturn. Oh, wait.
I think that will change once Windows 7 is mainstream. Everyone hated Vista. Now it seems like everyone loves 7 and Snow Leopard only got a "meh" response from reviewers (not because Snow Leopard is bad it just doesn't have anything revolutionary, the fact that 7 runs at a decent speed is considered to be "revolutionary" in the PC world). There are two people who use Macs, people who have grown up using Macs and people who prefer Macs. When faced with Vista, a lot of people started to realize they prefer Macs.
Because those people if they dislike the network enough, will leave eventually
"Eventually" isn't very soon when you have a 2 year contract with early termination fees that are through the roof.
That is the motivation to improve on what they have now, never mind they want to stop the customers bitching who are losing them new customers right now
All 4 major carriers suck though. Lets see here, AT&T has network issues and isn't cheap, T-Mobile might have great customer service, good phones but it has a pathetic amount of 3G coverage compared to the others. Verizon might have a great network, but it isn't exactly cheap and a lot of their phones (at least used to) suck terribly with many features being stripped out of them. Sprint might be cheap but their coverage isn't great.
And none of them have a phone with as many apps as the iPhone, yes, Android and WebOS are great, but they still don't have the amount of apps as the iPhone nor as much support from companies such as game developers and the like. And don't get me started on Windows mobile.....
For others, the "cost" is that they will not buy an iPhone while the AT&T network has issues.
Taxation is legalized theft, no more, no less.
I think that will change once Windows 7 is mainstream.
I don't think people care that much one way or the other. In fact studies have shown previous Windows releases increased Mac sales, and this will too - if you have to refresh a whole system, if you have to learn a new UI - why not a Mac?
I just can't see how Windows7 will have any impact at all in slowing down the Mac train.
All 4 major carriers suck though.
From experience with them all I totally agree, which is why I am not as much bothered by some people with the iPhone being AT&T only.
Verizon might actually improve if they don't tamper with the Droid much. ...And who is going to look at a few "geek" articles about the iPhone and decide not to get it?
It's not that at all. It's having a friend who complains about dropped calls all the time, or if he tries to the use the network and it's failing a lot. This kind of damage is all done at the word of mouth level.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Is to let AT&T engage in all the packet shaping and other fun stuff they want. Down with net neutrality? Wait? What's that? You think they'd manage to screw up their network even more then and probably fuck over other people in the process? Say it ain't so!
I think that what you are seeing is just the result of how business is conducted these days and how the money is allocated. currently I have the same perception about things where I work and I believe the change came when the company hired several "genius" executives that had degrees in finance, administration and such from reputable universities.
They came with the current trends in economic analisys "pre-programmed" (to be truthful, that is what is being teched at most universities now) and this type of mentality views projects just as a cash-flow problem, so any money spent "doing things properly" is money spent now, not in six months or two years or longer, so the calculate that a project finished early that just needs "tweaking" in a future date is better and cheaper than a project done the way it is supposed to be done (but taking longer). I see things like that every day and have been "bitten" by that kind of analisys in a couple recent projects.
Sorry if this seems like rambling, I tend to explain too much.
any money spent "doing things properly" is money spent now, not in six months or two years or longer, so the calculate that a project finished early that just needs "tweaking" in a future date is better and cheaper than a project done the way it is supposed to be done (but taking longer)
Yup, that's exactly what I am talking about, and I find it very frustrating. The time between project end and the final "tweaking" implementation where the project deliverable finally works as it is supposed to is both frustrating for the users, has a high support cost from a systems point and the "tweaks" normally end up adding much more to the cost itself than just doing it properly the first time.
I am reasonably lucky that these days I am involved in the early stages of some of the projects that I work on, and I start on the offensive for the most part, and ask for detailed analysis from project managers that I work with on the cost of the "cheap" and "proper" solutions over the space of a year or two if the project looks like it is trying to cut too many corners - and take that analysis to the program office - it's coming out of their pockets after all or on occasion directly to the business that is footing the bill for the project. While it works for the majority of the time, it's still amazingly frustrating to have to fight the same damned fight each time so that things are done properly. In my eye's it's up to the project managers to be ensuring that their projects are done properly and not end up as massive drains on support/systems.
Sadly, few of them see it that way. It's all about being cheap and cost cutting and meeting budget KPI's rather than arguing that the budgets are set too low.
Moved to http://soylentnews.org/. You are invited to join us too!
You're one of the few and I'm glad there are more of us around. It brings a little sense of peace to my world.
I've worked in the aerospace engineering and IT industries (both non-military/military companies) and it's like pulling teeth from a hippo to make sure some things are done properly. Only _one_ engineer understood this "do it once or don't do it at all" (verbatim, his words) philosophy. it was a quality you could see in his 45 years as a professional.
"They don't build them like they used to" was an old homage my grandfather always said. I'm starting to see what he might have meant.
Posting Anonymously because I don't feel like being associated with an unpopular viewpoint:
Sometimes 'good enough' really is the best option for the business as a whole. Techies and engineers often have a hard time accepting this until they've actually run a department, but it's true.
If we have $10m today, we might benefit more by doing 10 $1m projects 'pretty well' than by doing 7 $1.4m projects 'perfectly', for a number of reasons.
It could get something to market faster... it could be that the marketplace cares deeply about some features, but doesn't care quite so much about initial quality, it could be that it's just better to run 10 experiments and see which pan out well enough to put extra money in them later... it could just be that nobody is happy, but there's simply not enough cash to do what's really wanted.... so it's do it 'good enough' now, or not doing it at all.
I mean... there are shitloads of valid business reasons to purposefully do things a bit half-assed. Especially in very competitive markets, where there just isn't enough margin to pay for doing things "perfectly", or in markets that aren't meaningfully differentiated on quality.
I spend my life fighting with people like you... people who think I don't understand, or I'm shortsighted, or I'm just a robot. But here's the thing: I think *YOU* are the idiot.
I think you're too stupid to realize that something is better than nothing... and that's often the choice.
I think you're too stupid to realize that if customers aren't willing to pay for quality, then we likely won't have the budget to pay for it.
I think you're too stupid to realize that if we aren't sure what's happening to a product in a 5 year horizon, we'll take a lot more risk with it, on purpose. After all, if we think it might be going away, it's going to be the step-child. And we're not going to tell you in advance that it might be going away.
I think you're too stupid to realize that we're not idiots. We have reasons for our decisions.
And fortunately, not all techies and engineers are like you. Some are eminently reasonable people who understand that occasionally I just have to say "your budget is cut by 8%. Make it work." And that when I do so, I fully grasp that this will incur a number of tradeoffs... but there just isn't enough money.
But whatever... it doesn't matter. Because at the end of the day, the techie nerds will continue to have no respect for management... and then they'll wonder why they're treated with no respect in return.
Wouldn't be the first time, except maybe for AT&T.
I don't think that it's limited to just AT&T - I am in Australia, so have never even had to deal with them, but I am finding that in the vast majority of Australian companies as well, simple back to basics work quality is plummeting. Everything seems to be about making everything as cheap as possible - whether or not it even functions the way it is supposed to. That also goes for the majority of customer service dealings as well.
It seems that the "Do it once but do it properly" mentality is limited to very few people and businesses. I work as a business analyst and the amount of arguing I have to do with each project to get extra money spent to do things properly (the majority of the time it saves money in the long run anyhow for other projects - I am not even taking into account the maintenance and support savings into that equation) yet I seem to always have to fight the same battles over and over.
There's a simple reason for that: money is trivial to measure. Quality is much harder to measure. For example, failure rates like MTBFs often don't directly correlate into straight dollars and cents, but a small percentage chance that it might cost a large but unknown amount at some point in the future. This kind of thing confuses people, so they stick to the simple stuff. In an Excel spreadsheet, the solution that costs fewer up-front dollars is just "better" in the world view of most people.
I've had a conversation recently with the CIO of a major business who didn't quite understand why backups were worthwhile. He said something along the lines of "how does this help the business sell more widgets?".
I see the same thing, but often much worse, in big government or big bureaucracies. Project management is complex, so to simplify things, they just ignore the rest of the business or potential future requirements like they don't even exist. In the past, I've tried to point out that, say, with an additional 10% spend on one project they could halve the cost of a dozen future projects, but that's basically crazy talk to a project manager that has to minimize the cost of this project, right now. I've given up trying, and I bet a lot of other people have too.
The public has shown repeatedly that it will value cost above quality
Right, that is why Apple laptop sales have tanked in the downturn. Oh, wait.
Macs have a 10% market share. I'm not sure that really supports the suggestion that people value quality over cost, with 9/10 people voting against "quality".
Either that, or people don't think Macs are quality.
The big problem is that it's exactly the opposite of the way ATM networks are managed. They can't seem to wrap their heads around the way it's done on an IP network.
I believe we need to change the "They don't make em' like they used to" mentality.
They cut corners and cheapened out on stuff in the past too. It's just, none of those survived. So sure we can disassemble radio transmitters made in the 40's, see the craftsmanship that went into each one, and sigh that our equipment isn't made nearly as well. But there were a heck of a lot of transmitters and things made in the 40's that simply didn't survive because they were cheap junk.
It's not helpful to go to the upper management and say "they don't make them like this anymore." That's not something which is actionable. What can change decisions is "Look at this thing which has been making money for this company for 70 years. Now look at the others which all died in the 60's. It is worth it to make 'em like this."
The ______ Agenda
Really?
Perl - $Just @when->$you ${thought} s/yn/tax/ &couldn\'t %get $worse;
Oh noes, I'm feeding the trolls again.
But whatever... it doesn't matter. Because at the end of the day, the techie nerds will continue to have no respect for management... and then they'll wonder why they're treated with no respect in return.
So you think the techies that have taken the time to explain all the reasons *why* something needs to be done are stupid.
But you can sit back and say 'loose 8% from your budget - go do it'. No reasons, no explanations just a demand. (Brillant!).
I'm guessing your also the same arsehole that screams at the 'stupid' techies for not being able to restore that sales contract from two months ago that you accidentally deleted - Forgetting about that replacement broken tape drive you refused to pay for last quarter.
As a manager you have got to be the conduit between the workers and the directors. Here's a tip, how about try talking to your techies. No seriously, talk to them. Show them your budget, show them your overheads. Ask them to provide assistance in setting the priorities instead of telling them to get stuffed.
You may end up *earning* some respect from the people who are actually keeping your company running and who don't play musical employers when things start getting to hard.
Another phrase I've heard thrown around is "we don't have enough time to do it right, but time to do it over again" there was a time I thought it was funny, until it became too true.
So you think the techies that have taken the time to explain all the reasons *why* something needs to be done are stupid.
What percentage of techies take the time to explain the reasons why something needs to be done, in a way that a non-techie or less-techie can understand? Usually a non-techie with more bacon on the line than the techie themselves?
In my experience, without guidance or training, the percentage is way below 10%.
I can understand the troll GP's position - though it really serves no purpose to call people who are doing their best to deliver a quality project "stupid".
In my experience the techies that "won't listen" or "won't explain" aren't being stupid, they are usually just being myopic failing to understand that they have to make their case a little more solidly than "because we have to!" or "because I say so!". Anything that potentially costs extra time or money in a business project needs to be justified with a business case. A good tech manager knows what to listen for with her techies, and how to coax the right info out of them, but there needs to be some push from the techies themselves.
Techies: your manager simply can't take your word for granted until you make a solid business case for what you want to do. After a while, you may build up trust that just your saying so is enough, but you'll have to put some work in the first few times.
Techies: your manager simply can't take your word for granted until you make a solid business case for what you want to do. After a while, you may build up trust that just your saying so is enough, but you'll have to put some work in the first few times.
Remember, the manager doesn't understand everything (if he did, he'd be a techie rather than you) but you should be communicating with him to help him understand the key details. On the other hand, it's also important for you to understand the fundamentals of the constraints he's operating under too.
Think of it like an extra set of optimization parameters to work with on the overall problem; solving just the technical side is leaving the whole undone and any engineer worth his salt will appreciate the importance of doing things within budget. If you insist on not taking into account the business side, then you're going to have to provide a range of options with very careful descriptions so that those who do know the business side can see what the drawbacks of a particular decision are.
"Little does he know, but there is no 'I' in 'Idiot'!"
I'm not so sure that attempting to measure quality is necessarily the way forward when having the discussion with that CIO. Perhaps an alternative approach is to talk more about the value of the data which is being backed up. It can be quite illustrative to go through the data that is being backed up and work out the cost that went into generating it: "That took those staff 3 person-weeks to generate, their average salary is X" etc. It's fairly easy to get a notional cost of most data. Then you can look at the length of time that it would took to re-generate that data.
The answer to the "how does this help the business sell more widgets?" actually becomes fairly clear: 'N Employees of the company spent X hours and approximately $y generating this data, it is necessary to have for the company operate effectively, so when something happens to the data storage those N employees will have to spend X hours regenerating it, rather than working on their primary task of selling widgets. I cannot tell you exactly when the data storage will go bad, but I can tell you that given a mean time between failure of Z, it is likely to happen within the next T years.... Oh and there this data here, which we wouldn't have any practical way of generating. If that were to be lost, the impact on the business would be.. I"
and so on and so-forth.
Yes it sounds like a painful process, but the clarity introduced can be useful for all concerned.
Best thing our new management has done was to open the budgets. We all got an email telling us the expenditure for the previous year and expected expenditure for this year - the bosses were going to ask the managers to get the techie types, like myself, to go over each item and see if they were neccesary... but before they could organise a meeting they were already inundated with "this contract expired two years ago!" and "we can totally negotiate a better price on this!" messages from the front lines.
A management that understand and engages with underlings will result in a mutually beneficial relationship - I get to do the most techie work I can do and get away with minimal paperwork, simply because everyone from the top to the bottom understands the process and none of us want to waste time with redundancy. Management that don't understand their serfs and refuse to engage will continually create counter-intuitive, expensive and demoralising practices that will eventually cause all the gives-a-shit workers to leave for a management that cares... leaving the pencil pushing jobsworths to help run the company into the ground. This is true of all professions, not just computing - it's just more acutely observed in computing due to techies generally not being trustful of a (frequently closed) hierarchy.
Moderation Total: -1 Troll, +3 Goat
Declared (as opposed to observed) customer satisfaction is unreliable.
Except that it's relative to other satisfaction measurements using the same techniques. If more people say they are happy with a product than another, it does not matter if they are lying as long as they are lying in the same ratio.
I would also observe that humans are VERY willing to provide negative feedback for any imperfection they perceive. Proof, Internet.
Also, newer products / recent purchases tend to bring in better reviews. People are more careful with their new toy, and the device hasn't had time to break.
These surveys usually take into account long term ownership.
On the purely hardware side of things, I haven't noticed Apple being any better than other 1st or 2nd-tier vendors, nor good DYI.
I have. I had one hard drive die a few years back, Applecare shipped me a new one since I said I could install it myself. I've had no problems replacing video cards and other components in a desktop just as I would any other PC, in fact much easier since Apple designs desktops cases so well. Apple systems are just as good for DYI replacement of common components, it's really only the motherboard you can't mess with.
Again this is anecdotal, but I have two Apple laptops - one from 2001, it still works just fine and is usable for browsing, document editing, and as a VNC terminal into other stuff. The other is from around four years ago, again it is working fine. This is totally unlike any experience I have ever had with PC laptops.
"There is more worth loving than we have strength to love." - Brian Jay Stanley