A Possible Cause of AT&T's Wireless Clog — Configuration Errors
AT&T customers (iPhone users notably among them) have seen some wireless congestion in recent months; Brough Turner thinks the trouble might be self-inflicted. According to Turner, the poor throughput and connection errors can be chalked up to "configuration errors specifically, congestion collapse induced by misconfigured buffers in their mobile core network." His explanation makes an interesting read.
Wouldn't be the first time, except maybe for AT&T.
This is not really news at all. They spend little to nothing to keep their network up to the devices they have on it. This misconfiguration of buffers (if that is really a cause at all) is probably because they might not hire people with any knowledge of what they are doing to keep costs low.
Anything can be found funny, from a certain point of view.
His explanation makes an interesting read.
I'd like to think that's a given, considering it's a news story. At any rate, from TFA:
The bottleneck link is the over-the-air link, i.e. the connection from radio access network or UTRAN to the Mobile Statation (MS) in the above diagram, therefore the critical buffers are those at the UTRAN. In practice the UTRAN includes both the basestations (called Node-Bs) and the Radio Network Controllers (RNCs) which coordinate handovers between basestations (among other things). Because of hand-overs, the amount of data buffered at the Node-B is relatively small. It's the buffers at the RNC that must be large enough to deal with the delay variations in the radio network and yet small enough to induce packet loss when the network gets congested.
I am not a network engineer, but how exactly could 8 second ping time be not noticed by the AT&T engineers who set up, configured, and monitored their OTA link? I would think that we're not talking about some dude's set of bridged dd-wrt linksys routers, but some serious heavy-duty RF equipment. I'm thinking on the order of several zeros...
You see, most blokes, you know, will be buffering at ten. You're on ten here, all the way up, all the way up, all the way up, you're on ten on your buffer. Where can you go from there? Where?
I don't know.
Nowhere. Exactly. What we do is, if we need that extra push over the cliff, you know what we do?
Put it up to eleven.
Eleven. Exactly. One more buffered.
I find it just as problematic that applications software on Windows Mobile and other similar mobile OSes do not handle large network delays gracefully.
There is often very little feedback to the user of the software that actual progress is being made in attempt to communicate over the network. Sure, we can use the fuzzy "bars" indicator on the device to help diagnose what may be the cause of our trouble, but that doesn't indicate actual network conditions due to capacity. We also have animated indicators that web browsers and other applications use, but these still don't indicate any kind of actual success to communicate. In web browsers we get text alluding the DNS lookup, and connection attempt, but when you combine 'Connecting to...' with a simple spinning indicator or progress bar, that often doesn't convey that the message reached any destination or how long until you can expect any response from your local network based on its operating conditions.
The writers of the software may not fully understand the implications of being on a network with high packet loss or long round trip times. So they timeout or have errors that could be resolved by more delay or retry. In a mobile OS we should probably take this into account at the OS level, and opt out of this behavior only when the programmer or user specifies (if that's exposed).
it doesn't help that the safari client that the iphone uses will double load a page. Even if the user closes safari for a couple minutes, when reopening the browser the current page will reload. lose lose for everyone.
Whoever modded that down is a humorless dick.
One CPU cycle wasted on digital restrictions management is ONE TOO MANY.
How can you expect to have zero packet loss on a wireless network? That's just stupid.
Please fix this ATT!
(I'm not holding my breath... )
When did you last meet an unfunny penis?
Every time I deal with AT&T I am amazed that anything works at all over there. My phone almost always shows five bars at home, yet frequently calls don't cause the phone to ring - they go to voicemail after pretending to ring. The jaded amongst us could suspect a deliberate misconfiguration of phones and signal strength monitoring. Similarly, it would not surprise me if AT&T data networks weren't about as reliable as the signal strength indicator on my phone. The recent alleged blurb from an Apple "genius" in NYC that 1/3 of all iPhone calls get dropped seems to point in that direction.
That a cell-phone won't work everywhere and perfectly every time is a given. However, wouldn't it be nice if the companies that stood behind these networks would actually be held accountable for some of the advertising statements they make? What it comes down to is that we're dealing with an oligopolistic market, where only a few carriers can achieve the scale and the coverage to satisfy most mobile customers most of the time. On the flipside, that also means that said carriers can be truly dismal when it comes to customer service, back-end efficiency, etc. since consumers don't have many choices. Considering the ongoing consolidation in the industry, the only way out seems to be a trust-busting activity on the part of the DoJ to regulate the industry.
Not sure that is the better alternative... nor what the best structure for a regulated market would be.
The public has shown repeatedly that it will value cost above quality
Right, that is why Apple laptop sales have tanked in the downturn. Oh, wait.
Why should AT&T attempt to improve the quality of its network if people are a) willing to pay for what they currently have, and b) won't pay for any attempts AT&T will make to improve quality.
Because those people if they dislike the network enough, will leave eventually. That is the motivation to improve on what they have now, never mind they want to stop the customers bitching who are losing them new customers right now. They have plenty of reasons, they even have plenty of money from the influx of iPhone people. There's more than enough motivation, it's more a question of execution now.
People will pay for quality. For some the cost is financial. For others, the "cost" is that they will not buy an iPhone while the AT&T network has issues.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
I worked for AT&T in several parts of the country on their core networks, and in the early 2000's they had misconfigured all of their Solaris boxes and I worked with the infrastructure group to implement a startup script on Solaris to tune all the ndd settings for performance. The problem with Solaris is that by default all the TCP, UDP, Ethernet, etc settings are set for a Desktop workstation, not a server. Most system admins know to tune these settings, otherwise in a lot of cases a multi-CPU box will perform as slow as a 1 CPU box. Anyway, at specific companies I worked with (AT&T Broadband / Worldnet in St. Charles, MO was one big one), all the servers were configured without the proper settings for a server, so we had all kinds of issues as a result, a big one is that the tcp accept queue is not set high enough and so connections to daemons will drop after a low number of connections, making it appear that the box can't handle the connections...., As a result, they had spent millions on numerous servers (in one situation they had over twenty 12-cpu servers just for smtp...
These changes seem small, however, changing "ndd" kernel parameters on a Solaris box is not a single task, it is an infrastructure-wide task, and therefore requires the coordination of dozens of different groups, it really took a long long time to get this script implemented. It was called "S99nddfix" and it had all the ndd tunable parameters in it. Later when I worked at a different AT&T group in a different state, I noticed my script had been implemented on all the Solaris servers in the 200+ server environment.
quality, and fashion, one not having to be the other...
comment first, facts later. http://chem.tufts.edu/AnswersInScience/RelativityofWrong.htm
This is the problem. Thanks to the competitive barriers (such as the inability to move phones between all but two of the top four networks, and none of the top 3) moving can take a long time (2 year contract must expire) before someone can move networks unless they want to pay a large fee.
And then, you probably lose your phone. So even if you like it, you have to buyer either a different phone from the new provider, or the same one in their version. Both will cost you even more money, unless you're willing to be stuck on another 2 year contract.
The US system is very well setup, as far as carrier lock in goes.
It's rather amazing how many people go to AT&T for the iPhone. I think they said about 1/3 of their iPhone customers are coming from other networks. I wonder how many more people would get iPhones if it wasn't for their current contract? That's a big reason for many people I've talked to. The rest who want an iPhone are in the "I'd love it but I'm not touching AT&T again" camp.
Comment forecast: Bits of genius surrounded by a sea of mediocrity.
Brough
That's why to the greatest degree possible, libraries, programs, and algorithms should be auto-tuning. You can provide all the knobs you want, but people won't actually touch them. They'll choose which library, application, or operating system they use based on the default settings, so you'd better damn well make sure the default settings are good --- or even better, that you don't need settings at all.
quality, and fashion, one not having to be the other...
Indeed, and that is why many companies built atop the foundations of showy fashion are gone now. Fashion is transient and fickle. Apple however delivers a quality product that delivers new customers through loyalty and word of mouth. If this were not so Apple would not be a tenth of what it is now.
It doesn't hurt that it is fashionable, too. But that is not why I and so many other people buy Apple products.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
This is the problem. Thanks to the competitive barriers (such as the inability to move phones between all but two of the top four networks, and none of the top 3) moving can take a long time (2 year contract must expire) before someone can move networks unless they want to pay a large fee.
You say that like it does not matter because the period of switchover is two years. But it matters a great deal still, because many people will still leave then (or if they are mad enough pay the fee). A company like AT&T must be forward thinking in what problems now do for subscribers in future years, never mind the day when T-Mobile is also carrying iPhones (we have seen Verizon will not happen).
It's rather amazing how many people go to AT&T for the iPhone. I think they said about 1/3 of their iPhone customers are coming from other networks. I wonder how many more people would get iPhones if it wasn't for their current contract?
Probably quite a few, as I said that was a factor too. Which is why the two year thing doesn't really matter for motivation because they are losing new customers in addition to the ones potentially lost in two years.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Right, that is why Apple laptop sales have tanked in the downturn. Oh, wait.
I think that will change once Windows 7 is mainstream. Everyone hated Vista. Now it seems like everyone loves 7 and Snow Leopard only got a "meh" response from reviewers (not because Snow Leopard is bad it just doesn't have anything revolutionary, the fact that 7 runs at a decent speed is considered to be "revolutionary" in the PC world). There are two people who use Macs, people who have grown up using Macs and people who prefer Macs. When faced with Vista, a lot of people started to realize they prefer Macs.
Because those people if they dislike the network enough, will leave eventually
"Eventually" isn't very soon when you have a 2 year contract with early termination fees that are through the roof.
That is the motivation to improve on what they have now, never mind they want to stop the customers bitching who are losing them new customers right now
All 4 major carriers suck though. Lets see here, AT&T has network issues and isn't cheap, T-Mobile might have great customer service, good phones but it has a pathetic amount of 3G coverage compared to the others. Verizon might have a great network, but it isn't exactly cheap and a lot of their phones (at least used to) suck terribly with many features being stripped out of them. Sprint might be cheap but their coverage isn't great.
And none of them have a phone with as many apps as the iPhone, yes, Android and WebOS are great, but they still don't have the amount of apps as the iPhone nor as much support from companies such as game developers and the like. And don't get me started on Windows mobile.....
For others, the "cost" is that they will not buy an iPhone while the AT&T network has issues.
Taxation is legalized theft, no more, no less.
So in this case, zero packet loss is a setup for disaster instead of a desirable quality.
The trouble is that it's not an intuitive solution to a problem, the introduction of occasional packet loss. It's usually something to avoid.
The determined Real Programmer can write Fortran programs in any language.
Zero packet loss may sound impressive to a telephone guy, but it causes TCP congestion collapse and thus doesn't work for the mobile Internet!
I was in the standardisation group that specified the RLC/MAC layer (ETSI SMG2, later called 3GPP TSG GERAN) and our priorities were not the behaviour of TCP. We were designing the radio layer to provide a bearer service for the higher layer protocols, at that time they were X25, IP (UDP and TCP). The "problem" we were trying to solve was the tendancy of the radio layer to fade, have multipath and generally lose packets. The RLC layer was designed to deliver error-free packets, in sequence over the radio layer. Generally that is exactly what it does, and does well. If it didn't then tehre would be no mobile internet.
What we did find to be a significant performance problem was the asymetric channel. The uplink is usually the root of the TCP performance issues, UDP works much better. When the discrepancy is higher than 10, the downlink is ten times faster than the uplink, then the TCP Acks don't arrive in time and it stalls. Sadly a faster uplink is difficult and expensive to provide.
I think that will change once Windows 7 is mainstream.
I don't think people care that much one way or the other. In fact studies have shown previous Windows releases increased Mac sales, and this will too - if you have to refresh a whole system, if you have to learn a new UI - why not a Mac?
I just can't see how Windows7 will have any impact at all in slowing down the Mac train.
All 4 major carriers suck though.
From experience with them all I totally agree, which is why I am not as much bothered by some people with the iPhone being AT&T only.
Verizon might actually improve if they don't tamper with the Droid much. ...And who is going to look at a few "geek" articles about the iPhone and decide not to get it?
It's not that at all. It's having a friend who complains about dropped calls all the time, or if he tries to the use the network and it's failing a lot. This kind of damage is all done at the word of mouth level.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
When did you last meet an unfunny penis?
Probably when he met the guy that modded that comment down.
When did you last meet an unfunny penis?
Just now, when I looked down my pants!
CAPTCHA: horrify
Is to let AT&T engage in all the packet shaping and other fun stuff they want. Down with net neutrality? Wait? What's that? You think they'd manage to screw up their network even more then and probably fuck over other people in the process? Say it ain't so!
Roman Polanski.
"A Possible Cause of AT&T's Wireless Clog — Configuration Errors"
Oh wait, this is slashdot, we test on the live system.
Your hair look like poop, Bob! - Wanker.
TCP measures round trip time, and doesn't need packet loss to tell it that the round trip time is long. The retransmit interval will go up appropriately. TCP will behave reasonably with a long round trip time. If you're trying to do a bulk transfer, there's nothing wrong with this. The problem comes when short messages and bulk transfers are sharing the same channel. The short messages can spend too much time in the queue.
The solution is reordering the packets, not dropping them. That's what "fair queuing" is about. It may be worthwhile to implement fairness at the port-pair level, rather than the IP address level, at entry to the air link. Then low-traffic connections will get through faster.
"Quality of service" can help, but it's not a panacea. The network layer can't tell which of the TCP connections on port 80 is highly interactive and which is a bulk download, other than by traffic volume.
(I used to do this stuff.)
His explanation makes an interesting read.
I highly doubt that.
The public has shown repeatedly that it will value cost above quality
Right, that is why Apple laptop sales have tanked in the downturn. Oh, wait.
Macs have a 10% market share. I'm not sure that really supports the suggestion that people value quality over cost, with 9/10 people voting against "quality".
Either that, or people don't think Macs are quality.
5 bars might mean a strong signal received on both ends or just at the phone. You can also have a very strong signal along with horrible multipath interference that effectively kills data transmission.
When I still used SBC (which as far as I know is owned by AT&T) it wasn't uncommon for our service to drop. The yearly total for the downtime could easily be measured in days, and was part of the reason they got ditched. I found something interesting, though.
Each time I had to contact a service representative from the company, they'd give me the same old song and dance. Not only was their network fine, I was fine. We seriously had no clue why I couldn't access anything, so finally one day I did some snooping of my own. (This was back when I was really new to this stuff. Go ahead and laugh.) I learned to appreciate the tracert command afterward.
I was indeed online, and could, with the right information at my disposal, access some servers. What's more, my DNS wasn't down at all. What was happening was my requests for web pages were getting stuck in the net. I'd watch as my message in a bottle floated away, from Indiana to Illinois, to Minnesota, and finally through several provinces before getting stuck in a loop between two routers in western Canada and then expiring. This would go on for hours, with my Internet access crippled. Apparently somewhere along the line my normal route (which took me through Atlanta instead) was going down and the new one was taking me up north instead, to a place from which no traffic could escape. They apparently never rectified this.
Really?
Perl - $Just @when->$you ${thought} s/yn/tax/ &couldn\'t %get $worse;
As I recall, the story went: Mandelbrot was a mathematician at IBM lab. The engineers were attempting high speed data networking, but were encountering data/signal loss due to some noise. So like good engineers, they made things more robust, better isolation, grounds, shielding, etc. but the darn noise was still there.. They could not get rid of it. Determined to find the cause, they went to Mandelbrot with the request to analyze the noise, to determine its cause, in order to eliminate it.
Mandelbrot examined the data and found that there were periods of clear signal interrupted by noise. He examined the noise and found that within it were periods of clear signal, interrupted by noise and so on. Hmmm... He astutely determined that "shit happens" and what was needed was a redundant protocol, not better shielding. The noise you see, was inherent in a damped and driven system.
It was from this that he began his explorations of fractals and chaos theory, and we got robust network protocols.
apple was on the death bed, until the ipod got fashionable
Two things wrong here:
1) Apple had already seen a comeback from the iMacs at this point. Jobs had been back for a few years and the company was growing steadily. They were far away from "deathbed" status at this time.
2) The iPod didn't grow because it was fashionable. I bought one of the first ones because it was literally an order of magnitude faster to load songs onto than any other player at the time - and the interface was far better. The transfer speeds evened out but I never saw another MP3 player I liked as much for just everyday use. The iPod exploded in popularity mostly because of functionality, not fashion - the first of them were too bulky to really be considered fashionable.
but then this also depends heavily on where in the world one lives. USA seems basically saturated with apple (and thanks to the majority of the tech blogs being run out of USA, that gives a "interesting" slant on things), while europe seems to be mostly wintel
Europe is more wintel, but Apple marketshare grows there as well. I just was in Budapest an an Apple store there was doing very well... marketshare is a tricky thing because it's masked by business PC purchases, so you don't know what real-world marketshare is with people that live with computers, not the computers that are handed down to them from on high.
There is however the "new market", that is unix geeks that basically use osx for its bsd kernel and shell.
There's nothing new at all about this market. This core group of users is how Apple actually grew OS X from the beginning, because these people were the technical friends people looked to for computer advice, and they told friends and family "buy a mac". In part this was selfish because they knew they would have to help the users far less, but it's also in the end better for users that need help less often.
I would say that if HP or dell really took a good look at linux, or even one of the BSD's, they could match apple on quality.
HP used to have the potential to do that but I have a lot of doubts at this point in time, mostly because HP has shredded itself. Linux does need a champion to really tighten up the whole thing and then it has a lot of potential, but I'm not sure I see the company around right now that can do that.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
My phone almost always shows five bars at home, yet frequently calls don't cause the phone to ring - they go to voicemail after pretending to ring. The jaded amongst us could suspect a deliberate misconfiguration of phones and signal strength monitoring.
Signal strength alone does not guarantee the ability to make/receive calls. Even if your mobile is registered in the network, making and receiving calls depends on the availability of various scarce resources, namely:
In case of "lots of missed calls" in a particular area (your home) one could assume that
Note: outgoing calls should have the same problems; if they "fail less" it could be because your operator has chosen to reserve a (possibly large?) percentage of the slots/lines for outgoing calls. (Which obviously reduces the chances of incoming calls even more)
"Some years ago, there was talk of building some huge fiber-optic ring around the Pacific, connecting a bunch of countries. The only telco in Australia at the time that could afford to buy into the project was Telstra. One of the VPs of Telstra was quoted as saying "we have sufficient bandwidth right now". Think about it: the VP of a telco couldn't quite understand the need to maintain exponential growth in bandwidth right when broadband was taking off. Thanks to morons like that overpaid suit, Australia has been bandwidth-starved for a decade, which is why you don't see that many truly "unlimited" plans or free WiFi access points like in other countries."
A good example of incompetence, a bad example however as far as a solution for your problem. A fiber ring would help little for communications confined within a continent.
It is all too true.
When I saw Cowboy Neal's.
"Because those people if they dislike the network enough, will leave eventually."
and go where? to one of the other baby bells who have used their monopoly power to let their networks rot?
The western consumer values one thing above all else; price.
I kind of think that's just human nature (and I've traveled the world quite extensively).
When I first got DSL in 1999, I paid for a premium service but performance was measurably half of what it should have been.
After a lot of back and forth (the ISP was my corporate ISP as well and I used that as leverage) it turned out they had a switch port with a duplex mismatch. And this was at an ISP that had been supplying Internet connectivity since the beginning, formed as a consortium with a university and other high tech busineses in the late 80s.
So fuckups can happen. And one can only guess they happen more and worse at behomoths like AT&T.
AT&T: Those damned iPhone users are going to kill us. Disable VoIP!
Time passes...
AT&T: Those damned iPhone users are still killing us. It is Net Neutrality's fault!
Time passes...
AT&T: Those damned iPhone users are still killing us! We need... wait, we have to configure what now? No, no, no, it's the government's fault!
Stop-Prism.org: Opt Out of Surveillance
Declared (as opposed to observed) customer satisfaction is unreliable.
Except that it's relative to other satisfaction measurements using the same techniques. If more people say they are happy with a product than another, it does not matter if they are lying as long as they are lying in the same ratio.
I would also observe that humans are VERY willing to provide negative feedback for any imperfection they perceive. Proof, Internet.
Also, newer products / recent purchases tend to bring in better reviews. People are more careful with their new toy, and the device hasn't had time to break.
These surveys usually take into account long term ownership.
On the purely hardware side of things, I haven't noticed Apple being any better than other 1st or 2nd-tier vendors, nor good DYI.
I have. I had one hard drive die a few years back, Applecare shipped me a new one since I said I could install it myself. I've had no problems replacing video cards and other components in a desktop just as I would any other PC, in fact much easier since Apple designs desktops cases so well. Apple systems are just as good for DYI replacement of common components, it's really only the motherboard you can't mess with.
Again this is anecdotal, but I have two Apple laptops - one from 2001, it still works just fine and is usable for browsing, document editing, and as a VNC terminal into other stuff. The other is from around four years ago, again it is working fine. This is totally unlike any experience I have ever had with PC laptops.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
As far as the termination fee goes, AT&T waived mine 14 months into the (non-iphone) contract because they were no longer able to provide reliable 3G data service at the location where I primarily use it. I assume the fact that I called several times a week for several weeks checking on my tickets made it clear that I wasn't going to "let it go".
The contract works both ways (in theory). While you're obligated to remain a customer for the duration, they're obligated to provide the services as described in the contract for the duration. (That's why you get a 30 day window to terminate the contract any time they make a change.) Hell, I wish it went a step further and made THEM liable for an early termination fee if their crappy service forces me to change carriers and buy a new phone. I think $500 would have covered the research that went into it, pro-rated value of my old phone, cost of my new phone, time spent moving my contacts, etc.
a. retransmits, packet collisions and improper setup/teardown times == more latency x more users == more waiting == slower network.
b. configure your freaking TCP settings under "network settings" for your type of network (wireless)...
Isn't that what they teach you in wireless internet 101 (you don't need an advance degree to figure out why AT&T is slow if you're an AT&T service tech).