Slashdot Mirror
Comcast Accused of Congestion By Choice
An anonymous reader writes "A kind soul known as Backdoor Santa has posted graphs purportedly showing traffic through TATA, one of Comcast's transit providers. The graphs of throughput for a day and month, respectively, show that Comcast chooses to run congested links rather than buy more capacity. Keeping their links full may ensure that content providers must pay to colocate within Comcast's network. The graphs also show a traffic ratio far from 1:1, which has implications for the validity of its arguments with Level (3) last month."
Ever wonder what Comcast's connections to the Internet look like? In the tradition of WikiLeaks, someone stumbled upon these graphs of their TATA links. For reference, TATA is the only other IP transit provider to Comcast after Level (3). Comcast is a customer of TATA and pays them to provide them with access to the Internet.
1 day graphs:
Image #1: http://img149.imageshack.us/img149/78/ntoday.gif
Image #1 (Alternate Site): http://www.glowfoto.com/viewimage.php?img=13-224638L&rand=6673&t=gif&m=12&y=2010&srv=img4
Image #2: http://img707.imageshack.us/img707/749/sqnday.gif
Image #2 (Alternate Site): http://www.glowfoto.com/static_image/13-205526L/4331/gif/12/2010/img6/glowfoto
Notice how those graphs flat-line at the top? That's because they're completely full for most of the day. If you were a Comcast customer attempting to stream Netflix via this connection, the movie would be completely unwatchable. This is how Comcast operates: They intentionally run their IP transit links so full that Content Providers have no other choice but to pay them (Comcast) for access. If you don't pay Comcast, your bits wont make it to their destination. Though they wont openly say that to anyone, the content providers who attempt to push bits towards their customers know it. Comcast customers however have no idea that they're being held hostage in order to extort money from content.
Another thing to notice is the ratio of inbound versus outbound. Since Comcast is primarily a broadband access network provider, they're going to have millions of eyeballs (users) downloading content. Comcast claims that a good network maintains a 1:1 with them, but that's simply not possible unless you had Comcast and another broadband access network talking to each other. In the attached graphs you can see the ratio is more along the lines of 5:1, which Comcast was complaining about with Level (3). The reality is that the ratio argument is bogus. Broadband access networks are naturally pull-heavy and it's being used as an excuse to call foul of Level (3) and other content heavy networks. But this shoulnd't surprise anyone, the ratio argument has been used for over a decade by many of the large telephone companies as an excuse to deny peering requests. Guess where most of Comcasts senior network executive people came from? Sprint and AT&T. Welcome to the new monopoly of the 21st century.
If you think the above graph is just a bad day or maybe a one off? Let us look at a 30 day graph...
Image #3: http://img823.imageshack.us/img823/8917/ntomonth.gif
Image #3 (Alternate Site): http://www.glowfoto.com/static_image/13-205958L/4767/gif/12/2010/img6/glowfoto
Comcast needs to be truthful with its customers, regulators and the public in general. The Level (3) incident only highlights the fact that Comcast is pinching content and backbone providers to force them to pay for uncongested access to Comcast customers. Otherwise, there's no way to send traffic to Comcast customers via the other paths on the Internet without hitting congested links.
Remember that this is not TATA's fault, Comcast is a CUSTOMER of TATA. TATA cannot force Comcast to upgrade its links, Comcast elects to simply not purchase enough capacity and lets them run full. When Comcast demanded that Level (3) pay them, the only choice Level (3) had was to give in or have its traffic (such as Netflix) routed via the congested TATA links. If Level (3) didn't agree to pay, that means Netflix and large portions of the Internet
Am utterly shocked that anybody could be so cruel as to suspect a poor innocent cable company of trying to protect their cash-cow video delivery business by deliberately sucking at being an ISP(harder than they do simply by nature, that is) and using their oligopolistic incumbent position to shake down nimbler and more responsive competitors.
Anyone who is offended at the behavior of these ISPs could join http://www.stopthecap.com/ It may be futile, but at least it's better than whining.
Please someone tell me that Verizon is better, because I really want to switch to FIOS when it's available.
Is their company run by an evil troll who punishes all those who implement innovation and progress?
Yes, and MBA's don't appreciate being called names.
Blank until
Seems like they are intentionally congesting their links to force content providers to pay them extra for prioritisation. Ground rules for net neutrality are needed.. badly.
This is _ONE_ ten gig link. Lets assume they have another 10 gig to level3.
His point is pretty clear: ten gig links are NOT THAT EXPENSIVE. We're not talking about a 100 million dollar expense here, we're talking probably an extra 200k per month per link.
They're intentionally bandwidth starving themselves. I can't see any other explanation, and Backdoor Santa is right.
.
The Comcast argument is that they have a peering agreement with L3 (and TATA too) but that is simply not the case. Both L3 and TATA are providers for Comcast.
TFP (The Fucking Post) points out that Comcast runs its terminations with TATA at full capacity for most of the day and concludes that they do so on purpose to force services like Netflix to co-locate with them (= $$$ for Comcast.)
So L3 says to Netflix.. "Hey.. you dont need to be a slave to the Comcast overlord" and Comcasts reponse is to re-brand its business relationship with L3 as a "Peering Agreement."
Many slashdotters bought this bullshit hook, line, and sinker on the last Comcast vs L3 article. They did so because they learned about peering relationships at some point in other slashdot stories and took their 1:1 free peering knowledge and incorrectly applied it to the L3 and Comcast relationship.
L3 is Comcast's internet provider. Comcast's claim is like you claiming that you can charge your ISP because more stuff comes downstream to your LAN than goes upstream from it.
"His name was James Damore."
we're talking probably an extra 200k per month per link.
ps. I'm rounding _way_ up.
.
While I am also (slightly) doubtful of the "drive service providers to comcast colo"(though Backdoor Santa probably knows more than I do, if he has access to these data, so I'm giving him the benefit of the doubt) there is one major reason to suspect Comcast of perfidity rather than merely penny-pinching:
Comcast is a cable company. Their pre-internet business was realtime video delivery. This remains one of their more lucrative segments. As such, they have a built in conflict of interest when it comes to providing high quality internet service. They sure don't mind if you pay them to get your email really fast, or play video games with low ping, or download "linux ISOs"; but if youtube+netflix means that you cut the cord on your cable video, that is Bad News from their perspective. Thus, anything they do that would impact the reasonable performance of streaming video, online video downloads/rentals, etc. should be viewed as malice first and incompetence second.
Arguably, no, they don't care.
Most monopolies don't. Even in areas where they have to compete against DSL, there's only a small segment of the population that can purchase service that rivals theirs in terms of advertised speed / service. And even then ... who are they competing against? Well ... the phone company, which has a stellar reputation when it comes to customer service ...
No, it's more like the construction company is taking forever to finish the road, and by happy chance they also operate the toll booth on the only alternate road.
Seven puppies were harmed during the making of this post.
Multi-national providers are likely to be running their graphs in UTC - reading the graph that way makes a lot more sense.
There are two ways to get content to Comcast subscribers -- peer with them directly, or send your traffic through some other network that's connected to them. The graphs in the article show the "other network" links, and show that they're almost always running at capacity. if a link is running full, you can't get your bits through and that means packet loss. Peering directly to Comcast gets you uncongested links, which means you won't be dropping packets and your services will run as-intended, but it also means paying Comcast for the priviledge.
Well, in the past, lots of people have pointed out that Comcast is essentially a monopoly in places, so, it's not like they're competing with anybody.
They simply have no incentive to spend money. They've got all of these customers now, and spending money on infrastructure isn't going to make them any more money, so why do it? Upgrading is just straight cost, and without a benefit to them, why do it?
The very cynical answer is that until they're more or less forced to upgrade, they have no incentive to. They make money by overselling a service -- the closer to maxed out the service is, the more money they make. They don't really care about you, they care about their profits -- they're not gonna spend profits just so some people have a faster connection.
And, they're not going to give up on the revenue of having people co-locate with them, so they're doubly uninterested in fixing their capacity issues.
Welcome to the "free" market, it isn't really about customer choice and value -- it's abut maximizing profits and giving you the least amount of service they can get away with. This is a perfectly logical situation when you look at it from their point of view.
Lost at C:>. Found at C.
Can't this backfire on Comcast? I mean, if a Comcast customer tried watching Netflix and they can't get a good connection because of congested links, the user isn't going to think "Netflix is crappy" they're going to complain aboyt how they've got such a crap connection through Comcast.
That's only meaningful if there are alternatives/competition in the area, and there might be an argument that Comcast wants to push it's own video streaming service (which wouldn't crap out).
They're double dipping -- they charge you to deliver the bandwidth to you, and they charge the content providers to co-lo with them so that their users have a faster service experience.
So, the gouge you for shoddy service, and they gouge the content providers extortion-style so their content arrives in a timely manner.
Lost at C:>. Found at C.
From what I've heard about their infrastructure staff at a few conferences, they seem somewhat competent as they've been into IPv6 and DNSSEC from an early stage (doesn't always mean anything though). It is 10Gbits which is impressive, but I can't believe thats their only link out. They have tens of millions using their internet service right? How can it only be 10Gbits?
For the record, I am a comcast customer now (for only 2 months now) and I do agree it sucks balls compared to the fiber to the house I had before. But I also deliberately chose to go with cable internet for the first time because I wanted my own real experience to back up my suspicions instead of just angry posts by random people on forums.
Sounds kinda like smacking yourself in the face with a frying pan to confirm it hurts. :-P
Lost at C:>. Found at C.
A 100 percent full pipe is an efficient use of their resource.
It also limits the ability of Comcast's customers to use the 6 Mbps downstream burst capacity that Comcast has advertised to them. When an oversold link flat-tops, it's been over-oversold. If Comcast is not capable of bursting at 6 Mbps for the majority of the day, it shouldn't even be advertising 6 Mbps, let alone "PowerBoost".
They're certainly not made up numbers. That said, transit costs vary greatly by location and business negotiations. Getting a 10 gig link out of 60 Hudson when you have presence there is totally different than getting fiber run out to some middle-of-nowhere location.
I'm assuming we're talking about the opex cost of 10 gigs worth of transit from a fairly central hub. Capex to provide infrastructure to back that cost is not included. If we take the premise that Comcast's internal network isn't congested and only its transit links (which the graphs suggest is the major bottleneck), then there probably isn't significant capex cost in bringing online another link.
Of course I'm making huge assumptions. I'm on slashdot. Duh.
.
Actually a 100% full pipe is barely useable. You need a little slack even at the best of times - 95% full is much better, because when it goes any higher you start getting serious problems with retransmissions and burst latency from even the slightest irregularity in flow. According to these accusations, that's what Comcast wants.
One way to make profit in business is to maximise the use of your resources - but another is to deliberatly restrict supply of your product, in order to maintain a high price. You may shift less volume, but you make more per unit.
"what kind of idiot would locate their service inside this boundary, effectively guaranteeing crappy service to everyone who isn't a Comcast customer?"
Both in and out. Large sites like netflix or youtube use a CDN - servers placed all over the world, because no one place could be optimal in providing service to all customers.
Not really. I used to work for an ISP for 7 years that dealt in both DSL and Cable (don't ask how). I know the technologies behind DSL and Cable modems and know that the design of DSL usually wins out in situations where lots of people are online in the same area. Most people don't understand this and only pay attention to the marketing and data rates. For many years I had either direct ethernet, high speed wireless link,DSL and fiber to the home. So I wanted to try cable out to see how it was because all the marketing and clueless people making claims can really confuse the issue. I'm just familiarizing myself with my industry so that I have first hand experience when I give others advice. There is nothing wrong or sadomasochistic about that.
AFAIK bittorrent has better-than-normal conjestion management, not "very crappy congestion management".
It uses either TCP (almost the definition of bog-standard) or uTP (see http://en.wikipedia.org/wiki/Micro_Transport_Protocol); designed for the express purpose to improve upon TCP traffic management.
Perhaps the uTP devs failed; but there's no evidence for that that I can see.
Let's put this to rest. You are a taxpayer in a state/city where there are 4 lanes of road going in and out of the heart of your city. During rush hour, congestion hits and it's go, stop, go stop. You and all the other citizens who pay taxes in the city/state complain that something must be done. 5 years later there are 6 lanes of traffic. Rush hour is still go, stop, go stop. See where I'm going here? This is a cyclical problem that has no solution.
Comcast is a business designed to make profit yes?
I see this argument way too much. It's very true, but it's entirely irrelevant. We don't CARE if they want to make profit. It's no excuse. It's like saying that it's run by a sociopath sadist that just wants to hurt you, so it's perfectly fine when he breaks your kneecaps. And that simile isn't that far off, "making profit" is at the expense of the customer.
What I care about is getting the damn thing I paid for.
And the false advertising, that bugs me too.
Arguably, no, they don't care.
Most monopolies don't. Even in areas where they have to compete against DSL, there's only a small segment of the population that can purchase service that rivals theirs in terms of advertised speed / service. And even then ... who are they competing against? Well ... the phone company, which has a stellar reputation when it comes to customer service ...
The phone company's 100-year reputation isn't always a reliable predictor: I recently had an excellent experience with the local phone company. My Comcast download speed, advertised as “up to” 12 million bits per second, was actually between 6 and 7. I had been waiting for DSL to be available for years, and when it finally was, I invited Fairpoint, the local telco, to install it on 30 days approval.
They sent me a DSL modem, which I hooked up, and then waited for the service to be switched on. To my surprise, they dispatched a technician. I walked him around the property, showing him where the wires were buried, and he then followed the pair of wires that connected me to the neighborhood fibre termination point, making sure I had a straight run. When he was done I had an excellent signal to noise ratio, and was able to actually get the advertised 15 million bits per second of download speed.
The technician told me that mine was the first 15 million bits per second installation he had done, so that might be why he went the extra mile (literally—the neighborhood fibre termination point is a mile away) to make sure I got good service. Nevertheless, it shows that when you get down to the level of individuals, the reputation of the organization doesn't tell you much.
No, you pay for a high speed connection to the INTERnet and they are providing a high speed connection to their own INTRAnet with a congested gateway to the internet. Then they make money on both ends by charging content providers to get onto the intranet providing their customers with the connection they already paid for in the first place. They get away with it because the consumers are ignorant, the politicians are crooked. The end result is the consumer will be charged more plus the carrier gets greater control of what is on their networks. This decreases competition further eroding what the consumer gets in the end. Eventually maybe it won't even be possible to discuss and make others aware of what is going on. How long until Slashdot for example is inaccessible through Comcast?
The free market retort doesn't work because cable companies have a government-granted monopoly on that technology. Even if someone wanted to, in most areas, they can't legally start a competing cable service, it has to use a different technology, so you're not going to have real like-for-like competition, you're not going to have DSL-for-DSL or cable vs. cable competition in the same area in most places. Fiber is faster, but you're also starting out the gate with a much more expensive system to lay.
First of all, the low points in the graph prove that users don’t “always” consume all of what is available. They only do when the demand is actually that high (wow, what a revelation).
Go look at that graph again (here it is) and instead of the flatline, imagine that curve extrapolated up to where it ought to be. Where does it peak, somewhere around 200%? So you could actually double that network’s capacity and still be thinking “oh my god they’re just using it all up!” No, that’s just the normal demand... triple the network’s capacity, and you’d likely find that you have excess capacity at all times.
So no, the users don’t just “ALWAYS consume what is available”. They consume all of what is available when what’s available is less than the normal demand ought to be.
You’re just stuck in the position of being so ridiculously over-sold that the demand is 2x what you can supply. And you really only have yourselves to blame for that.
Distributed Denial of APK: It takes 15 seconds to reply to him anonymously, but wastes tons of his time if we all do it.
Problem is the road is bumper-to-bumper for 18+ hours a day. Congestion is expected at rush hour, but if the road can't handle normal loads it's not performing to need and needs upgrading.
According to this article, Comcast's public image is about the same as Halliburton or ExxonMobil. They're one of the most despised companies in America and they really don't give a shit.
Check out my world simulator thingy.
Everyone has a choice in the US.
Only if you're going to include alternatives that actually aren't alternatives, or aren't even in the same class of product as broadband, like dialup and satellite.
Over the last decade I've watched as the tide has swung from the "I'll work hard to get what I want in life" attitude to the "Society owes me something" attitude.
Apart from being a very shitty strawman that equates to "Get off my lawn" or "Back in my day...", you forget to mention that business attitudes have also swung from the "Let's produce a good product and compete on the means of that product and our customer service" to "Fuck the customer, we need more money" attitude.