Slashdot Mirror
Comcast Accused of Congestion By Choice
An anonymous reader writes "A kind soul known as Backdoor Santa has posted graphs purportedly showing traffic through TATA, one of Comcast's transit providers. The graphs of throughput for a day and month, respectively, show that Comcast chooses to run congested links rather than buy more capacity. Keeping their links full may ensure that content providers must pay to colocate within Comcast's network. The graphs also show a traffic ratio far from 1:1, which has implications for the validity of its arguments with Level (3) last month."
Ever wonder what Comcast's connections to the Internet look like? In the tradition of WikiLeaks, someone stumbled upon these graphs of their TATA links. For reference, TATA is the only other IP transit provider to Comcast after Level (3). Comcast is a customer of TATA and pays them to provide them with access to the Internet.
1 day graphs:
Image #1: http://img149.imageshack.us/img149/78/ntoday.gif
Image #1 (Alternate Site): http://www.glowfoto.com/viewimage.php?img=13-224638L&rand=6673&t=gif&m=12&y=2010&srv=img4
Image #2: http://img707.imageshack.us/img707/749/sqnday.gif
Image #2 (Alternate Site): http://www.glowfoto.com/static_image/13-205526L/4331/gif/12/2010/img6/glowfoto
Notice how those graphs flat-line at the top? That's because they're completely full for most of the day. If you were a Comcast customer attempting to stream Netflix via this connection, the movie would be completely unwatchable. This is how Comcast operates: They intentionally run their IP transit links so full that Content Providers have no other choice but to pay them (Comcast) for access. If you don't pay Comcast, your bits wont make it to their destination. Though they wont openly say that to anyone, the content providers who attempt to push bits towards their customers know it. Comcast customers however have no idea that they're being held hostage in order to extort money from content.
Another thing to notice is the ratio of inbound versus outbound. Since Comcast is primarily a broadband access network provider, they're going to have millions of eyeballs (users) downloading content. Comcast claims that a good network maintains a 1:1 with them, but that's simply not possible unless you had Comcast and another broadband access network talking to each other. In the attached graphs you can see the ratio is more along the lines of 5:1, which Comcast was complaining about with Level (3). The reality is that the ratio argument is bogus. Broadband access networks are naturally pull-heavy and it's being used as an excuse to call foul of Level (3) and other content heavy networks. But this shoulnd't surprise anyone, the ratio argument has been used for over a decade by many of the large telephone companies as an excuse to deny peering requests. Guess where most of Comcasts senior network executive people came from? Sprint and AT&T. Welcome to the new monopoly of the 21st century.
If you think the above graph is just a bad day or maybe a one off? Let us look at a 30 day graph...
Image #3: http://img823.imageshack.us/img823/8917/ntomonth.gif
Image #3 (Alternate Site): http://www.glowfoto.com/static_image/13-205958L/4767/gif/12/2010/img6/glowfoto
Comcast needs to be truthful with its customers, regulators and the public in general. The Level (3) incident only highlights the fact that Comcast is pinching content and backbone providers to force them to pay for uncongested access to Comcast customers. Otherwise, there's no way to send traffic to Comcast customers via the other paths on the Internet without hitting congested links.
Remember that this is not TATA's fault, Comcast is a CUSTOMER of TATA. TATA cannot force Comcast to upgrade its links, Comcast elects to simply not purchase enough capacity and lets them run full. When Comcast demanded that Level (3) pay them, the only choice Level (3) had was to give in or have its traffic (such as Netflix) routed via the congested TATA links. If Level (3) didn't agree to pay, that means Netflix and large portions of the Internet
Am utterly shocked that anybody could be so cruel as to suspect a poor innocent cable company of trying to protect their cash-cow video delivery business by deliberately sucking at being an ISP(harder than they do simply by nature, that is) and using their oligopolistic incumbent position to shake down nimbler and more responsive competitors.
Comcast needs to be stopped before NBC goes cable only and maybe even comcast only in area with more then one cable system.
I don't want to lose CSN CHICAGO on Dish / Directv / WOW cable / RCN cable and ATT uverse
Anyone who is offended at the behavior of these ISPs could join http://www.stopthecap.com/ It may be futile, but at least it's better than whining.
Please someone tell me that Verizon is better, because I really want to switch to FIOS when it's available.
Is their company run by an evil troll who punishes all those who implement innovation and progress?
Yes, and MBA's don't appreciate being called names.
Blank until
Seems like they are intentionally congesting their links to force content providers to pay them extra for prioritisation. Ground rules for net neutrality are needed.. badly.
This is _ONE_ ten gig link. Lets assume they have another 10 gig to level3.
His point is pretty clear: ten gig links are NOT THAT EXPENSIVE. We're not talking about a 100 million dollar expense here, we're talking probably an extra 200k per month per link.
They're intentionally bandwidth starving themselves. I can't see any other explanation, and Backdoor Santa is right.
.
The Comcast argument is that they have a peering agreement with L3 (and TATA too) but that is simply not the case. Both L3 and TATA are providers for Comcast.
TFP (The Fucking Post) points out that Comcast runs its terminations with TATA at full capacity for most of the day and concludes that they do so on purpose to force services like Netflix to co-locate with them (= $$$ for Comcast.)
So L3 says to Netflix.. "Hey.. you dont need to be a slave to the Comcast overlord" and Comcasts reponse is to re-brand its business relationship with L3 as a "Peering Agreement."
Many slashdotters bought this bullshit hook, line, and sinker on the last Comcast vs L3 article. They did so because they learned about peering relationships at some point in other slashdot stories and took their 1:1 free peering knowledge and incorrectly applied it to the L3 and Comcast relationship.
L3 is Comcast's internet provider. Comcast's claim is like you claiming that you can charge your ISP because more stuff comes downstream to your LAN than goes upstream from it.
"His name was James Damore."
we're talking probably an extra 200k per month per link.
ps. I'm rounding _way_ up.
.
While I am also (slightly) doubtful of the "drive service providers to comcast colo"(though Backdoor Santa probably knows more than I do, if he has access to these data, so I'm giving him the benefit of the doubt) there is one major reason to suspect Comcast of perfidity rather than merely penny-pinching:
Comcast is a cable company. Their pre-internet business was realtime video delivery. This remains one of their more lucrative segments. As such, they have a built in conflict of interest when it comes to providing high quality internet service. They sure don't mind if you pay them to get your email really fast, or play video games with low ping, or download "linux ISOs"; but if youtube+netflix means that you cut the cord on your cable video, that is Bad News from their perspective. Thus, anything they do that would impact the reasonable performance of streaming video, online video downloads/rentals, etc. should be viewed as malice first and incompetence second.
What makes Backdoor Santa think this is done to drive service providers to Comcast? Occam's razor has a much simpler explanation: Comcast doesn't want to spend more money upgrading their capacity.
That makes sense.
Their users don't necessarily have it that bad anyways. So Comcasts' links are just congested -- that means their users have some packet loss. Unfortunately, those graphs don't show discard rates, so it's not really known from those graphs just how badly things are congested.
It could be a lot of customer high-speed transfers bursting to use the full link. As the link becomes more congested, transfer conditions will become slightly less conducive, and those "high speed transfers" will back off in transfer rate, as more customers get fair treatment.
The TCP protocol is designed to deal with it by backing off trasmit speeds. So everyone's download/upload speed drops; your transfers still complete, unless drop rates get too high.
Most traditional applications deal with it just fine; VoIP and streaming video do not fare so well.
And protocols with very crappy congestion management, such as BitTorrent, are capable of causing some serious problems in such scenarios, without the ISP taking additional measures.
However, I don't see there being an issue with Comcast allowing 100% of their links to be utilized when there is demand for it. And there is no immediate requirement to upgrade if any applicable SLAs are being met, and congestion is within reasonable limits based on packet drop rates and latency.
Arguably, no, they don't care.
Most monopolies don't. Even in areas where they have to compete against DSL, there's only a small segment of the population that can purchase service that rivals theirs in terms of advertised speed / service. And even then ... who are they competing against? Well ... the phone company, which has a stellar reputation when it comes to customer service ...
No, it's more like the construction company is taking forever to finish the road, and by happy chance they also operate the toll booth on the only alternate road.
Seven puppies were harmed during the making of this post.
Multi-national providers are likely to be running their graphs in UTC - reading the graph that way makes a lot more sense.
There are two ways to get content to Comcast subscribers -- peer with them directly, or send your traffic through some other network that's connected to them. The graphs in the article show the "other network" links, and show that they're almost always running at capacity. if a link is running full, you can't get your bits through and that means packet loss. Peering directly to Comcast gets you uncongested links, which means you won't be dropping packets and your services will run as-intended, but it also means paying Comcast for the priviledge.
Well, in the past, lots of people have pointed out that Comcast is essentially a monopoly in places, so, it's not like they're competing with anybody.
They simply have no incentive to spend money. They've got all of these customers now, and spending money on infrastructure isn't going to make them any more money, so why do it? Upgrading is just straight cost, and without a benefit to them, why do it?
The very cynical answer is that until they're more or less forced to upgrade, they have no incentive to. They make money by overselling a service -- the closer to maxed out the service is, the more money they make. They don't really care about you, they care about their profits -- they're not gonna spend profits just so some people have a faster connection.
And, they're not going to give up on the revenue of having people co-locate with them, so they're doubly uninterested in fixing their capacity issues.
Welcome to the "free" market, it isn't really about customer choice and value -- it's abut maximizing profits and giving you the least amount of service they can get away with. This is a perfectly logical situation when you look at it from their point of view.
Lost at C:>. Found at C.
Can't this backfire on Comcast? I mean, if a Comcast customer tried watching Netflix and they can't get a good connection because of congested links, the user isn't going to think "Netflix is crappy" they're going to complain aboyt how they've got such a crap connection through Comcast.
That's only meaningful if there are alternatives/competition in the area, and there might be an argument that Comcast wants to push it's own video streaming service (which wouldn't crap out).
They're double dipping -- they charge you to deliver the bandwidth to you, and they charge the content providers to co-lo with them so that their users have a faster service experience.
So, the gouge you for shoddy service, and they gouge the content providers extortion-style so their content arrives in a timely manner.
Lost at C:>. Found at C.
From what I've heard about their infrastructure staff at a few conferences, they seem somewhat competent as they've been into IPv6 and DNSSEC from an early stage (doesn't always mean anything though). It is 10Gbits which is impressive, but I can't believe thats their only link out. They have tens of millions using their internet service right? How can it only be 10Gbits?
For the record, I am a comcast customer now (for only 2 months now) and I do agree it sucks balls compared to the fiber to the house I had before. But I also deliberately chose to go with cable internet for the first time because I wanted my own real experience to back up my suspicions instead of just angry posts by random people on forums.
Sounds kinda like smacking yourself in the face with a frying pan to confirm it hurts. :-P
Lost at C:>. Found at C.
I agree. Monopolies have rights too...
Wait, what?
Some people have no choice but Comcast, however others have competition, allowing change to other providers.
Just because you live in the city, doesnt mean everyone does. (General statement, not directed soley at you. But it could be).
It's simple economics. If the cost of problems with annoyed customers remains below the cost of upgrading the system, then they won't upgrade.
Comcast makes no money on the traffic that traverses their network, and has nothing to gain by upgrading except their customers' good will. Since every third post here begins with "Comcrap" or ends with "sucks", I don't think they're too worried about their quality of service image.
Here's the deal breaker for Santa's conspiracy theory: what kind of idiot would locate their service inside this boundary, effectively guaranteeing crappy service to everyone who isn't a Comcast customer? There would have to be a compelling reason that this would improve Comcast's networking business for this theory to be true, and I see nothing compelling about this.
There's a perfectly simple explanation, backed by a mountain of evidence: Comcast is cheap.
John
MRTG is good enough for carrier-class deployment, and has been since 1993 or so. I relied on it to keep track of various metrics for our ISP business back then, everything from link utilization to Usenet volume to disk free space to modem utilization. (Side note, that #3 modem that had WAY more connection attempts than all the rest? That's a defective mode, boss, let's move the blade to the end of the pool until we get a replacement, ok? Just a thought...)
But damn, our first T-1 never looked like that. I would have been into the second T-1 in a day.
As a general note, if you're seeing 80% utilization on a regular basis for an Ethernet link, you're seeing true packet loss beyond what anyone should have to suffer. If comcast wants to argue this, show us your other links, and then call the gang at TATA and have them DO SOMETHING :) Riiiight.
Just a quick look at TATA, and they seem to be a services provider, not an in-the-business ISP or peering provider. Comcast chose them for cost. But I'm also thinking that TATA can't be Comcast's primary provider. We might be seeing some bad design here, and a change to some hop costs or metrics could improve this a lot. But I haven't seen their other charts, which would settle this.
deleting the extra space after periods so i can stay relevant, yeah.
A 100 percent full pipe is an efficient use of their resource.
It also limits the ability of Comcast's customers to use the 6 Mbps downstream burst capacity that Comcast has advertised to them. When an oversold link flat-tops, it's been over-oversold. If Comcast is not capable of bursting at 6 Mbps for the majority of the day, it shouldn't even be advertising 6 Mbps, let alone "PowerBoost".
They're certainly not made up numbers. That said, transit costs vary greatly by location and business negotiations. Getting a 10 gig link out of 60 Hudson when you have presence there is totally different than getting fiber run out to some middle-of-nowhere location.
I'm assuming we're talking about the opex cost of 10 gigs worth of transit from a fairly central hub. Capex to provide infrastructure to back that cost is not included. If we take the premise that Comcast's internal network isn't congested and only its transit links (which the graphs suggest is the major bottleneck), then there probably isn't significant capex cost in bringing online another link.
Of course I'm making huge assumptions. I'm on slashdot. Duh.
.
Actually a 100% full pipe is barely useable. You need a little slack even at the best of times - 95% full is much better, because when it goes any higher you start getting serious problems with retransmissions and burst latency from even the slightest irregularity in flow. According to these accusations, that's what Comcast wants.
One way to make profit in business is to maximise the use of your resources - but another is to deliberatly restrict supply of your product, in order to maintain a high price. You may shift less volume, but you make more per unit.
"what kind of idiot would locate their service inside this boundary, effectively guaranteeing crappy service to everyone who isn't a Comcast customer?"
Both in and out. Large sites like netflix or youtube use a CDN - servers placed all over the world, because no one place could be optimal in providing service to all customers.
Not really. I used to work for an ISP for 7 years that dealt in both DSL and Cable (don't ask how). I know the technologies behind DSL and Cable modems and know that the design of DSL usually wins out in situations where lots of people are online in the same area. Most people don't understand this and only pay attention to the marketing and data rates. For many years I had either direct ethernet, high speed wireless link,DSL and fiber to the home. So I wanted to try cable out to see how it was because all the marketing and clueless people making claims can really confuse the issue. I'm just familiarizing myself with my industry so that I have first hand experience when I give others advice. There is nothing wrong or sadomasochistic about that.
AFAIK bittorrent has better-than-normal conjestion management, not "very crappy congestion management".
It uses either TCP (almost the definition of bog-standard) or uTP (see http://en.wikipedia.org/wiki/Micro_Transport_Protocol); designed for the express purpose to improve upon TCP traffic management.
Perhaps the uTP devs failed; but there's no evidence for that that I can see.
Comcast doesn't give SLA's, even on business class. There SLA is "We're awesome, we'll keep everything running, trust us. And if it goes down, we'll get it back up soon. We promise"
Except if you're paying for Service Y, Company X owes you Service Y. That you're incapable of seeing this is mind boggling.
Let's put this to rest. You are a taxpayer in a state/city where there are 4 lanes of road going in and out of the heart of your city. During rush hour, congestion hits and it's go, stop, go stop. You and all the other citizens who pay taxes in the city/state complain that something must be done. 5 years later there are 6 lanes of traffic. Rush hour is still go, stop, go stop. See where I'm going here? This is a cyclical problem that has no solution.
Being as state and municipal governments are the ones giving Comcast the franchise agreements (read: protected monopoly status) in exchange for various benefits, it is dramatically unlikely they would do anything with regard to opening competition or regulating their practices.
upon the advice of my lawyer, i have no sig at this time
Comcast is a business designed to make profit yes?
I see this argument way too much. It's very true, but it's entirely irrelevant. We don't CARE if they want to make profit. It's no excuse. It's like saying that it's run by a sociopath sadist that just wants to hurt you, so it's perfectly fine when he breaks your kneecaps. And that simile isn't that far off, "making profit" is at the expense of the customer.
What I care about is getting the damn thing I paid for.
And the false advertising, that bugs me too.
Users ALWAYS will consume what is available
I guess that explains the low points in the graph....
It's the company that destroyed TechTV. Nuff said.
SJW: Someone who has run out of real oppression, and has to fake it.
Arguably, no, they don't care.
Most monopolies don't. Even in areas where they have to compete against DSL, there's only a small segment of the population that can purchase service that rivals theirs in terms of advertised speed / service. And even then ... who are they competing against? Well ... the phone company, which has a stellar reputation when it comes to customer service ...
The phone company's 100-year reputation isn't always a reliable predictor: I recently had an excellent experience with the local phone company. My Comcast download speed, advertised as “up to” 12 million bits per second, was actually between 6 and 7. I had been waiting for DSL to be available for years, and when it finally was, I invited Fairpoint, the local telco, to install it on 30 days approval.
They sent me a DSL modem, which I hooked up, and then waited for the service to be switched on. To my surprise, they dispatched a technician. I walked him around the property, showing him where the wires were buried, and he then followed the pair of wires that connected me to the neighborhood fibre termination point, making sure I had a straight run. When he was done I had an excellent signal to noise ratio, and was able to actually get the advertised 15 million bits per second of download speed.
The technician told me that mine was the first 15 million bits per second installation he had done, so that might be why he went the extra mile (literally—the neighborhood fibre termination point is a mile away) to make sure I got good service. Nevertheless, it shows that when you get down to the level of individuals, the reputation of the organization doesn't tell you much.
No, in most areas there are these things called Franchise Agreements. Telecommunications in the US is mostly a command economy.
Besides, since when was it a businesses right to advertise one thing, take money for it and not provide it? If a restaurant fails to provide what you order do you go start a new restaurant? What a bunch of NeoCon bs!
Holy fucking shit, thats a car analogy if I ever seen one.
Stop Computers/Cars Analogies on S
No, you pay for a high speed connection to the INTERnet and they are providing a high speed connection to their own INTRAnet with a congested gateway to the internet. Then they make money on both ends by charging content providers to get onto the intranet providing their customers with the connection they already paid for in the first place. They get away with it because the consumers are ignorant, the politicians are crooked. The end result is the consumer will be charged more plus the carrier gets greater control of what is on their networks. This decreases competition further eroding what the consumer gets in the end. Eventually maybe it won't even be possible to discuss and make others aware of what is going on. How long until Slashdot for example is inaccessible through Comcast?
The free market retort doesn't work because cable companies have a government-granted monopoly on that technology. Even if someone wanted to, in most areas, they can't legally start a competing cable service, it has to use a different technology, so you're not going to have real like-for-like competition, you're not going to have DSL-for-DSL or cable vs. cable competition in the same area in most places. Fiber is faster, but you're also starting out the gate with a much more expensive system to lay.
First of all, the low points in the graph prove that users don’t “always” consume all of what is available. They only do when the demand is actually that high (wow, what a revelation).
Go look at that graph again (here it is) and instead of the flatline, imagine that curve extrapolated up to where it ought to be. Where does it peak, somewhere around 200%? So you could actually double that network’s capacity and still be thinking “oh my god they’re just using it all up!” No, that’s just the normal demand... triple the network’s capacity, and you’d likely find that you have excess capacity at all times.
So no, the users don’t just “ALWAYS consume what is available”. They consume all of what is available when what’s available is less than the normal demand ought to be.
You’re just stuck in the position of being so ridiculously over-sold that the demand is 2x what you can supply. And you really only have yourselves to blame for that.
Distributed Denial of APK: It takes 15 seconds to reply to him anonymously, but wastes tons of his time if we all do it.
Problem is the road is bumper-to-bumper for 18+ hours a day. Congestion is expected at rush hour, but if the road can't handle normal loads it's not performing to need and needs upgrading.
According to this article, Comcast's public image is about the same as Halliburton or ExxonMobil. They're one of the most despised companies in America and they really don't give a shit.
Check out my world simulator thingy.
Comcast claims that a good network maintains a 1:1 with them, but that's simply not possible unless you had Comcast and another broadband access network talking to each other. In the attached graphs you can see the ratio is more along the lines of 5:1, which Comcast was complaining about with Level (3). The reality is that the ratio argument is bogus.
Comcast claims that free peering arrangements should have close to 1:1 ratio. And if you don't maintain that ratio, then you should pay for transit, just like Comcast is doing with TATA. So this is entirely consistent with what Comcast is saying and if anything supports their argument, not undercut it like Backdoor Santa is claiming. His argument about saturating transit to force other to peer with Comcast is valid though.
I personally think it is garbage to apply Tier-1 peering standards to (what should be) a CDN-ISP peering arrangement as they are completely different situations with different economics. It would save Comcast money and improve their customer experience if they were to enter into a free peering relationship with L3-the-CDN, because without the peering agreement Comcast-the-ISP would have to pay someone transit to access this data.
But to play devils advocate, here is the issue from another perspective. Comcast actually has a it's own Tier 1 network now, in addition to the last-mile network that we normally associate them with. This includes many business customers who are content providers not consumers. Comcast is using this CDN issue to force L3-the-Tier 1 to start treating them like a Tier 1. L3 wants a traditional CDN-ISP peering agreement where they to route their CDN data over their backbone network and connect with Comcast at the closest possible location to the customer, with only data intended for those customers. Comcast wants a Tier 1 peering agreement where their networks connect at a smaller number of points, and more data would be routed over their Tier 1 network, and then they balance the ratio by sending more traffic L3's way for free. Think about it; if Comcast was paying any other Tier 1 for transit, then L3-the-Tier 1 would have no issue peering with them. So if Comcast builds out their own Tier 1, why shouldn't L3 treat them the same?
L3 is trying to use it's backbone capability as an advantage to support it's CDN, and Comcast is trying to leverage it's position as a huge ISP to push it's Tier 1 network. In the end, because there is a lot of competition between CDNs and not so much between ISPs, Comcast has the upper hand.
Yes, as a Comcast customer, I can confirm this.
90% of the time we watch Netflix between 12 PM and 12 AM, we end up getting very low quality video (1 'bar'). I pay for the mid-tier service (not the "super-fast" gaming connection, but not basic either). Nothing else at the time is using the internet, other than maybe a bit of Stumbleupon on 1 PC.
Heck, even trying to stream Youtube at 720p requires a several minute wait as the video buffers.
It is unacceptable. If I actually thought it would accomplish something, I would complain. Sadly, Comcast is still the only reasonable option I have. DSL service equivalent to what I'm getting now would cost about $20 more a month than what I'm paying Comcast.
Just to test...I tried Netflix here at work...less than two blocks from where I live. They pay for an internet connection from Level 3. Full HD quality. HD Youtube streams instantly.
I think it would be a delicious irony if as result of the scrutiny Comcast is receiving due to their proposed acquisition of NBC regulators not only to denied the acquisition but further split the company in half. One half would be Comcast cable and the other half would be Xfinity broadband. Comcast cable would be forced to lease the last mile lines to Xfinity as well as any other broadband provider that is interested. That would be justice and therefore it will never happen. We're just going to see a ban on charging for traffic that terminates in their network.
You could build your own infrastructure by using a collection of high power 802.11b/g modems to multiplex a small amount of bandwidth from every unsecured wireless connection in your neighborhood.
There's no reason one cannot provide for excessive users. However, it needs to be upfront and clearly seen what that means.
That said, bit torrent is not the bandwidth hog of today. It is everyday, common, and reasonable usage of services such as streaming video from the likes of Netflix, Hulu, etc.. If the network cannot support the everyday, common, and reasonable usage of the majority of its customers then they are not fulfilling their obligations.
Two of my imaginary friends reproduced once
Apples and Oranges. You oversell *individuals* 5-10x because the average, or even peak, aggregate use isn't the same as everybody maxing out their connections at once.
What Comcast is doing is overselling individuals at a rate so much higher that the *aggregate* use is oversold by a factor of 2x. That's an enormous problem.
Everyone has a choice in the US.
Only if you're going to include alternatives that actually aren't alternatives, or aren't even in the same class of product as broadband, like dialup and satellite.
Over the last decade I've watched as the tide has swung from the "I'll work hard to get what I want in life" attitude to the "Society owes me something" attitude.
Apart from being a very shitty strawman that equates to "Get off my lawn" or "Back in my day...", you forget to mention that business attitudes have also swung from the "Let's produce a good product and compete on the means of that product and our customer service" to "Fuck the customer, we need more money" attitude.