Internet Transit Provider Claims ISPs Deliberately Allow Port Congestion

An anonymous reader writes "Level 3, an internet transit provider, claimed in a recent blog post that six ISPs that it regularly does business with have refused to de-congest most of their interconnect ports. 'Congestion that is permanent, has been in place for well over a year and where our peer refuses to augment capacity.' Five of the six ISPs that Level 3 refers to are in the U.S., and one is in Europe. Not surprisingly, 'the companies with the congested peering interconnects also happen to rank dead last in customer satisfaction across all industries in the U.S. Not only dead last, but by a massive statistical margin of almost three standard deviations.' Ars Technica reports that ISPs have also demanded that transit providers like Level 3 pay for access to their networks in the same manner as fringe service providers like Netflix."

Comments

What Level 3 can do

[Z00L00K]

Is just to cut the connection to those ISPs and see how long they will be around.

Re:What Level 3 can do

[jythie]

That would be terribly amusing. I can just imagine what a 'dark day' would do to those ISPs, though I suspect Level 3 has contracts that prevent it, which is sad.

Re:What Level 3 can do

[bill_mcgonigle]

Is just to cut the connection to those ISPs and see how long they will be around.

But why are they peering with them if there are better routes available?

The incentive structure on all these things is wrong. One neat thing the bitcoin network does is to attach a fee to each transaction that occurs (which is due to be reduced to reasonable levels soon - pay attention...).

There's too much turmoil going on in Internet routing with regard to pricing now. Some sort of BGP extension that includes transit cost has to come along to make it all automatic and lowest-cost. It's really not much different than how power producers will bring capacity online when the market demands or when they have excess capacity they need to get rid of. The dam near me has a realtime market price terminal they watch to see when to open the gates, but Internet providers would just automate the whole thing, and then the transit pricing wars would shake out. I wouldn't mind seeing it extended to the last mile either, though with monopoly protection in place there would need to be some very reasonable connection fee floor and controls on fees, since competition can't impose those controls. But one of the ways we encourage lowest-cost is with efficient protocols and there's very little incentive to demand that from the end user right now.

Re:What Level 3 can do

[Bengie]

But why are they peering with them if there are better routes available?

ISPs hold a monopoly on their customers, there is no other way to get to their network.

Re:What Level 3 can do

[pla]

ISPs hold a monopoly on their customers, there is no other way to get to their network.

ISPs only have customers because of peerage agreements that let their customers get to the rest of the internet.

Any two-bit ISP (and in this context, that includes even the likes of Comcast) that thinks they can twist L3's arm has one hell of a nasty surprise waiting for them when their current contracts expire. This doesn't work quite the same as not getting to see this week's episode of Glee because of a pissing contest between cable companies and content providers - A week where Comcast customers can't get to Por... er... Google, means a week where Comcast loses half its customer base.

Re:What Level 3 can do

[Lumpy]

90% of comcast customers are held hostage, they CANT GO ANYWHERE ELSE for internet.
This is what happens when you have a government sponsored and allowed monopoly.

Re:What Level 3 can do

[Archangel Michael]

ISPs think that they offer "high speed", and they do, but only on the "last mile". They think that last mile is the only thing that counts as a metric.

What good is a phone call if you are unable to speak?

Congestion has always been the bigger underlying issue, because Comcast customers are clueless about what "high speed" means. The best thing Level 3 (and other peering companies like them) could be doing is running national TV advertizements announcing (without naming) that "slow internet" may not be a last mile problem. I could design a 30 second commercial that describes the issue.

"Yes, you do have high speed internet, however your ISP may not be able to deliver the promised speed".

And trust me, congested pipes are worse issue than appears on the surface. Once you hit that max, you start compounding the problem with duplicate (and beyond) packets needing to be resent because the first packet never go there. Once you get to that point, the ONLY solution is more and bigger pipes(series of tubes???) .

Re:What Level 3 can do

[Anonymous Coward]

This isn't about peering in the strict sense of the word. Level 3 is a transit provider, a so-called tier 1 network. Colloquially Level 3 is an internet backbone operator. The end-user ISPs don't have global networks and need other networks to pass data through in order to reach the entire internet. These ISPs buy transit from backbone providers, i.e. their peering is not settlement free. For an end-user ISP to unclog their "connection to the internet", they have to buy more bandwidth from the backbone provider, so Level 3's "public service announcement" is a little self-serving. But of course they're right: What's happening is that last-mile providers are selling internet bandwidth that they in turn haven't bought from their upstream providers. If a last-mile ISP has oversold their access to the rest of the internet to the point where there's significant congestion, then it is that last-mile provider's job to buy a bigger pipe (or, if they feel ambitious, build a fast global network and become their own tier 1 network operator).

There is no need for a fragile real-time settlement protocol. The cost of bandwidth is determined by peak loads, so nothing needs to happen on a timescale smaller than typical demand cycles. Demand for network bandwidth can be planned, and links are not built at a touch of a button. There's nothing to be gained from opening this up to speculators.

Re:What Level 3 can do

[sjames]

It is a bit karmic. I'm not claiming that L3 are just a great bunch of guys fighting the man or anything.

However, L3 is a Tier 1. They have many massive datacenters for colo as well as an international network. The only thing they don't have is last mile networking.

A fair bit of the internet would either go away or get much more expensive to reach if L3 cut off peering.

Re:Ads

[Anonymous Coward]

Sounds like its an ad targeted just at you. There isn't anyone else here gets their jollies from that sort of thing.

Besides..who see's ads on this site anyways?

NO COMPETITION -- NO INCENTIVE

[AnontheDestroyer]

Why would the ISPs do this? They have no incentive. The correlation with customer service is a good thing to note, too. The American people are being bent over a barrel on this.

cry of a dying business

[alen]

Level 3 has been awesome, but the ISP's now have national footprints and transit prices are dropping fast. Verizon and AT&T have it because of the wireless business. Comcast will be a national network once they buy time warner.

  figure that as transit prices drop L3 and Cogent have to carry more and more data to pay the bills but they don't have enough money left to upgrade the links and want the ISP's to upgrade them. maybe the ISP's are being dicks and trying to run L3 and Cogent out of business by denying them more links and then taking their business like what happened with netflix

at this level there is no more need of transit providers as more and more content sellers will connect directly to the ISP's. so L3 and Cogent are crying network neutrality to save their business

Biased

[Charliemopps]

So, just to make it clear up front... Level3 is a Tier1 provider. Basically they are an ISP to the consumer ISPs. This is how your ISP connects to the internet (that's an over simplification but it will serve our purposes here) There are other Tier1 networks that the ISPs can connect to.

The point to these peering agreements is that Netflix and other companies like them make agreements with the ISPs to elevate congestion. So Google (random example) goes to AT&T (another random example) and says "We want to sign a peering agreement with you. We'd like to use Level3 for 2 years." and if AT&T agrees they do the same. So now both companies know there will be 10gig of traffic coming at them for 2 years and they can sign a reciprocal contract with Level3. This is standard

What Netflix does that angers pretty much every ISP on the planet is that they refuse to negotiate on these agreements at all. Instead they show up and say "We're going to use Level3, and we're not going to tell you for how long. Here's a long list of conditions that may cause us to switch without notice" so the ISP is stuck not knowing how long of a contract to sign and end up losing a lot of money when Netflix switches without notice.

The Tier1 providers love this. There's nothing better if you're a network provider than a customer locked into a contract they can't get out of stuck paying for bandwidth they aren't using.

The ISPs in question are likely in negotiation with Level3 on contracts. Level3 has been using the Netflix situation to their advantage. I suspect that this blog post by their VP is just an attempt to push the issue and get them to sign deals more lucrative for level3.

Not saying the ISPs aren't sucking. But this guys words need to be taken with a grain of salt. He's not out trying to help the consumer.

Three Weeks in ISP Hell

[fullback]

I just spent three weeks in the U.S.

The internet service was like being in a third-world country, but no one would believe it if you told them.

Dead Wrong

[Anonymous Coward]

Peering agreements are between two organizations, not between two organizations with a tier-1 between them. Netflix's peering agreement was not through level-1, it was direct between comcast and netflix. Tier-1 providers are the intermediary between non-peering entities, and tier-1 providers peer with those entities.

Re:Dead Wrong

[Anonymous Coward]

Exactly. @Charliemopps, that is not how the internet works.

As AC tries to explain:

Netflix _pays_ Level3 for internet access (Level3 is a tier1 so has connectivity to the whole internet). _Pays_ being the important word here
You _pay_ your ISP for internet, and they _pay_ a tier1 for access. From the money you pay. No reason to ask Netflix for money.

The actual situation is more difficult because ISPs and content providers also peer. That is, they connect to each other, and pay each other nothing for the privilege. This makes sense because both parties pay less to their tier1 or transit provider.

Re:USA=Third World Internet

[wiggles]

>providers are FORBIDDEN to upgrade any portion of their networks to IPv6 without NSA direct approval.

Source? The signal you're picking up through your tinfoil hat doesn't count.

Nationalize the Internet in America

[fallen1]

First, let me be clear -- I'm not a big fan of this idea, but after looking at the problem from multiple angles this idea keeps coming up as the best way to spur competition and end the debate on network neutrality.

A few steps to stop this greed from happening, hopefully:

a) A clear, concise Bill of Internet Rights.
-- This must be done in order to alleviate a lot of the crap going on now. There should be terms that explicitly disallow government agencies from piping internet traffic through their data centers for "analysis" of anything WITHOUT A NON-SECRET COURT ORDER. If it can't stand up the light of day, it doesn't fit with principles this country was founded upon and which hundreds of thousands of men and women have died to uphold. Stop being assholes and running roughshod over the Constitution.
-- This must be done to guarantee privacy. As much as can be, anyway.
-- This must be done to guarantee that all data is treated equally with the obvious need for quality assurance. No more congesting nodes, no more content owner also owns the delivery network so it can shutout competition, no more "you pay us, again, for the bandwidth that our customer who requested your info has already paid for."

b) Nationalize the Distribution Lines
-- All copper, fiber, interconnects, and so on are nationalized.
-- A plan is put into place to guarantee (almost) everyone in the United States good data speeds (10mb/s up and down - minimum) by adding more and more fiber. I say (almost) because there are some VERY remote places where people live and it will take time (plus more money) to reach them. If 90% of the population can be served, including rural areas, then that would be great.
-- Everyone who wishes to be an ISP pays THE SAME per connection. Yes, that would mean someone in Small Town, Iowa costs the same as someone in New York to connect to the internet. The overhead of the ISP will determine what $XX.xx is added to the government mandated $YY.yy and here's the rub - customer service comes back to the forefront and actually means something because the Public will know what the $YY.yy is. Competition to gain and keep customers based on price alone should vanish as value-added services and real customer service return to the industry.
-- We have a glut of workers needing work. Teach them to lay fiber optic cable and copper if needed. Put them to work moving the United States back to the top of the chart in broadband/internet access. In this day and age it is a necessity, not a luxury. Easily as ubiquitous today as the telephone and mail were in their days.

I'm probably missing a massive hole in my theory (greed being at the top of that list), but if this was done it would foster intense competition and new ideas as one would not be held back by thinking "I will get blocked out by Company A because they have a grip on distribution of a similar idea." Freedom from the so-called content creators of today locking down sections of the internet or using their power to double and triple-dip the pockets of consumers and competitors.

Re:L3, Cogent and Others Crying Wolf

[Bengie]

Let me get this strait, Level 3 sells bandwidth of the highest quality to a company, routes it around the world with nearly no congestion, then offers to peer with an ISP for free, meaning that ISP doesn't need to route the data around the world themselves, the ISP refuses because they think the data should be not only handed to them on a silver platter, but also get paid; and you think Level 3 has a "horrible" business model?

Keep Pressing The Public Comment Channels

[Bob9113]

Yesterday, the net neutrality petition passed the halfway mark, with 18 days left to go. The FCC request for comments is still live and looking for your feedback, and Mozilla has an alternative in the offing.

Keep the pressure on, keep posting these things on your social networks, keep telling your friends. The only thing less effective than telling the government what we want is not telling them what we want. It is a double edged sword; either they do as we say, or we get one more bit of documentation to support reforming the government.

Re:Details please

[fullback]

Her are those details:

I live in Japan and not in the middle of a city. I'm in a suburban area and have lived in what would be considered almost rural at one time.

I've had fiber for over 13 years. The only time I've ever had a service interruption was during the major earthquake 3 years ago. Internet came back up within an hour, though. That was the only time I've had a power outage too, in over 22 years of living in Japan.

I was in several major cities the southeast U.S. - Orlando, Atlanta, Nashville, etc. I needed to ftp data to my servers and it was almost impossible. So slow that I had to give up and wait until I returned home. I was at a friend's house and he lost internet service at least once per week. He had to scream answers for 5 minutes through his phone to a silly automated service before he could talk to a person. He said he has to do it weekly...

I couldn't get any emails from Asia through his connection. They're all blocked, and those were from the major ISP's in Japan - NTT and Softbank. Blocked! Every foreign web server was like pulling teeth.

Public WiFi was, well, pathetic.

It has nothing to do with size of the country. I had faster, more reliable service in the middle of nowhere surrounded by rice paddies in Japan 10 years ago than exists in U.S. urban areas now.

The reason is that there is competition in Japan. No area franchises. It's a free country.