Internet Partitioning - Cogent vs Level 3?

slashmicah asks: "Internet partitioning and Tier 1 ISPs are something most people don't know much about (myself included). Today, however, some Slashdot readers might have run into some issues involving these two topics. Cogent Communications and Level 3, both Tier 1 ISPs, are apparently having some 'undisclosed' disagreements, causing an Internet partition by turning-off or deactivating their peering point. Cogent Co. has released a statement explaining their side of the problem, however they have no mention of when the problem will be fixed, or when they will sort it out. This partitioning is a problem because any [single-homed] computers that are connected through Cogent Co, can not connect to [single-homed] computers connected through Level 3. Having spent all day sorting out this problem, I ask Slashdot: Isn't there a better way that the issue of peering can be handled/regulated? If not, does the future hold a scenario in which the Internet is split into several separate networks, only to be connected at the whims of large corporations?"

Comments

PITA but move along

[Halvard]

Okay, it's a pain in the ass for customers and others BUT they are businesses that negotiate peering, sometimes with monopoly money changing hands, sometimes without. It's a business dispute and it's not like Germany and France closing roads and making you drive through Belgium. They'll resolve it or lose business. Their's more of a back story we don't know.

At least it's not like UUNET more than one, some years ago, wanting to charge other Tier 1's per packet for transfer when peering while their traffic they wanted to pass for free. They were a big dog and were trying to make everyone pay. No one did and threatened to or did kill off traffic until UUNET got the sh*t together. But the did try to pull it off more than once.

Been dealing with this all day.

[QuantumRiff]

we got an email this afternoon from our provider, who let us know that cogent will be reachable by their second link, which is WilTel. However, the link is slower than the Level3 link. There will also be more traffic being routed through less points, meaning congestion. (and obvious lack of redundancy, if the WilTel connection has problems, no Level3) We have had users complaining about sites being unreachable at random times this afternoon. One of our providers very big customers is the OSU Open Source Lab, home of Drupal, mozilla download servers, master.Kernel.org servers, and many, many others. If your having problems reaching these sites, that is probably why.

Re:Consider switching to someone less petulant

[urlgrey]

As a cogent customer, it's *really* not their fault, IMHO. I gather L3 pulled the same stunt with XO last week.

As to the notion by another poster of not expecting peering with someone bigger for free forever: 38.0.0.0/8 Class A is Cogent/PSI... how much bigger than being an entire Class A (and then some?!) does one have to be to be considered [ahem] "equal"?

It was a mutual arrangement: they both allowed transit for one another's packets... pretty fair given the size and stature of them both, I'd say.

Cogent's message (via NANOG)

[wayne]

I can't get to Cogent's website, but according to a NANOG post, this is whatit says:

Cogent Network Status/DNS Server Status Description:
Date: 10/05/2005

Level 3 has partitioned its part of the Internet from Cogent's part of the Internet by denying Level 3's customers access to Cogent's customers and denying Cogent's customers access to Level 3 customers. Level 3 terminated its peering with Cogent without cause (as permitted under its peering agreement with Cogent) even though both Cogent and Level 3 remained in full compliance with the previously existing interconnection agreement.

Cogent will offer any Level 3 customer, who is single homed to the Level 3 network on the date of this notice, one year of full Internet transit free of charge at the same bandwidth currently being supplied by Level 3. Cogent will provide this connectivity in over 1,000 locations throughout North America and Europe.

Cogent is committed to an open Internet. The existing interconnection facilities between Level 3 and Cogent remain intact. Cogent hopes that Level 3 will reactivate these connections, restoring a full level of service to their customers.

For more information about the sales offer, please contact the numbers listed below.
NORTH AMERICA: 1-877-875-4432
ANYWHERE ELSE IN EUROPE: +33 (0)6 1101-7382

Level 3's official statement

[The AtomicPunk]

"Recently, certain peers have been disconnected from their direct connection to the Level 3 IP network. Some disconnected peers may elect to block access to certain IP addresses as a result of the disconnection. If a peer elects not to restore connectivity to the Level 3 network through alternative means, customers seeking continued access to the Level 3 network should make alternate arrangements."

They're saying Cogent is intentionally not advertising routes to them via other providers, presumably because they're upset about not having a peering agreement in place. Anyone affected by this presumably needs to harass Cogent.

http://ws.arin.net/whois?queryinput=AS174

Some thoughts on this mess

[Anonymous Coward]

First, I think that Level-3 is within it's legal rights in terms of dealing with Cogent, but is probably in trouble with it's customers. I am a customer of Level-3 and of Cogent (in the same facility). When I buy IP transit from Level-3, I am not buying "part of the internet". This peering issue places 45+ Million IP addresses out of reach of the Level-3 network (and vice versa). Level-3 did not notify me that they were making this type of change. There is nothing on Level-3's website that even implies that everything is not hunky dory. If you buy a Level-3 line today, will they disclose to you that you are not connecting to the entire internet. I know I am being a little niave here, but not disclosing such a large change of policy is unconscionable.

Second, it is dishonest for Level-3 to blame Cogent for this exclusively. Level-3 had a peering arrangement with Cogent for a long time. If you look at Level-3's interconnection policy page:

    http://www.level3.com/1511.html

It still looks like Cogent and Level-3 could peer under these terms. It was Level-3 that pulled the plug, not Cogent.

What is really annoying is that this is only traffic from Level-3 to Cogent, not to other parts of the internet. Level-3 wants money for Cogent customers to connect to Level-3's network but does not understand that this is a two-way connection and that Cogent's customers and Level-3's customer both benefit from this equally.

Up until this point, I was very happy with Level-3. They run an excellent network and I pay top-dollar to be on it. This blatent disregard for the impact on their customers is a diservice to their customers, to their reputation, and only begs for regulation.

That's transit

[fuzzy12345]

Peering is when you agree to send traffic destined to network X directly to network X via a direct connection between you and X. If you're using X's network to send traffic to Y, that's transit, and X will naturally expect you to pay for the privilege.

Peering 101

[Anonymous Coward]

The short version goes something like this:

Provider A and Provider B peer, be it public or private, normally they do this in several places and alternate who pays for the circuit, etc. Now, under normal circumstances, they both push enough traffic from one to the other to justify this mutual payment plan. However, in some cases, you find that B is either intentionally dumping traffic into A thinking A won't notice, or A discovers that its sending so little traffic to B in comparison to the amount B is sending to A that its not worth the continued cost.

When the first sort of thing happens, it usually gets resolved -REALLY- quick, that sort of behavior is not tolerated and will result in B getting de-peered by A (and potentially others once the abusive behavior is discovered and known) exceptionally quick unless B can show that it wasn't done knowingly or intentionally.

When the second instance happens .. well .. you get what happened today (I'm making an educated guess here based on what I know of the two carriers involved). A decides that spending 30 grand a month for what is a very lopsided bandwidth agreement is no longer economically feasible or reasonable. They go to B and say 'look, we're not doing this anymore, we're basically paying a hell of a lot of money every month for you to send a ton of traffic to us, and we don't send much of anything to you. You can either pay for all (or some larger portion of) the circuits, pony up some $$ per megabit, or we'll just cut it off at the stub and be done.'

Based on Cogent's 'oh poor us' post from this morning, I'm leaning towards them having given L3 the finger when L3 said 'look, this isn't equitable, we're going to have to re-arrange the money'.

YMMV of course, but I'm betting I'm not terribly far off.

Re:Consider switching to someone less petulant

[BeBoxer]

As a cogent customer, it's *really* not their fault, IMHO. I gather L3 pulled the same stunt with XO last week.

Yeah, Level 3 is really out of line in my opinion. It's not that they shut down the peering link. That wouldn't be that big of a deal. The traffic would just flow through other providers on less efficient routes. It's not as though every single backbone carrier peers with every other. But I just checked my BGP sessions, and Level 3 is not advertising the Cogent route at all. And you know for a fact that Level 3 is receiving the Cogent route from many of it's other peers. But it appears that they are intentionally filtering out the Cogent route. Which is pretty much not playing by the rules. It's one thing to shut down a peering agreement. It's something else entirely to refuse to accept that route from any of your other peers.

Re:WRONG!!!

[zmq503o1]

I wish that was true, but I am both a Cogent customer (100meg Link) and a Level3 customer (Multiple T1's) and the worst is true. My Level3 connection can not reach my Cogent connection, and my Cogent connection can not reach my Level3 circuts. Level3 is no longer BGP peering with Cogent so all routes normally advertised to Level3 no longer exist (from the prospective of Level3 customers). And although the request packet from Cogent might make it to Level3's network via another provider (i.e. route around the problem) with no route back to Cogent's network (BGP ASN 174) there is no way to get the traffic back to the user on the Cogent network.

How to Complain

[randalny]

You can call:

720-888-2518 (Level3 Investor Relations)

and complain.

Or call 877-453-8353 (Main customer service number).

Re:Interesting scenario, though most likely untrue

[bernywork]

Anyone want to explain for those of us that don't get it?

What they are essentially saying is: "We haven't done anything. We haven't made any changes on our side" Level 3 have terminated their connection to Cogent "Without cause". Now, that's probably legal speak on Cogent's side for we haven't got the letter in triplicate yet telling us what the reason is. Or otherwise whoever put up the notification about it doesn't know.

Now, Cogent may have tried to change the peering arrangement, or Level 3 may have too, one side probably didn't agree, or otherwise an agreement ran out and the switch got flipped. This has happened previously with Cogent in their peering arrangement with AOL.

What Cogent are trying to do is get business from Level 3 customers because Level 3 stopped the connection. Cogent is offering them connections to the Cogent network (And subsequently Cogent's customers) for a year with no fee on the amount of data they put through. That connection itself they will obviously have to pay for, but the customer can connect into (presumably) the closest of any of 1000 points across North America and Europe.

Now some people are already connected to both Cogent and Level 3. These people won't have any problems as they will be able to go direct into either ISP. These people would probably have never have used the interconnect between Cogent and Level 3 either, unless one of their connections into either Cogent or Level 3 went down.

I understand this is still rather technical, for a simpler version, take a look through the document that I linked to.

Re:OK, WTF time here

[shadowmatter]

The Internet is a power law network, meaning there are some very well-connected routers out there that a lot of the end-to-end transfers through the Internet go through. These are typically the peering points, owned by Tier 1 ISPs. It's not inconceiveable that if two ISPs don't peer with each other anymore, at some level a partition is created.

When Paul Baran had the task of designing a network that could withstand a nuclear attack, he envisioned a "distributed network". By today's lingo, it's a mesh network where each router is connected to approximately the same number of other routers. But now that routing infrastructure is driven commercially, with tit-for-tat contracts between Tier 1 ISPs, we ended up with what he said was a "decentralized network" -- that is, power law. Not what Paul Baran had in mind. If the underlying topology were his distributed network, you wouldn't be reading this story.

You can read his paper here. The Internet could withstand one nuclear attack. Several well-placed nuclear attacks? That's debatable...

- shadowmatter

Re:Level3 has a reason to be scared of Cogent

[nodmc]

Clarification, Cogent Communications Group, Inc. which is the Cogent being discussed has only a ~$240 million market cap. The Cogent you refer to as a $2.1 billion market cap company is Cogent Inc which does biometrics, not telco services.

Partitioning occasionally happens

[scoove]

Level 3 is not advertising the Cogent route at all.

I'd bet L3's argument is that they will not provide transit across their AS to Cogent. It's a play that's been made several times before. The first time I know of it being done was in 1995 when Sean Doran pulled this at the CIX-W router, preferring to take commercial traffic via NSFNET or Sprint reseller service. Not only didn't it work, but it caused some immediate political backlash as Sean's action (presumably made without his boss's approval, who was the chairman of the CIX board and took some political grief for Sean's latest stunt) caused several state's to literally drop off the map.

If my memory's right, I think this got pulled again around 1998 timeframe on Exodus by someone like Genuity (I may be wrong about the culprit), only for the higher ups at the culprit to discover they couldn't see half of the world's worthwhile websites and search engines. Much of this was in the transit battle - e.g. if you had consumers, you felt your eyeballs were the value of the Internet and all other ISPs should pay you to get to your consumers, while if you were a content provider, you had the stuff all those consumers were paying their ISP to get to and someone had better pay you for that content.

What can you do about it? Let your ISP know you're not paying them for 80% of the Internet. When UUNET considered pulling this stunt around 1997, I worked for a small software shop that had a couple bonded UUNET T1's and we let them know we were going to drop them the moment they were only selling partial Internet. Then follow through if they do (UUNET backed off). Bilateral agreements are weird things in the world of settlement-free IP exchange, so unless you want a settlement-driven Internet (which will have unusual effects you might not want, like driving a per-packet pricing model), just expect this occasionally and drop those who don't play well with others. When L3 drops customer base, even the Denver boys will figure out their customers aren't happy.

*scoove*

Level 3 = Failing Business = Retribution

[Anonymous Coward]

I worked there and its a deathmarch shop for IT and network people; they treat their peopel like crap, and leaving there was the best thing I ever did.

The arrogance of Jim Crowe [workign on his 7th manion and 8th large layoff at Level 3] and Kevin O'Hara (President, CEO) is only matched by the jailbird Bernie Ebbers. They only reason they have yet to decalre bankruptcy and liquidate thier debt (and clear away their bad business model with a fresh debt-free start) is that all their Omaha cronies have tied up money in the company stock, which would be flushed.

That they would resort to stunts like this against companies that undermine their pricing is not surprising. Level 3 have amassed BILLIONS in debt that they cannot service at current pricing levels, while Cogent and other more nimble competitors can sustain operations and drain Level3 into bankruptcy. So Level 3 execs do what arrogant desperate people do: lash out.

Level 3 is playing the "Sampson" card - if they cant make people price it their way, they will take the internet down with them.

And they did this trying to kill XO and now Cogent. Watch for more until they finally admit their business model is a failed one, and they declare bankruptcy, wipe the debt, and then begin to price lower and rake in the profits that their debt service is now eating.

Re:Level 3 = Failing Business = Retribution

[EQ]

You may have a point here - remember that Level 3's main exec, Jim Crowe, was an exec under Ebbers at Worldcom. Plus I've heard from a one or two people where I work who worked there and they pretty much back up what you said: Level 3 treats the tech people liek pieces of facility, not people down there in Broomfield (I worked tech up in Boulder a while back at Adaptec, and saw them build those buildings up on the hill).

Putting on my investor's cap, and taking a quick look at financials, its obvious that Level 3's burn-rate on cash, and billions in debt is not looking good for them if they cannot start generating both higher margins and more revenues. Neither one individually will save them at this point. The debt service is eating what EBITDA revenue they have coming in faster than they can produce it. And with companies like Cogent undercutting them Level 3 is dying; it seems the only question now is how much interconnectivity they will destroy in fits of pique like this.

I think you may also be right on another point, after considering it and runnnign the numbers: if Level 3 were to reorganize in bankruptcy court, dump the current shareholders, turn the debt holders into stock holders to ditch the debt, then they would probably be very profitable at even lower pricing levels. After all, that is what a lot of their competition has done. If they do that, Level 3 will cut the throats of every company out there, and make a bundle doing it, free-market style. Pretty interesting scenario.

But first they have to drop the stockholders, and from your post, it sounds like cronyism is a big factor, so its only going to happen when there has been far too much damage to Level 3 as a company. Thats a shame, because looking at their web site, they have some good ideas, but the wrong time and place for them.

Thanks for the post AC (wow an AC that actually said something useful!)

Transit vs. nontransit service

[IGnatius T Foobar]

I can't believe nobody has yet bothered to explain the difference between transit and nontransit service.

When you buy Internet bandwidth from your ISP, you are getting transit service. This means that you can use the link to send traffic to that ISP and to other ISP's upstream from it.

Nontransit service means that the link is to be used exclusively for sending traffic to that one ISP.

All of the Tier 1 ISP's provide nontransit service to each other, because at tier 1 there is no such thing as "upstream." This is not people playing stupid, this is how it's done at the top. It's the reason why the major peering points exist.

Any ISP who wants to shut off a peering arrangement for stupid business-o-political purposes is creating a hole in its own connectivity, and therefore shooting itself in the foot, plain and simple.

Level 3/Cogent Deal

[Anonymous Coward]

I work for Time Warner Cable and we use Level 3. I can't talk about the details of what's going on but the notes on the ticket opened for this issue and the conference call/bridge have covered alot. Alot of people in high up positions are working for find us a work around until a solution can be found. Level 3 is still routing traffic TO Cogent but they are not routing it back at this time. I was working on this issue all night. My suprise to come home and find it on Slashdot. I sure hope they can come to some kind of agreement soon (contract was terminated at 5:30am 10/5/05) but from what's been said thus far it's not looking like it's going to be a quick fix.

Re:Interesting scenario, though most likely untrue

[thegameiam]

lack of resiliency in BGP routes has nothing to do with "level 1" peering points, whatever those are. The vast bulk of private interconnections are richer, more geographically diverse, and generally better managed than the legacy MAE peering points. However, there hasn't been the sort of "peering of last resort" available since well before those days either: the CIX was the peering point of last resort, and eventually both Sprint and UUNet withdrew from it, fundamentally changing the Internet topology from a star to a partial mesh.

Read about it here - warning: it's a 120-page pdf...

-David