Level 3 and Cogent Reach Agreement on Peering

Armour Hotdog writes "Level3 and Cogent have announced an agreement on a modified peering contract that provides for settlement-free peering subject to certain unspecified conditions. This is a welcome announcement considering the disruption caused earlier when Level3 depeered Cogent. After that earlier dispute, Level3 temporarily restored peering, but announced that they would once again depeer Cogent on November 9th, unless the parties could come to an agreement."

Comments

I thought title said

[Anonymous Coward]

Level 3 and Cogent Reach Agreement on Peeing

Re:I thought title said

[Anonymous Coward]

The parent might be a troll but I thought the same thing when I first read it.

Re:I thought title said

[Anonymous Coward]

Me, too. That's funny.

Hah

[matr0x_x]

November 9th?

*waits nerviously for the November 9th press release*

Re:Hah

[Armour Hotdog]

Oops - I guess I could have phrased that a little more clearly. The November 9th ultimatum came when Level 3 originally restored peering back on 10/07. Today's agreement supercedes that, so the danger of another depeering (between these two ISPs) has passed unless somebody (read: Cogent) violates the terms of the new agreement.

Re:Hah

[smcallah]

"somebody (read: Cogent)"

So Level(3) can't violate the agreement?

Re:Hah

[Anonymous Coward]

I think that statement was made as an educated guess based on Cogents priors. I.e., their troubles with various other tier1 providers that have casued depeering squabbles in the past.

Cogent is well known in certain circles for getting themselves in trouble.

Re:Hah

[Armour Hotdog]

Sure they can, but Cogent has a history of annoying peers due to traffic disparities (as I understand it, Cogent's customers tend heavily toward webhosts, with relatively few end users. This leads to much greater bandwidth utilization for traffic outbound from Cogent's network than for inbound traffic.) This was Level3's reason for depeering Cogent, and is likely to the the cause of any future disagreement between them, unless the composition of Cogent's customer base changes significantly.

How was this allowed to happen?

[Jeff DeMaagd]

I think the "depeering" probably shouldn't have happened, and should not have affected people that weren't involved in the dispute, i.e. the rest of us. Had this happened with any other utility, there would be investigations.

Re:How was this allowed to happen?

[Dharkfiber]

WARNING: Trollbait Investigations on what? This is an agreement between two private companies. The only investigation that happened with the NYC power outage was that state investigators went and bothered some poor saps in Edison Power and wondered why they were drawing power from another country. When they got the answer which was "Does everyone want to pay another $200 a year for more power?" they dropped it. This is nothing more than Cogent abusing a public aggreement and then pissing and moaning that they finally got cut off (because they are using someone else's backbone to transit traffic to another area of the world). Let face it ... noone needs to make any generalizations or comments on this sort of thing until they are part of it.

Re:How was this allowed to happen?

[boarder]

You think that because you are an uninformed generalist. You don't really know the facts behind the story, only the results; so you immediately come up with this "conclusion."

The fact is this: Cogent wasn't paying for the bandwidth they were using from Level3 and wasn't giving Level3 an equal amount of their own (equal sharing of bandwidth is what these peering agreements are all about, and Cogent was upside down by about 10:1), then they were openly courting Level3 customers by offering to charge 1/2 of what Level3 charges. They were getting something for free from Level3 due to a gentleman's agreement, and then they were turning around and selling that to Level3's other customers at a lower cost. Biting the hand that feeds you effectively breaks a gentleman's agreement.

I think it's funny you put depeering in quotes, because it wasn't much of a peering situation. Level3 didn't so much de-peer anyone as just scraped a leech off their system.

Internet Latency

[JhohannaVH]

Good! Everytime they do this, businesses are affected, on the backend if nothing else. It screws up B2B and EDI xactions like mad. If companies can prove that it affected their bottom line, what recourse do they have???

Re:Internet Latency

[elm3r]

Not only are businesses affected, but educational institutions are as well. At my workplace, we conduct many internet-based courses that completely *halted* when Level3 depeered Cogent. The professors of those courses had to quickly make major changes to their deadlines and course structure, to work around the outtage. That is the first time such actions have ever had to be taken here (including changes made due to hurricanes).

Re:Internet Latency

[Anonymous Coward]

Get Internet from another provider.

Re:Internet Latency

[Sensible Clod]

Interesting sig you got there...

To keep this on topic: maybe they can poop on the Internet?

Re:Internet Latency

[JhohannaVH]

Absolutetly. We also had problems with people getting access to our University site, which *SUCKS* because how else are we supposed to help train all these new insurance adjusters. I feel your pain, seriously.... Especially since many of our transactions were backlogged for days. :( And I don't think our company has ever been seriously affected before like this. We're looking into alternative solutions.

Re:Internet Latency

[JhohannaVH]

*lol* Thanks.... a friend of mine said this yesterday when he got chopped off of Google Talk during the midst of a conversation, and I laughed so hard, it was hysterial. Go fig. :P

I think they (they being L3/Cogent) already *did* poop on the Internet, hence why the agreement. I'd like to know that this kind of stuff *isn't* going to happen anymore with the agreement. But one can never be so sure.

Re:Internet Latency

[Anonymous Coward]

multihome instead of single home with just cogent or just level3. or, better yet, find a suitable "tier 2" provider that will always have a full view regardless of peering arrangements.

Re:Internet Latency

[dekemoose]

Important applications should not be run on single homed networks.

Re:Internet Latency

[pturing]

What recourse do they have? They can look around for their frikken SLA. If they don't have one, then it's time for them to realize they aren't paying for guaranteed connectivity.

Re:Internet Latency

[JhohannaVH]

I'm certainly sure that we do. I'll pass this on. :) Thanks!

Re:Internet Latency

[Sensible Clod]

Well, I was referring to the businesses that were affected, and humorously suggested that as their means of recourse.

Your friend is very funny, though. Incredible.

Re:Internet Latency

[elm3r]

Agreed. But when considering the amount of bandwidth needed for a specific institution, multihoming is sometimes cost-prohibitive. Lord knows that it'd cost a fortune to have enough multihomed bandwidth to support this institution.

Re:Internet Latency

[jacksonj04]

Doesn't this indicate a problem with the net's underlying design? I was always under the impression that if one peering went offline, even at tier 1, the traffic would simply redirect through any available route (As soon as the routers update).

Correct me if I'm wrong here, but unless Level 3 and Cogent are the only two tier-1 entities in a network there should be a non-direct route between them, even if it involves going through other peers?

Re:Internet Latency

[Anonymous Coward]

I think Level 3 was actively filtering all packets from Cognet's IP range

Re:Internet Latency

[Vancorps]

Its sort of the big issue with our infrastructure now. The companies are too big to provide that kind of security because of politics involved but if they were smaller then they wouldn't have the capital to make the infrastructure what it is.

I'd say they made their point with the depeering. One disruption now is better than the company going under later.

Still, it is scary that one event can have such an impact.

Re:Internet Latency

[Anonymous Coward]

What recourse? Just switch to other providers that aren't going to let their customer's connectivity suffer by playing these silly games.

Re:Internet Latency

[Leroy Brown]

Like the previous poster said, any important applications or networks should be multi-homed.

A couple solutions that would have avoided outage:

Single link without BGP:
Your organization could have purchased its DIA from an ISP that was multi-homed.

Dual links with BGP:
Your organization could have purchased a small amount of DIA (relative to your main pipe) from another provider, weighted/prefixed it less desirable than your primary connectivity, and not lost connectivity to L3/Cogent. You could have possibly even worked a deal with your vendor because of its use as last-resort/emergency connectivity.

Free market solution regulation

[dada21]

After hearing hundreds of posts clamoring for government regulation (ie, slow to respond expensive monopoly), the best solution came quickly.

Why did this agreement happen? It happened because the market required it. Customers were unhappy, producers lost money, no one profited on either side.

If we pushed for regulation, how many years and billions of dollars would replace what two corporations did in a week or two on the demands of their customers?

Stronger ties, but still breakable

[elm3r]

In the new agreement, there are clauses that state that Level3 can again try to charge Cogent if their traffic amount is grossly over that of Level3's. So, while this is definitely an improvement, it doesn't rid all potential future problems.

If anything, this definitely hammers home the idea of multihoming...

Re:Stronger ties, but still breakable

[wfberg]

In the new agreement, there are clauses that state that Level3 can again try to charge Cogent if their traffic amount is grossly over that of Level3's. So, while this is definitely an improvement, it doesn't rid all potential future problems.


Though you have to wonder how they determine whose traffic is bigger. Surely, if a level3 customer is downloading from a cogent customer, the reverse is also true. If a connection is cut off, there will 2 unsatisfied customers, one on each end of the connection.

Of course, level3 has more customers than cogent. So if level3 cuts off cogent, a lesser proportion of level3 customers will get pissed off than cogent customers. So it's just a matter of which party has the deepest pockets, and who's the biggest bully.

Re:Stronger ties, but still breakable

[Eunuchswear]

In the new agreement, there are clauses that state that Level3 can again try to charge Cogent if their traffic amount is grossly over that of Level3's.

How do you know that? Have you just broken your NDA?

The press release says:

Under the terms of the agreement, the companies have agreed to the
settlement-free exchange of traffic subject to specific payments if certain
obligations are not met.

You're right about the multihoming at least. Only a fool isn't multihomed.

Re:Free market solution regulation

[Jeff DeMaagd]

When was the last time you remember that Sprint customers were cut off from being able to call MCI subscribers?

I don't want massive regulation, but something simple to prevent deliberate cut-offs would be nice, and it appears that the free market didn't solve that problem.

"Pay up or we disrupt your business..."

[DamienNightbane]

Is it just me, or does Level 3's ultimatum sound alot like an old fashioned protection racket? How is this any different from the Don sending someone to smash up someone's shop after the owner misses a payment?

Is there any way to get law enforcement involved? What about a class action lawsuit?

Re:"Pay up or we disrupt your business..."

[ergo98]

Is it just me, or does Level 3's ultimatum sound alot like an old fashioned protection racket? How is this any different from the Don sending someone to smash up someone's shop after the owner misses a payment?

No more than the Boston Tea Party was a protection racket. Standing up for your own self-interests isn't insidious or evil.

In this case Peer1 thought they were getting screwed and wanted out - the other party can either agree to new terms, or take advantage of a competitive marketplace and find someone else to do business with.

From the sounds of it (obviously based on a limited amount of information), Peer1 was getting shafted, and rather than it being a peering agreement where they cover each other's backs, they were being leeched while getting little in return. Supposedly they tried many other options to get a new agreement in place.

Re:"Pay up or we disrupt your business..."

[Professor_UNIX]

The only people Level3 is hurting around their own customers. Everyone else could still reach Cogent through other providers but Level3 removed the static routes on their network that directed traffic to Cogent, essentially blackholing them for Level3 customers.

Re:"Pay up or we disrupt your business..."

[aredubya74]

Sorry, but that's plainly ignorant. There's no "static routes" on Level3 that send traffic to Cogent. The way peering works is that Cogent announces their routes to Level3 (and UUNet, and Sprint etc.) using the BGP routing protocol at interconnections in their network. UUNet does the same for their routes, Sprint for theirs, and so on. The big boys do NOT announce each others' routes to each others peers, only those routes that they originate or that their downstream customers own. If all of those connections to Cogent are somehow severed (manually, or in a backhoe-derived outage scenario), Level3 cannot route to Cogent, and vice-versa. The same would go for any other provider.

Back to the concept of peering. The idea is that Cogent will only send traffic to Level3 that Level3 announces routes for. They will also agree to only peer if there's a relative balance of traffic across their interconnections (say, within a few percentage points up or down of a 50/50 split), or a split of costs (e.g. if Cogent is going to send more bits to Level3, they pay for the interconnection costs, and the router ports, and perhaps the colocation costs for the gear to make up the difference). In this case, Cogent was dumping on Level3 at a 10-to-1 ratio of bits sent to Level3 to bits received from Level3. In those cases, the relationship is no longer that of peers (equals in utilization), but of provider (Level3) and customer (Cogent). Like any other legitimate enterprise that uses Internet bandwidth to sell a product, Cogent needs to buy their bandwidth from large providers. If they choose to use Level3 as one of those providers, that's their preogative. They could choose UUNet, Sprint, AT&T and many others to provide transit. However, that would cost them money. Instead, they cajole providers like Level3 into setting up peering, promising relatively equal traffic exchange, and then dump as much traffic to Level3 as they can get away with (in some cases, even routing to other tier 1 providers through Level3). This breaks the agreement, so Level3 had every right to depeer Cogent, no matter how sucky it turns out for the rest of us.

Disclaimer: I used to (5+ years ago) work for Genuity, the bankrupt ISP bought by Level3. I know how this stuff works, and have no axe to grind or promotion to give either provider.

Re:"Pay up or we disrupt your business..."

[Dudio]

in some cases, even routing to other tier 1 providers through Level3

Is there any public corroboration for this? From what I've seen, it certainy appears that Cogent is taking advantage of hot potato routing to make their peers take care of global routing for them, but I haven't seen anything to indicate that they have been using their peers for transit to other networks.

Re:Free market solution regulation

[dada21]

I use neither Level3 nor Cogent. I use an ISP with a multitude of backbone connections. That's provided by competition which isn't hampered by expensive regulation or licensing.

As for Sprint and MCI, I've had 5 occasions where my LD provider lost connectivity. I've been using risky 1c/minute phone cards for years and the companies often go belly up, with their 800 #'s pointing to nowhere.

Don't harm my choices because you use a bad provider.

Needs to be regulated

[nexUK]

Tier 1 peering needs to be regulated in certain situations. The Cogent and Level 3 "who has the bigger dick" contest has caused isolated pockets where full routes/reachability to certain parts of the Internet wasn't available for some Cogent and L3 downstream customers. Get these big boys to maintain settlement free peering when a certain amount of the routing table "belongs" to them. simple.

Re:Needs to be regulated

[fuzzy12345]

Tier 1 peering needs to be regulated in certain situations.

I'm for some regulation of the Internet, but not here. These guys went back to the table because they each had guns to their heads; their customers (on both sides) didn't really care whose fault it was and would've started leaving.

Calling for regulation would likely lead to California energy crisis-type situations: PG&E and Con Ed were both required to retail (at a fixed price) stuff they had to buy wholesale (on the open market) and when the wholesale price went above retail, bankruptcy. (Don't get into market manipulation, that's a peripheral issue). The Internet has been remarkably successful precisely because any yahoo with a router and a cable crimper could build out more of it, without a license, approvals or anything else.

Re:Needs to be regulated

[Uhlek]

You're missing the point. Cogent isn't a Tier 1 ISP. They're close, but not quite. To be a Tier 1, that means you don't pay for peering -- period. Cogent does.

This was a fairly straightforward business problem. Settlement-free peering only occurs when its in the best interests of both parties to do so. There are massive costs still incurred on each end, but they simply don't exchange money. The traffic in both directions is equal enough that neither side is incurring a loss. L3 determined that they were, and announced to Cogent that their settlement-free peering agreement was going to end.

Rather than doing what they should have done, and either ponied up the cash to L3, or reached a transit agreement with another ISP (say, a tier 2) to receive L3's prefixes and get its own prefixes onto L3's network, Cogent allowed the depeering to occur and used the resulting disruption to the Internet to their own advantage by calling L3 out.

They, in effect, allowed a major outage to occur in order to avoid paying for transit to L3. L3 gave them something like 90 days notice, plenty of time for Cogent to develop a contingency plan.

Yet, they didn't. Thier customers immediately became unreachable from L3's network, and their customers were unable to reach L3. They allowed this situation to continue, leveraging it for a public relations backlash against L3, and attempted to lure L3 customers to Cogent.

I'll be the first to admit my understanding of the issue is not 100% -- so if I'm missing a critical point, please let me know. But, from my understanding, let me be the first to say this is not a major problem with the Internet, nor is it something that regulation would do anything to fix. This is a bullshit back-room business decision by an ISP trying to save a buck and make a name for itself.

Re:Needs to be regulated

[nolife]

A simple approach but obviously not the status quo method of deciding on who should pay for what..

L3 customers are requesting more traffic from Cogents customers then is going the other way. Why is any one direction of traffic considered a load and another considered a source for income and different from each other? It seems to me as these two companies are concerned, more L3 customers desire and need Cogent traffic then Cogent people that need the L3 traffic as noted by the obvious business difference that is raising this issue. Why is the traffic from Cogent considered Cogents traffic when in reality, you could consider it traffic requested by L3 paying customers? The traffic from Cogent to L3 is a mutual benefit to both Cogent supplying and L3 customers requesting equally. Cogent could claim that L3 is sucking down a disproportionate amount of its bandwidth compared to what they are own customers are getting back?

I know, not a status quo, that's the way is always has been, your an idiot, blah blah, but what makes one way right and accepted and the other not? Both ends have customers paying for the bandwidth they send and recieve.

Re:Needs to be regulated

[DDumitru]

Level-3 and Cogent are not all that differnet in size. They both cover the US from coast to coast. They both go to Europe. Level-3 goes to Hawaii, but that is the only real footprint difference.

Cogent has fewer US POPs, but still hits all four corners. Cogent claims 80G backbones. Level-3 claims 110G backbones. Both networks have real 24x7 operations centers and skilled operators.

You cannot deny that there is a differnce in size between the two companies, but it is pretty easy to describe both as "Tier-1" networks.

If you read the Level-3 peering policy statement:

http://www.level3.com/1511.html

(which is where the Level-3 bw numbers come from, so they might be a little old), it would look on the surface that peering with Cogent is perfectly reasonable.

The complaint of Level-3 is twofold. The first one is, "we are bigger, so you should pay us". It is true that L3 is bigger, but whether they can throw that around and make Cogent pay is not 100% clear. When you buy from Level-3, you "expect" to be able to talk to Cogent. I am a Level-3 customer and L3 did not imform me that they were cutting these lines off. I was pissed. I knew that L3 could legally do what they wanted, but I thought it inappropriate for them to do it without giving their customers notice. Level-3 chopped off 45 million IP addresses when they de-peered Cogent. Even if L3 thought it was Cogents "fault", it was still L3 that threw the switch.

One other issue of this is that some major suppliers buy from L3 and don't multi-home. Thus, RoadRunner cable modems lost contact with all Cogent sites. This left RR answering the phones from customers. I am sure it cost RR a lot of hassle, and probably some real money. I cannot imaging that RR was not threatening L3 with "we are going to multi-home and you will lose 60%+ of our traffic if you don't get this working now". L3 had a lot to lose.

This is the real problem with Level-3 taking action. Whether Level-3 is right or wrong, it's customers will still blame L3. L3 did enter into peering with Cogent. L3 did initiate the termination of the peering arrangement. In doing so, L3 did cutoff a large chunk of the internet to it's own customers. Image if Level-3, on it's home page, prominently stated that you were not buying connection to the "whole" internet. No one would buy service from them. Level-3 gets customers because, on the whole, we trust Level-3 to work hard to keep us connected. This trust is what L3 is selling. The de-peering has damaged this trust and will, in the end, cost L3 a lot more money than they were trying to extract from Cogent.

The only real, technical issue between L3 and Cogent is traffic symmetry. Cogent sells to a lot of hosters. L3 sells to a lot of end-users (cable modems, dial-up, etc.). Thus more bytes move from Cogent to L3 than from L3 to Cogent. If the two networks only touched at one point, this would still be fair. After all, it takes a L3 customers to request a Cogent porn page. But the peering between L3 and cogent happens at multiple points. Thus, you get the "hot potato" case where traffic from Cogent dumps to L3 early (and vice versa, traffic from L3 dumps to Cogent early). This means more Cogent bytes travel on L3's network than on Cogents network. This is presumably what L3 wants Cogent to pay for. While this may be a "real" traffic issue, I suspect L3 wants more from Cogent than is actually reasonable. What would be interesting would be for L3 to advertise its routes to Cogent differently at each peering point. Thus Cogent would carry it's own outbound traffic over more of it's own network and the "hot potato" effect would not happen. Perhaps BGP does not work that way, but it should be able to. I suspect L3 was more interested in "monetizing" it's relationship than in making it "fair".

I also just noticed that L3 is starting to compete with Cogent on price. It used to be that the $3000/mo for a 100mbit

Re:Needs to be regulated

[Anonymous Coward]

Level 3 and Cogent aren't even remotely the same size. Level 3 OWNS it's network from coast to coast. It does not lease it. Well, perhaps small portions. But as far as being a long haul carrier, they could very well be one of the biggest. Certainly WAY beyond Cogents league. That comparison is just way off base. Also, I think some of you (not directed at parent) are confusing peering and transit?

Cogent isn't without fault here

[Dracos]

Are they going to learn their lesson and strike peering agreements with more tier ones then just Level 3?

Re:Cogent isn't without fault here

[fuzzy12345]

(with-cluehammer "You don't get it. Tier 1s have lots of peering agreements. A peering agreement with someone else DOESN'T entitle you to use their network to get to a third -- that would be transit. Basically, these guys said they wouldn't exchange traffic directly for free, and they wouldn't pay some other provider to act as a go-between, which, if you understand how these things work (reputation, game theory and all that) is perfectly logical.")

Re:Cogent isn't without fault here

[Dracos]

I do get it. The reason why this debacle was so visible is that Cogent only had (or still has) one (1) peering agreement, and that was with Level 3. If they had more, as you claim all tier 1's do, then they could have routed over other peer(s).

Re:Cogent isn't without fault here

[Anonymous Coward]

*wrong* They have several peering agreements. The reason this happened is because they refused to pay transit to get back to L3 after the peering with L3 was shut off. The *ONLY* people that cogent was cut off from were single homed L3 customers, and vice versa. Please do some more research before you post.

Re:Cogent isn't without fault here

[slive]

Nope. Routing over one network to get to another is called *transit*. Cogent refused to pay for transit to get to L3. So they still had plenty of peering arangements with other networks, but none of those peering arangements allowed Cogent to reach L3. (And the same could be said in reverse.)

Re:Cogent isn't without fault here

[Anonymous Coward]

No, as others have said, you don't get it. Routing over another peer to another Tier 1 provider is a TRANSIT. The other tier 1 provider will charge you for that, no matter than you have a peering arrangement with them.

Don't ask them to learn their lesson until you have a clue about what the lesson should be.

Re:Cogent isn't without fault here

[Big_Al_B]

No. Cogent peers with many, many networks and buys partial transit from one, Verio.

The way peering works is:

1. Peers will propogate transit (paying) routes to everyone (other transit customers and all peers).

2. Peers will propogate routes learned from peers to transit customers.
d

3. Peers DO NOT propogate routes learned from any direct peer to OTHER direct peers.

It was _rule number three_ that prevented networks single-homed to Cogent to be unreachable from networks single-homed to L(3) and vice versa.

There were many small and large networks, including other "tier 1's" that peered with both networks (or purchased transit from them) who could have provided transit to keep those single-homed Cogent/L(3) folks connected, but they didn't. Why?

Because neither Cogent or L(3) was paying them to do so--and carrying that much traffic is not cheap, even for the largest providers.

Re:Cogent isn't without fault here

[Big_Al_B]

FYI

http://www.fixedorbit.com/AS/0/AS174.htm

Re:Free market solution regulation

[Anonymous Coward]

Actually the dispute dates back to early July of this year. Hardly a quick resolution.

consider an aphorism

[Quadraginta]

Give a man a fish, he eats for a day. Teach him to fish and he eats for the rest of his life.

If a government agency just enforced some prior restraint on the companies, what have they learned? Not to do what they did. What have they learned by being forced to solve their problem themselves? Not to do what they did, and also how to successfully negotiate with each other when things go awry, what the market really wants from each firm, how to rapidly re-evaluate corporate strategy in the face of adverse external events -- in short, how to be more "grown-up" in managing their own affairs.

Re:consider an aphorism

[dada21]

Great thought and one that really brings the point across.

The only place it falls short is that, historically, government intervention never heeds the consumers' needs, just the needs of the best briber.

Re:consider an aphorism

[Anonymous Coward]

"Teach a man how to make a fire and you keep him warm for a day. Put a man on fire and you keep him warm for the rest of his life."

In this case I don't know if government action would have been what was best. I really feel for Level 3 in this mess since they are being regarded as the bad guys while if they kept the status quo they would have been out of business.

Cogent was the one that abused their peering arrangement. Level 3 had been talking to them for a while to get them to find more peers to share the load but they didn't seem interested. What else could Level 3 have done other then pull the plug on them?

Re:Free market solution regulation

[lucifuge31337]

The best solution to this problem isn't regulation, it's the replacement of DNS with something non-centralized.

Care to explain what DNS has to do with a layer 3 peering arrangement?

Oh...that's right...nothing at all.

The interesting bit

[fuzzy12345]

Oct. 7: We determined that the agreement that we had with Cogent was not equitable to Level 3. [...] Cogent was sending far more traffic to the Level 3 network than Level 3 was sending to Cogent's network.

Oct. 28: The modified peering arrangement allows for the continued exchange of traffic between the two companies' networks, and includes commitments from each party with respect to the characteristics and volume of traffic to be exchanged. Under the terms of the agreement, the companies have agreed to the settlement-free [i.e. no-charge -- ed.] exchange of traffic subject to specific payments if certain obligations are not met.

So what happened? It's unlikely Cogent could say "Oh yeah, we'll get 50% more retail customers so as to send traffic your way." Level 3's customers squawked and Cogent insisted they wouldn't pay? (That's Internet Mutually Assured Destruction)

Re:The interesting bit

[petermgreen]

the exact details of what was agreed to will probablly never be public

one possible condition could be moving some of the peering to other locations so level3 has to do less work and cognet has to do more to get the traffic between the desired endpoints. I belive depeerings have caused changes like that in the past.

another possiblity is the peering is theoretically setlement free but cognet may end up paying some of the "fines" mentioned.

yet another possibility is as you suggest level 3's customers said enough and they backed down but put in some secret conditions to try and save face and make it look like not thier fault.

Re:Free market solution regulation

[Taladar]

How does decentralized DNS help with backbone depeering?

Re:Free market solution regulation

[Big_Al_B]

Disclaimer: I'm not a "freemarketsrulefreemarketsrulefreemarketsrule" neocon freak, and I'm not defending the GPs argument *at all*.

But your analogy sucks because the size comparisons are wrong. If L(3) == Sprint, Cogent == MacleodUSA, not Cogent == MCI.

Using Sprint & MacleodUSA as an example, they are not required by regulation to exchange minutes of use or maintain any business relationship unless some they have a negotiated a contract. And they may terminate any such agreement, to either one's advantage unless some special circumstances apply. And, in my experience, those circumstances apply more to accounting than actual traffic exchange.

Re:Free market solution regulation

[Anonymous Coward]

This has nothing to do with DNS, moron. It's about peering between L3 and Cogent. You know, IP routing... bgp... all that crap. Furthermore, how do you suppose we replace DNS? You need some centralized authority to prevent namespace collisions and provide consistent results. You don't have bitching rights about dns until you can come up with a solution that will serve those two goals and be reliable on top of that.

Pissing contest

[sunderland56]

Level 3 and Cogent are in a pissing contest?? Oh, wait a minute. That was peeRing. My bad.

Re:Pissing contest

[Joe U]

No, You had it right the first time.

Re:Pissing contest

[groot]

They should wait 'til December when their pee rings can make a real dent in the snow.
--laz

Re:Pissing contest

[Anonymous Coward]

is that similar to a circle jerk?

Re:so basically ...

[jcnnghm]

Except this has less than nothing to do with ICANN and DNS...

alas too often true

[Quadraginta]

I was, arguendo, taking the most optimistic possible view of what government could accomplish.

Re:Free market solution regulation

[dada21]

First, posting as an AC lessens the likelihood of a reply.

The ad hominem doubles it.

Now, to prove who the "moron" is...

http://www.ultradns.com/news/articles/051011.cfm

DNS is fully at fault for the reliance we place on peering arrangements. By nuking single endpoint routing via DNS, we could introduce the need for multiple routing paths.

If you live on an island with on bridge and the bridge (peering) goes boom, you're screwed. DNS allows people to "go cheap" by building only one bridge.

Dump DNS after implementing IPv6. Let Google, AOL, and others provide the solutions to mapping a name to an IP.

Oh, only big corporations can afford SEO? Bullshit. Why can small companies receive phone calls and mail? Because of third party services. DNS is monopoly, dump it.

How does the phone company handle this?

[koehn]

I assume that the phone companies and mobile companies have similar (though not identical) issues to this. Aren't they mandated to provide access to their networks to other providers (e.g., Vonage)? What restrictions/costs are typically involved?

Re:How does the phone company handle this?

[Anonymous Coward]

the telephone system is regulated, access is mandated and guranteed.

You think they would sell access to vonage if it was their choice?

Re:How does the phone company handle this?

[gzunk]

In the UK you have something called Termination Call costs, which is a price that the company A charges company B when a customer on company B's network calls a customer on company A.

There has been much gnashing of teeth over this, because everyone was charging everyone else an arm and a leg (therefore ensuring that a nice fat profit was made) and the government regulator investigated and told the companies to reduce their prices. Don't know if they've done it yet.

Re:How does the phone company handle this?

[Anonymous Coward]

In the US, it is by way of "settlements." When a long distance provider sends a call to a local telco, they pay a fraction of a cent per min. to the local telco.
Exactly how much is really a really complex mess of negotiations and regulation. Millions are spent in legal expences and lobbying when arguing about the exact payment. The one good thing is that it does not result in disconnection, just big lawsuit that lasts for years.

Re:How does the phone company handle this?

[aramps]

Phone companies to have similar issues. They are not mandated to provide access if a carrier refuses to pay termination fees. The fuzziness is in exactly what fees are paid (it's different for different kinds of traffic (local v. inter or intra state long distance, international, etc.) for different kinds of carriers (small rural phone companies, urban competitive carriers, etc.)

The area that's interesting is how small competitive carriers interact with the big boys - no different from tier 1 and tier 2 networks in the IP world, except that there's this underlying notion of Universal Service glommed onto the top of it - meaning we have traditionally offset the cost of networks by charging for access to them. This is different than charging for the incremental cost the network bears per unit of traffic.

All of this combines into a big ugly mess where people game the system one way or the other and it falls apart. (basically this is the fault of the way the FCC and congress treated VoIP as a disruptive technology) - no judgement on my part if it should have been treated that way or not.

There is a world view that says the network is better the more people that are on it, then there's the competing view that says the network is better when it's controlled by one entity to ensure maximum efficiency. Either one is an imperfect market - one lets the Cogents cream for profitable traffic (this is called arbitrage), the other lets monopoly pricing occur as a result of market power. Part of the concern about the acquisition of worldcom and at&t is that a very large network of unique endusers (think DSL consumers) coupled with a huge chunk of the IP backbone are going to be able to leveraged by some very, very savvy people with market power (with a lot of regulatory experience related to the termination of certain kinds of traffic) and the IP community isn't gonna know what hit them until they're crying uncle.

ugh

[Quadraginta]

Right, take the big decisions on networking away from IT CEOs who, whatever their faults of arrogance, need customers if they want their pensions to be secure, and who have stablesful of actual experienced engineers working for them, and instead give control to a bunch of Washington 9-4 lawyers who majored in Government because algebra was hard.

The problem with wishing abstractly for regulation is that it overlooks the profound difficulty of finding competent regulators who are not already in the business.

Re:ugh

[nexUK]

Generally these decisions are not made by CEOs. They are made by powerhungry network architects at NANOG shindigs. You can picture the scene, vendor sponsored "Cocktail Parties" with bad 80's music, a 1990's wardrobe and ZERO women. These dudes strike up their lovely peering deals after they are wasted on 2 Bud lights, of course they can remove this privilege in heart-beat if you diss their hardware, or top-post on their mailing list.

Re:ugh

[Quadraginta]

You can picture the scene...

Indeed I can. Poor bastards. Bud Light and no women, you say? Dante never wrote of this circle of Hell.

Re:so basically ...

[cwtrex]

Does a company's ethical position have nothing to do with the toothbrushes they sell? How can you be so short sighted as to try and say my post is offtopic? Of course it has to do with ICANN! What they do in this situation will make a case that can be used against the capitalistic way the internet is run. Can't you see that? But if my logic is wrong on this, then please, by all means explain why.

MOD PARENT UP!

[Sensible Clod]

He hit it on the head. This was an agonizing, tortuous, path to resolution, and it was hardly necessary to drag all the paying customers through it.

More like a bum behind a year on rent

[Anonymous Coward]

The true analogy is that of a landlord coming to collect rent way overdue, and finally evicting the tenant...

If you had a car, and a friend borrowed your car for that majority of the day every day, would you tell them they needed to start helping with the payments or just let them keep going?

In short why should leeches be allowed to keep leeching money from another company with no repercussions.

Re:so basically ...

[dekemoose]

Do you even have any idea what it is you're discussing here? This is two companies who had a business agreement, one company abused it and got smacked by the other company.

Think this is about the US? Why don't you look into France Telecom's de-peering of Cogent awhile back.

This is not an Internet thing in that it affects the entire Internet. It is an internetworking thing in that it affects the way two ISPs exchange data.

Sit your knee-jerk, loud-mouthed, over-opinionated, under-educated ass down and shutup until you can remove your head from where you have it stuck.

Leve3 and us. Also why did it break stuff?

[starwindsurfer]

I am a sys admin for a small atlanta ga ISP, when Level3 de-peered you dialup users that were connecting to Level3 POPs couldnt connect, our call center was flooded and we were scrambling like mad to switch everone over to the Aligence Telcom POPs. L3 really needs to think about the broader ramifications of their actions.

Also, The internet is suposed to be dynamicaly routed, ya know BGP3 and so on. Why did this break so many things? If the route was down, shouldnt the routers just use the next best preffered?

Re:Leve3 and us. Also why did it break stuff?

[Uhlek]

The core of the Internet is not run like you think it would be. While BGP is dynamic, when and where various prefixes (network address blocks) are advertised is tightly controlled.

When you peer with an ISP, that means you only exchange their prefixes for yours. Any other networks that may be reachable via that ISP are not advertised back to you, just like they don't send your prefixes to the rest of the Internet.

Access to other parts of the Internet via an ISP is called transit -- this is what we're all most familiar with. You give your prefixes to them and they take care of exchanging it with the rest of the Internet, and they give you the rest of the Internet's prefixes.

Lets say there are two ISPs. Lets say its Alpha (a Tier 1 ISP) and Beta (a non-Tier 1). Alpha and Beta have a settlement-free interconnect agreement -- meaning they peer with each other. Remember, that means that they can only access each other directly. To go to the rest of the Internet, it means that they have to go to other providers. Alpha uses its other SFI agreements with all other Tier 1 providers to do so, Beta may use other SFI agreements and paid-for peering or transit agreements with other providers. For so-called "tier 2" ISPs, it's often a complicated mess.

Then, lets say Alpha decides that its SFI agreement with Beta is no longer in its best interest. So, they tell Beta they're going to depeer if they don't start paying for the peering.

Beta has two options to ensure that their customer base will stay functioning. They can either come to an agreement with Alpha, or come up with a transit agreement with another ISP, either another Tier 1 ISP or Tier 2.

It gets more complicated than that, but, that's the basic jist of it.

Re:Leve3 and us. Also why did it break stuff?

[starwindsurfer]

Wow, thats the best ive ever seen it explained, thankyou.

Re:Leve3 and us. Also why did it break stuff?

[Anonymous Coward]

There was no alternate route because the traffic between the two networks via the peering backbone. This is made so that, basically usuers could use the peer "highway" instead of flooding all the smaller interconnected networks trying to find the best route. All of Level 3's and cogents traffic funnels into their backbones which are linked by massive lines. This massive line is apparently what they "cut" during their little tantrums. The bottom line is money, one company believes the other should pay because of traffic, or whatnot. It's unfortunate, tho, that your customers had harships.

Re:Free market solution regulation

[BridgeBum]

DNS and routing really have very little to do with each other.

Most websites of any size whatsoever not only have multiple IP addresses assigned to the site (DNS), but also multiple links to the internet across carriers (routing). A problem in either area can cause diruption to clients, but that doesn't make them the same system.

The link you provided (minus the marketing noise) sounds like a proximity based DNS solution...also not revolutionary. Many site-to-site load balancing solutions use response time for DNS queries to determine the best IP address to return. In the event of fragmentation like happened between L3 and Cogent, if only one DNS server was avaiable on the isolated cloud (Cogent), the IP returned should be a web server on the same network. Voila, no customer disruption.

In short, yes it is possible to use DNS to work around some routing issues. It's also possible to use routing to work around some DNS issues. (example: cached DNS entries leading to a dead site, but natting the traffic to a real server and then routing the traffic accordingly). In reality though, they are separate systems. There already is a need for 'multiple routing paths', at least for any web site which wants to come close to 99.999% availability.

It wasn't de-peering.

[Anonymous Coward]

Level 3 didn't just switch its connection to Cogent off, it left it running and tarpitted any traffic going through it. Like other people are saying, there'd be a class action against them if, say, they were a power company deliberately sending surges into other companies' grids.

STOP THE CLUELESSNES!

[fuzzy12345]

Dudes, there's a lot of cluelessness here, about tarpitting, routing around failure, next best route etcetera.

Being a tier 1 means, essentially, HAVING NO DEFAULT ROUTES. You make deals with all the other tier 1 providers for direct connections at various places around the country and, if you can't colocate with a particular tier 1 in a particular geographic location, you pay another provider for transit from you to that tier 1. Being at the top of the pyramid, there's no default route you can hand packets off to when one of your connections fails - because that would mean somebody else was providing you with a free lunch.

Of course, these guys are constantly squabbling ("we're bigger than you, so you should be paying us for the privilege") but, since disconnecting affects both peers' customers, it's really cutting off your nose to spite your face.

Re:STOP THE CLUELESSNES!

[Big_Al_B]

I agree with you that there's a relatively large clue void in this whole discussion, but...

Being a tier 1 means, essentially, HAVING NO DEFAULT ROUTES. ...is wrong too. I work for a regional service provider, we have no default route, and we're not--by any stretch--a tier 1.

Tier "X" is just a marketing droid term and is technically meaningless. The networks popularly described as tier 1 DON'T BUY TRANSIT SERVICES from other networks. Transit services can be defined as a service where the "transit provider" agrees to:

1) Advertise routes learned from all peers and other transit customers to the transit customer.

2) Advertise routes learned from the transit customer to all peers and other transit customers.

3) Carry transit traffic based on those routing agreements.

Instead, "tier 1's" engage in "settlement free peering" with networks whom they're reasonably sure they will share an equitable traffic load. This also means they agree NOT to share non-transit (e.g. peer) routes and traffic with other peers.

You make deals with all the other tier 1 providers for direct connections at various places around the country and, if you can't colocate with a particular tier 1 in a particular geographic location, you pay another provider for transit from you to that tier 1.

Again, not as clue-rich as you probably intended. You're confusing paying for "transport" with paying for "transit". Colocation and geography is coincidental to peering or transit agreements.

We, for example, have a transit agreement with a provider one suite away from one of our POPs. In contrast, at one point we peered with more than 70 networks over a long haul OC3 to a midwestern NAP several hundreds of miles from us. We pay that transit provider nothing to transport an OC12 between rooms (because we pulled the fiber), but we paid a transport provider a significant amount for the OC3.

On the other hand, we do pay the transit provider for routing and bandwidth, but nothing to any of the peers we meet over the OC3. See?

Being at the top of the pyramid, there's no default route you can hand packets off to when one of your connections fails - because that would mean somebody else was providing you with a free lunch.

Step away from the default route stuff and you've pretty much got it. Peers don't give away transit is the moral to the story.

Of course, these guys are constantly squabbling ("we're bigger than you, so you should be paying us for the privilege")

Peering with a given sized network is a complex proposition. If you peer with someone who you could/should be selling transit, you've set a precedent for all other similar size networks to say "me too" and your transit revenue goes dry. For years we sold transit to a metro ISP who kept pushing peering as an option. We politely declined due to the above.

but, since disconnecting affects both peers' customers, it's really cutting off your nose to spite your face.

Not nearly in all cases.

Re:so basically ...

[cwtrex]

No, I'm not talking about simply US control ... I'm talking about government control. Part of what I got out of other countries wanting a piece of the internet pie was that they wanted capitalism out of it and this would support that argument.

Re:Free market solution regulation

[dada21]

Yes, exactly!

Why did you end your post before you were done?

. There already is a need for 'multiple routing paths', at least for any web site which wants to come close to 99.999% availability.

Yes, with today's antiquated DNS and point-to-point IP structure. But why stick to ancient rituals?

We want information, we want it now, we want it fast. The web as we know it is slipping, specifically because of DNS and PTP services.

Why should McDonalds get mcdonalds.com? Boring. Let McDonalds hive a site into a WWWtorent. Type "mcdonalds food" into GoogleWWWikiTorrent and get that site. You're a seed for others, closer to some than the old McD servers. Trusted seeds gain power. Untrusted don't.

You want "McDonalds", the Firefly season 3 episode, you can get it. Domain structure is dead, boring, monopolistic, over-regulated.

Yes, my 'solution' isn't well thought out. People can created poisoned seeds (and their IPv6 will be worthless or their permanent IPv8 will fail to seed. Whatever, the solution is probably uninvented yet, but within our grasp.

I fail to see the need for DNS or peering agreements. Give me complex 3D routing and massive search engine utilization. We'll soon forget that WWW, MP3, XVID, POP3, DSL, and VoIP were considered different. Data is data, held back because of old manipulation techniques.

Re:Free market solution regulation

[bn-7bc]

Well you have a point, but tha fact is (at last here in norway, dont know about the us) that MCI gets payed bu sprint everytime one of sprints costumer calls an MCI costumor, this is called a caal termination fee so MCI actualy get revenue as apposed to ip peering whitch is not a revenue generator.

PS: as I said I'm unshore about the system in the us, so if I'm wrong plz correct me but blz don't mod me down, we all make missatkes but I'm willing to lern as song as pople give me the chance.

On Peeing?

[Anonymous Coward]

WTF?

Re:Free market solution regulation

[blueskies]

I don't want massive regulation, but something simple to prevent deliberate cut-offs would be nice, and it appears that the free market didn't solve that problem.

So you are all for the free market except when the free market comes to a conclusion that you disagree with? Is that what you are saying? The free market has solved this problem it just took a little bit of time to converge on the answer.

Re:Free market solution regulation

[Pharmboy]

I don't want massive regulation, but something simple to prevent deliberate cut-offs would be nice, and it appears that the free market didn't solve that problem.

Capitalism is a self correcting system by itself, it is just not instantanious.

Regulations are not self correcting, and require that people who are being paid under the table by corporations, make the right choice and do what is right. If ABC, Inc. donates enough to congressmen, they get the better end of the regulation stick.

True Capitalism is similar, except the customer benefit, not congressmen. They have to "pay off" the consumer with lower prices or better quality, or they won't get what they want, money.

As to regulation making these "cut offs" less likely, it comes at the expense of higher prices and less choice for the consumer. In the end, they get a lower quality, higher priced product with a theoretical higher reliability rate. Regulations always equals higher prices, if for other reason than the expense of compliance.

The free market, on the other hand, means you get cut offs, but very seldom since the companies don't want to piss their customers off. It sounds like Cogent just didn't follow up, and didn't believe Level 3 would cut them off. Who is "right", I don't know and it doesn't matter since they have already decided this amoungst themselves in what *must* be an equitable arrangement, since both parties agree.

I am not against all regulations, but the purpose of any regulation should only insure there is a level playing field for the companies to compete in, thus giving the customer more choice and eventually better prices. I have no confidence that the government can set rules for a business it doesn't understand.

The best example is the regulations that required phone companies to open up their wires for competing DSL companies. It is still more expensive in most areas, still nowhere near as available as cable, and the actual implimentation is generally poorer.

Regulations are like lawyers. Yes, we need them every now and then, but if they are not used sparingly, they tend to fuck everything up and make a bad situation worse.

Fodder for the UN

[Carcass666]

It's this kind of corporate squabbling that lends ammo to those looking to "internationalize" the Internet. You know, to make sure that responsible countries, like China and Iran, can keep the Internet open and free...

Re:Free market solution regulation

[vertinox]

Capitalism is a self correcting system by itself, it is just not instantanious.

True... However this is very dangerous when it leads to public suffering.

Take the Great Depression in the 1930's. This of course was because of "total free market" situation without very little government intervention that just went "boom".

Technically, the failure of free market capitalism in the States lead to Fascism in Europe. (Yeah there are a gazillion other reasons it happened, but without the depressions and the economic failure of the Weimar Republic the NASDAP would have never come to power without the Socialists/Stalinists splitting the vote with the Christian Democrats), but if there is not a safty net then the people will not tolerate 10 years of economic depression until the market corrects itself.

Had FDR not stepped in then chances are there would have been more revolt risk in the US with either Right Wing or Left Wing extremists.

Re:Free market solution regulation

[Anonymous Coward]

There is no season three of Firefly.

Re:Free market solution regulation

[Leroy Brown]

If you, hypothetically, bought a single connection to either Level 3 or Cogent, and expected it to be reliable, and were bit by the recent de-peering event, then the problem is that you designed an un-reliable solution.

If you must buy a single dedicated Internet connection, then buy it from someone who has the redundancy built into their network _AND_ transit agreements.

If any regulation happens, it shouldn't be to interfere with the business arrangements between Tier 1 providers, it should probably be in the form of a waiver or notice to consumers of single-homed transit.

The problem I've noticed is that most consumers don't seem to realize the redundancy lost in single-homed transit. DIA is DIA they think. Even after the recent de-peering event, many are more confused than before because of all the misinformation.

Re:Free market solution regulation

[aramps]

The US uses a similar system, currently under review - particular attention being paid to the termination fees for rural telco providers (which have higher per line expenses). Larger carriers are moving to a system called bill and keep, where even traffic flow cancels out any termination or origination fees. Peering is a bit different on the IP side of things owing to it already being a bill and keep environment for tier I providers. This issue gets particularly messy where IP and telephony interact.

Re:Free market solution regulation

[Pharmboy]

This of course was because of "total free market" situation

It was because a great number of speculators had bought stocks on margin, and once the fall began, it rippled down quickly because no one could make their margin calls. 10% margins on many stocks.

I am not against ALL regulations, but there is a broad difference in the Great Depression, and customers going a few days without internet access, which no one had 10 years ago anyway.

If there was a "Great Internet Crash" or something, then I would be more open, but in this case, it was settled faster than the ink on any new regulation could have dried. In the big scheme of things, this is a tiny blip.

Writing regulations because of this one event would be a knee jerk reaction, where the "cure" could easily be more damaging than the disease. When all is said and done, the system WORKED. It caused minor problems for lots of people (including me) and loss of service for a few. The "damage" was much less than the outages you would get with a hurricane, and much shorter lived.

The free market isn't always pretty, but she's still the prettiest girl at the dance.

Corporate Greed is the Issue

[soren100]

Calling for regulation would likely lead to California energy crisis-type situations: PG&E and Con Ed were both required to retail (at a fixed price) stuff they had to buy wholesale (on the open market) and when the wholesale price went above retail, bankruptcy. (Don't get into market manipulation, that's a peripheral issue). The Internet has been remarkably successful precisely because any yahoo with a router and a cable crimper could build out more of it, without a license, approvals or anything else.

That's a really bad example -- The industry itself wrote the regulations! . Not only that, they wrote the regulations so that they could manipulate the market! In most regulatory situations, where the process hasn't become corrupted, the government allows the utility to always make a guaranteed profit percentage, so that it is one of the sweetest deals you can find. No other business form allows companies to make guaranteed profits, and zero competition. Greedy businesses, on the other hand, usually create all kinds of problems for themselves and everybody else, not to mention Fraud , which was the direct cause of the blackouts you mention. The only reason a bankrupty occurred was because a subsidiary was allowed to transfer the customer overcharges to the parent corporation!


The intetnet is successful precisely because it was a government-supported and funded project. (And yes, Al Gore did help a lot here, right-wingers!) -- A router and a cat-5 cable are absolutely useless without other routers and computers to connect to. As another example, you are perfectly free to build your own road, but it is useless unless it is connected to other roads. If roads were built under free market conditions, every road would be a toll road, with wildly varying conditions on each one.


If roads were like the Internet, this would be like having traffic stopped between two states because they couldn't figure out the proper toll road agreement. The fact that they had to have guns to their heads to get back to the discussion table is not evidence that the system works, it is evidence that corporate greed is not a factor that works toward consumer well-being!

Why not slow the link instead?

[Anonymous Coward]

Why do they have to cut peering entirely? Why not just cut back on the bandwidth of the link?

Wouldn't this be a better way to 'control costs' of an unbalanced link? The lower you drop the bandwidth, the closer those two traffic flows are likely to become, right?