Blackout Shows Net's Fragility

It doesn't come easy wrote to mention a ZDNet article discussing a recent outage between Level 3 Communications and Cogent Communication. A business feud inadvertently highlighted the fragility of the Internet's skeleton. From the article: "In theory, this kind of blackout is precisely the kind of problem the Internet was designed to withstand. The complicated, interlocking nature of networks means that data traffic is supposed to be able to find an alternate route to its destination, even if a critical link is broken. In practice, obscure contract disputes between the big network companies can make all these redundancies moot. At issue is a type of network connection called 'peering.' Most of the biggest network companies, such as AT&T, Sprint and MCI, as well as companies including Cogent and Level 3, strike "peering agreements" in which they agree to establish direct connections between their networks. "

Comments

Internet can route against natural calamities

[anandsr]

Internet cannot route when your providers do not want you to communicate.
Nothing can protect you in this case.
If on the other hand there was a natural calamity and every one was trying to get you access
then you would get it. Like it happened during Katarina.
This is not a natural calamity.

The best option is to ditch your provider if they are not a monopoly and if they are lobby to your government to create multiple providers.

Re:Didn't notice at all.

[lostlogic]

You would only notice if you are on one of these two networks. I am personally on UUNet at home and MCI at work, and my server is on SpringLink (via Schlund, who I am not familiar with). As a result, all of my traffic is completely unaffected. Customers on a single-homed connection through Cogent, or through L3 cannot see other single homed customers on the other network. The rest of us don't know the difference. The dumb thing that this article points out is that both Cogent and L3 are refusing to route packets destined for each other through the rest of the internet (probably for fear of fucking up other peering agreements by dumping too much traffic on their other peers). I believe there was a comment in the previous thread about this issue saying that traffic in one direction could be routed, but that even return packets were being null-routed at some point, preventing any type of connection from being established.

The fragility of the net

[elfguygmail.com]

It's very true, and anyone can see how a few big companies basically make the net work in north america. Simply do traceroutes to various big web sites, and you'll notice the packets always go across the same networks. The biggest one seems to be alter.net (MCI), with others including Level3, above.net, AT&T and UUnet. Basically you remove any of these and the North American part of the Internet would be in chaos. The problem is because most ISPs do the same thing. They pick a primary provider, and get a backup one. The problem is they all pick the same few primary companies, and their backup links are much smaller pipes.

Re:It always will be fragile

[brunes69]

Take, for instance, the connections running between Europe and America. I bet most of them run in almost exactly the same place on the sea bed because it's the cheapest / shortest path to take. A fairly localized geological disaster (at least in geological terms) could cut all the cables at once; or at least enough to make to difference.

This isn't a good example, because in this case most traffic would automatically be re-routed to go through Asia and the trans-Pacific cables. And if those went down it would go over South America Oceana.

It would get much slower, sure, but would not cause an outage.

There is no *technical* reason this peering relationship breaking down should be causing an outage either. If the both also peered with some third party that could service them both, like MCI or something, then the traffic would still get through. The companies are just being bull-headed.

Re:The small should pay for the big?

[Daniel Boisvert]

NANOG has been on fire with posts about this issue over the past few days. The following two from Leo Bicknell do a good job of explaining why this sort of thing would happen, why nobody in particular is The Bad Guy[tm], and why this issue has no relevance to the issue of internet resilience in the case of natural or manmade disaster:

http://www.merit.edu/mail.archives/nanog/msg12302. html
http://www.merit.edu/mail.archives/nanog/msg12350. html

Re:The small should pay for the big?

[Cally]

Check the NANOG archive over the last few days for far, far more than you ever wanted to know about "The Art of Peering: The Peering Playbook"... or read the book yourself.

Re:The small should pay for the big? (mod this up)

[gskouby]

About 4 months ago I got a call from a sales critter at Cogent saying "We will knock 50% off of the price you are paying for your L3 connectivity if you drop them and come be our customer." I was kind of surprised at the boldness of this proposition because they were specifically targeting current L3 customers. I was even more surprised to find out from others that this sales pitch from Cogent was company wide. Of course this pissed off L3 and that was the start of this pissing contest.

Re:A New Approach

[BeBoxer]

I know there's been talk of wireless mesh networks where everybody is both an end point and a router. This would work in populated areas but I'm not sure how well it would work for "long haul" connections which is what the issue is here.

If by "work in populated areas" you mean "slow the network to a crawl" then yes, it would work. Mesh networking is cool stuff, but you aren't going to build a backbone out of it. Wireless is really fast compared to your DSL line or cable modem. But it isn't even in the same ballpark as what you can do on fiber. Backbone links are running at 10Gbps or even 40Gbps. Full duplex, so that is 20Gbps or 80Gbps of "marketing bandwidth". Compared to what, 22Mbps or 54Mbps half-duplex for your wireless? You aren't going to build a comparable backbone out of wireless links running at roughly 1/1000th of the speed. Physics pretty much guarantees that fiber links will always be faster than wireless.

Re:It's Nobody's Fault

[Alioth]

Cogent COULD route around the damage - if they wanted to, but they don't.

If the peering point had been taken out by a bomb, the re-routing would have been performed in fairly short order. However, this is not the case here.

Level3 think that Cogent is taking the piss and is not a real peer. Level3 want Cogent to buy transit to reach Level3, either directly from them (or from someone else) because at the moment the peering is very lopsided, and costing Level3 a bucketload of money and giving Cogent a boatload of free bandwidth.

Cogent on the other hand doesn't want to pay for transit to Level3.

Right now, Cogent could route all their traffic for Level3 over transit they pay for. They don't want to do that because it won't force Level3 back into the peering agreement. So what they do is leave the link severed and do not re-route so that Level3 customers cannot get to sites hosted by Cogent. This means Level3 customers will grumble at Level3. Additionally, they offer a year's free transit to single homed Level3 customers just to raise the brinkmanship with Level3 a notch higher. Basically it's war between L3 and Cogent.

If Cogent re-routes their traffic, they are defeated and L3 will never re-peer. What Cogent are hoping is that enough angry customers on the L3 end will whine at L3 so L3 will be forced to re-peer.

For the rest of us in the peanut gallery (i.e. those of us who aren't single homed customers of Cogent or Level3) we can just watch the fun and games and throw peanut shells at the squabbling combatants because we don't see any black hole at all.

Fixed now?

[dereference]

The availability grid for the past 4 hours shows ~40% and the grid for the past 1 hour shows 100%. As noted by "Cally" below, I honestly have no idea how exactly this grid has been generated (hence my original disclaimer) but this certainly seems to indicate, from a practical standpoint, that the L3/Cogent issue has been very recently resolved. Indeed, from my (single-homed) L3 server I can now traceroute directly to a (single-homed) Cogent host.

That's not how peering works - here's the diff

[billstewart]

There are two basic ways that networks connect to each other - peering and transit. In a transit arrangement, one network (typically the big one) agrees to deliver any traffic the other network hands it, in return for a bunch of money, and it typically either advertises a default route (telling a small customer that they can send it all their packets) or a bunch of detailed routes and a default (telling a dual-homed medium-large customer how good its connections are to lots of places, but that customer might use another carrier for destinations that are closer with that carrier.) If you're an end customer, or a small ISP buying service from a big ISP, that's usually what you buy.

Peering arrangements are different. Two networks that have a lot of traffic for each other will set up direct connections, split the direct costs of the connections, and not charge for accepting packets from the other carrier. But they'll only advertise the routes for their *own* customers. If two small ISPs peer with each other, typically they're each also buying transit service from big ISPs, but it's cheaper for them to dedicate a connection or put bits on a public peering point like MAE-West than to both pay their upstream ISPs.

The biggest ISPs in the US are called "Tier 1" ISPs, and they all peer with each other rather than buying transit, though they might buy transit for international connections, if they can't get the other side to buy transit from them. It seems flaky, but it makes business sense, or at least it did for a while. In some sense, being big enough that all the other Tier 1s will peer with you is what defines Tier 1, and aside from technical issues, it's a marketing thing - "See, we're one of the big players!" Peering and Transit don't mix very well - you either connect to a given carrier by peering, or by transit, or else you spend a long time hammering out custom arrangements about exactly which routes you'll accept and tweaking routing tables.

Cogent is a Wannabe-Tier-1. Their main business model is to put fiber into big multi-tenant office buildings and sell everybody 100-meg Ethernet for about the price other carriers charge for one or two T1s. If I were a customer, I wouldn't expect there to be enough upstream to really get that much bandwidth all the time, but I'd expect to get more than a T1 all the time, and a lot more than a T1 almost all the time. Level 3 has apparently decided they're not getting enough value out of the relationship (i.e. not sending Cogent enough packets to make it worth their while) to keep peering, and wants Cogent to either pay them for service or get transit from somebody else. They gave them about 50 days to make other arrangements, but Cogent decided to play chicken with them.