Slashdot Mirror
Tier One ISPs Dying
xbmodder writes "Two tier one ISPs are down today. At about 23:30PST both Verio and Level 3 starting having problems with routes. According to Level 3 this is a software upgrade gone awry. Is this the end for Level 3?" Many, many reports about this are coming in, and if you're wondering why the stories were rather sparse overnight, it's because it's difficult to post them without internet access. Hope everyone else is back online too.
Take a look at the scoreboard now. The mentioned problems are gone and Level 3 is no longer in the red.
One that doesn't lease their infrastructure.
Eg. you have your own large backbone, you own all your equipment.
In effect, a small and wholly owned internet that peers with other internets.
Tier 1 are the huge ISPs, which peer with eachother (and don't pay eachother transit fees) and sell transit services to smaller ISPs (which do pay fees to send traffic through the Tier 1 ISPs). So yeah, bandwidth wholesalers is pretty accurate. See this wikipedia article.
The Official Steve Ballmer Webpage
This is not _just_ their server being down, this is the entire network of two tier 1 carriers.
The (basic) implications of this is that a good chunk of the internet as a whole is inaccessible to the rest of the internet.
"When a ball dreams, it dreams it's a frisbee"
While this only lasted a few hours, it still caused a mess across the North American Internet during those hours. The point is a small amount of big networks are responsible for over 90% of the traffic on the Internet. If alter.net went down it would be total chaos. If just one of the major peering points went down, sure the traffic would be rerouted, but overloading the other points at such high latency that it would be almost unusuable. You better hope no one destroys MAE-EAST or we'll have a live example of what ife without the Internet is like.
When a Tier 1 provider goes down, their customers go down too. That picture on the Boing Boing page shows a list of the Tier 1 providers. Every ISP that is NOT a Tier 1, gets their access from a Tier 1.
People speculate that Level 3 is dying because they've been making some really bad decisions lately, resulting in a lot of outages. A couple of weeks ago, they actively filtered out traffic from their competetor, Cogent, over a dispute from how much to charge at the point their networks exchanged traffic (called a 'peering point'). Now this. The rumor is that the company is in financial trouble.
It's not a rumour that Level 3 is in financial trouble - it's clear for all to see. They have crushing debt repayments right now.
The Cogent spat isn't over yet either - Level 3 are going to de-peer Cogent again on November 9th. They are trying to force Cogent to pay for transit, but right now it looks like Cogent holds the strongest hand and Level 3 will be once again forced to back down.
Oolite: Elite-like game. For Mac, Linux and Windows
Domesday is something like a census of Britain circa 1085. It has nothing to do with internet outages, which is more akin to doomsday.
Toronto-area transit rider? Rate your ride.
This sort of event provides motivation for overlay routing schemes, which can compensate for major outages along various routes of the backbone:
e rs/subramanianOver/subramanianOver.pdfn focom.pdf
http://www.usenix.org/events/nsdi04/tech/full_pap
http://www.eecs.umich.edu/~farnam/pubs/2005-hwj-i
An unjust law is no law at all. - St. Augustine
It was maybe 2 hours or so before new routing tables started spreading to bypass Level3's and Verio's networks, and afterwards it started stabilizing again, then it seems Level3 has since then woke up again. The XO network also had routing troubles from this btw, maybe more too. Sites and services such as AOL, SpeakEasy (when asked, they were stumped and could only say it affected all their customers, hehe), Google, and Wikipedia had access problems depending on where you lived during this timeframe.
Beware: In C++, your friends can see your privates!
Tier 1 ISPs own huge swaths of networks-- literally miles and miles of cable, and sometimes radio and other links. They route the traffic across these lines.
:-)
More precisely, Level3 seem to own 23,000 miles of optic fiber.
The rumor is that the company is in financial trouble.
Yeah, not so much of a rumor anymore either -- Level 3 loss widens.
Beware: In C++, your friends can see your privates!
I dropped my BGP session to Level3 but they did not retract the routes, so not only could they not route my packets but they claimed (via the routing table) that they still could. From my vantage point (Chicago) the problem was resolved in about an hour,
I'm sure you know this, but for the rest: "flapping" is the common term for when a router's routing tables rapidly cycle between two invalid states. The dead bird analogy is pretty descriptive, but the term "flapping" has technical and not allegorical origins.
Dewey, what part of this looks like authorities should be involved?
The reason that Level 3 isn't happy with the peering arrangement currently is that it's not even remotely even. Level 3 sends almost nothing over Cogent's network and Cogent sends over a vast majority of their traffic through Level 3. A peering agreement is based on the premise that the companies will be sending almost equal amounts of traffic through each network. Level 3 has been analyzing that for a time now but the last straw was when Cogent had a sales blitz targeting Level 3 customers saying that they would dramatically drop their prices to almost nothing to get them to switch away from Level 3. They are now also using the downtime that was experienced due to the peering problem in their advantage even though Cogent is in the wrong. Cogent knew about the depeering and did nothing to resolve it.
Oh yes, I'm aware of all of that - but (generally speaking) Cogent has the content, and Level 3 has the users. Guess who catches the most heat from the de-peering from its customers - Level 3 - as their customers will tend to see the problem first.
I predict that Cogent will do the same again as well - not lift a finger to fix the problem when they are de-peered on November 9th, and Level 3 will probably end up being forced to re-peer as customers whine that they are not getting the whole Internet and threaten to take up Cogent's free 1 year offer.
Oolite: Elite-like game. For Mac, Linux and Windows