Slashdot Mirror


Blackout Shows Net's Fragility

It doesn't come easy wrote to mention a ZDNet article discussing a recent outage between Level 3 Communications and Cogent Communication. A business feud inadvertently highlighted the fragility of the Internet's skeleton. From the article: "In theory, this kind of blackout is precisely the kind of problem the Internet was designed to withstand. The complicated, interlocking nature of networks means that data traffic is supposed to be able to find an alternate route to its destination, even if a critical link is broken. In practice, obscure contract disputes between the big network companies can make all these redundancies moot. At issue is a type of network connection called 'peering.' Most of the biggest network companies, such as AT&T, Sprint and MCI, as well as companies including Cogent and Level 3, strike "peering agreements" in which they agree to establish direct connections between their networks. "

16 of 287 comments (clear)

  1. The small should pay for the big? by hkmwbz · · Score: 5, Interesting
    As I understand it, these were about the same size and had an agreement, or didn't bother to bill each other. Then suddenly one of them figured out that "hey, we are bigger, so they should pay us!"... And the smaller one cut off the connection because they didn't want to pay since they considered themselves to be as big as their rival.

    What I don't get is why one of them would suddenly want the other to pay up. What's changed now, and why does the smaller company have to pay the big one's bills?

    Am I missing something here?

    --
    Clever signature text goes here.
    1. Re:The small should pay for the big? by Daniel+Boisvert · · Score: 5, Informative

      NANOG has been on fire with posts about this issue over the past few days. The following two from Leo Bicknell do a good job of explaining why this sort of thing would happen, why nobody in particular is The Bad Guy[tm], and why this issue has no relevance to the issue of internet resilience in the case of natural or manmade disaster:

      http://www.merit.edu/mail.archives/nanog/msg12302. html
      http://www.merit.edu/mail.archives/nanog/msg12350. html

    2. Re:The small should pay for the big? by Cally · · Score: 4, Informative

      Check the NANOG archive over the last few days for far, far more than you ever wanted to know about "The Art of Peering: The Peering Playbook"... or read the book yourself.

      --
      "None are more hopelessly enslaved than those who falsely believe they are free." -- Goethe
  2. No worries by WormholeFiend · · Score: 5, Funny

    The pr0n industry was designed to find alternative routes of delivery in case of Internet outages.

  3. Efficiency can be the enemy of robustness by dpilot · · Score: 5, Interesting

    This statement popped up in some of my security readings. It's most "efficient" to have one path between two places, and it's most "efficient" to set up peering agreements to route packets. But these efficient measures can introduce single points of failure.

    On a similar note, that's why there are 13 root DNS servers, and why most of us aren't supposed to use them. The DNS example though, is one where efficiency and robustness agree. It's more efficient, at least in terms of net bandwidth, to use a DNS server closer than the root servers.

    --
    The living have better things to do than to continue hating the dead.
  4. Call the helpdesk...wait, THEY don't even know! by digitaldc · · Score: 4, Interesting

    http://www.gamergod.com/article_display.cfm?articl e_id=329
    Good article on this situation here

    This situation has adversely affected various users of both companies' services. The inability of Level 3 to handle this situation in a fair and equitable manner to the consumers has alienated many customers and will continue to do so until the current situation is remedied. At what point is it good customer service to discontinue services due to no fault of said consumer base? Market history shows us that the single worse thing a company can do is to arbitrarily allow influences beyond the control of consumers to negatively impact services, determined by consumers to be status quo, without any warning or notification. If left unresolved and unaddressed, the current situation could set dangerous precedents for internet users across the country by allowing service providers to instantly discontinue provided services at the moment they feel that the services they provide are not being adequately compensated for from outside companies.

    On a side note, I was listening to Howard Stern (oh no!) this morning and he said that his Time Warner internet connection at home didn't work. Howard then called a tech guy to come and fix the problem, only for him to call a help desk to figure out what happened. The help desk didn't even know what was wrong. It sounds like Level 3 just pulled the plug and didn't notify ANYONE. Or maybe it was Cogent, the point is nobody outside of that dispute KNEW what was going on.
    This sounds like a good way to alienate your customers and/or ruin your business model. But that is just my opinion.

    --
    He who knows best knows how little he knows. - Thomas Jefferson
    1. Re:Call the helpdesk...wait, THEY don't even know! by peragrin · · Score: 4, Funny

      You want scary, I can show you scary. I emailed Roadrunner saying I would drop them if they couldn't due something.

      I got a semi canned response but it did have some techincal details. It also stated that if you wish to discuss the techincal nature of the problem go to www.ask.slashdot.org With a full link to the other article.

      Yep Roadrunner sent me to slashdot to get more information.

      --
      i thought once I was found, but it was only a dream.
  5. Peering by Neurotoxic666 · · Score: 5, Funny

    At issue is a type of network connection called 'peering.'

    In other news, the RIAA announced they've stopped an extremely large P2P network.

    --
    You are more than the sum of what you consume. Desire is not an occupation.
  6. Internet can route against natural calamities by anandsr · · Score: 4, Informative

    Internet cannot route when your providers do not want you to communicate.
    Nothing can protect you in this case.
    If on the other hand there was a natural calamity and every one was trying to get you access
    then you would get it. Like it happened during Katarina.
    This is not a natural calamity.

    The best option is to ditch your provider if they are not a monopoly and if they are lobby to your government to create multiple providers.

  7. It always will be fragile by squoozer · · Score: 4, Insightful

    The Internet will IMVHO always be quite fragile. While the design lends itself to robustness the reality is that there is only money for a few very big connections and therefore a disaster that affects one of these connections is going to cause wide spread outages.

    Take, for instance, the connections running between Europe and America. I bet most of them run in almost exactly the same place on the sea bed because it's the cheapest / shortest path to take. A fairly localized geological disaster (at least in geological terms) could cut all the cables at once; or at least enough to make to difference.

    If we wanted the network to be robust we would need to run cables up over the north pole and round the equator and probably stick in some satelite links as well. There just isn't money for that. People are willing to accept the risk that it might fail in extreme situations.

    FWIW I think the problem is worse on the global scale than the country scale. I imagine most developed countries probably have enough redundancy in their own country. It's the interconnects between countries that are probably the biggest problem.

    --
    I used to have a better sig but it broke.
  8. Not a redundancy issue... by boldtbanan · · Score: 4, Interesting

    As I understood the problem, redundancy wasn't an issue. Level 3 was actively filtering out request to Cogent, however they came in. The redundancy was working, but Level 3 was playing NetNanny and blacklisting all Cogent IPs.

  9. The fragility of the net by elfguygmail.com · · Score: 5, Informative

    It's very true, and anyone can see how a few big companies basically make the net work in north america. Simply do traceroutes to various big web sites, and you'll notice the packets always go across the same networks. The biggest one seems to be alter.net (MCI), with others including Level3, above.net, AT&T and UUnet. Basically you remove any of these and the North American part of the Internet would be in chaos. The problem is because most ISPs do the same thing. They pick a primary provider, and get a backup one. The problem is they all pick the same few primary companies, and their backup links are much smaller pipes.

  10. This was predictable by PhilipPeake · · Score: 4, Interesting
    The Internet was designed to be resiliant to malfunctions and automatically take appropriate action to ensure connectivity.

    Unfortunately, that is not the Internet that we have today. In the original Internet, every router knew about every network connected to the Internet. Most networks had connectivity to many other networks. Discovery protocols allowed alternative routes to be discovered if one failed.

    Today, we don't have a (mostly) fully connected net, we have ISPs who don't know anything about networks which they don't "own", only that certain IP prefixes need to be passed to ISP x, y or z.

    This makes the infrastructure much more fragile than it was originally intended to be. We ended up with this for a few reasons. First, the wimpy routers in use at the time had limited memory available to hold the network maps. The answer chosen was to no longer attempt to hold a full world view, but to divide the world into regions, certain IP prefixes would "belong" to those regions, and all any router would need to know about was networks in its region, plus how to route traffic to other regions, who would take care of routing within the region. This led to "backbone" connections - high capacity links needed because all traffic between regions now didn't "diffuse" through the network, but was channeled into specific connections. It also set the scene to allow the net to be commercialised, those regional centers were obvious "choke points" that an enterprising company could own and pretty much dictate the pricing to lower level enterprises who would do the dirty work of dealing with end-users.

    Slowly but sureley the Internet evolved into a system dependent upon a few companies with high-speed links between them - prime candidates BTW, as locations for government control to be imposed. The self-healing nature of the original Internet was lost because all traffic HAS to pass via the top level companies infrastructure and over their interconnect backbone connections.

    The "self healing" Internet is long gone.

  11. Re:The small should pay for the big? (mod this up) by gskouby · · Score: 5, Informative

    About 4 months ago I got a call from a sales critter at Cogent saying "We will knock 50% off of the price you are paying for your L3 connectivity if you drop them and come be our customer." I was kind of surprised at the boldness of this proposition because they were specifically targeting current L3 customers. I was even more surprised to find out from others that this sales pitch from Cogent was company wide. Of course this pissed off L3 and that was the start of this pissing contest.

  12. Monitor it yourself by dereference · · Score: 4, Interesting
    I found this site while trying to research the problem. I wish I had known of it earlier; it provides a very nice (near) real-time snapshot of all the Tier 1 peering:

    http://www.internetpulse.net/

    I'm not affiliated with them in any way, and I'm sure there are other similar sites, but I thought it was worth mentioning.

  13. Re:A New Approach by BeBoxer · · Score: 4, Informative

    I know there's been talk of wireless mesh networks where everybody is both an end point and a router. This would work in populated areas but I'm not sure how well it would work for "long haul" connections which is what the issue is here.

    If by "work in populated areas" you mean "slow the network to a crawl" then yes, it would work. Mesh networking is cool stuff, but you aren't going to build a backbone out of it. Wireless is really fast compared to your DSL line or cable modem. But it isn't even in the same ballpark as what you can do on fiber. Backbone links are running at 10Gbps or even 40Gbps. Full duplex, so that is 20Gbps or 80Gbps of "marketing bandwidth". Compared to what, 22Mbps or 54Mbps half-duplex for your wireless? You aren't going to build a comparable backbone out of wireless links running at roughly 1/1000th of the speed. Physics pretty much guarantees that fiber links will always be faster than wireless.