Slashdot Mirror
Blackout Shows Net's Fragility
It doesn't come easy wrote to mention a ZDNet article discussing a recent outage between Level 3 Communications and Cogent Communication. A business feud inadvertently highlighted the fragility of the Internet's skeleton. From the article: "In theory, this kind of blackout is precisely the kind of problem the Internet was designed to withstand. The complicated, interlocking nature of networks means that data traffic is supposed to be able to find an alternate route to its destination, even if a critical link is broken. In practice, obscure contract disputes between the big network companies can make all these redundancies moot. At issue is a type of network connection called 'peering.' Most of the biggest network companies, such as AT&T, Sprint and MCI, as well as companies including Cogent and Level 3, strike "peering agreements" in which they agree to establish direct connections between their networks. "
What I don't get is why one of them would suddenly want the other to pay up. What's changed now, and why does the smaller company have to pay the big one's bills?
Am I missing something here?
Clever signature text goes here.
The pr0n industry was designed to find alternative routes of delivery in case of Internet outages.
Hey, I've found some interesting background info on this novel story here.
This statement popped up in some of my security readings. It's most "efficient" to have one path between two places, and it's most "efficient" to set up peering agreements to route packets. But these efficient measures can introduce single points of failure.
On a similar note, that's why there are 13 root DNS servers, and why most of us aren't supposed to use them. The DNS example though, is one where efficiency and robustness agree. It's more efficient, at least in terms of net bandwidth, to use a DNS server closer than the root servers.
The living have better things to do than to continue hating the dead.
http://www.gamergod.com/article_display.cfm?articl e_id=329
Good article on this situation here
This situation has adversely affected various users of both companies' services. The inability of Level 3 to handle this situation in a fair and equitable manner to the consumers has alienated many customers and will continue to do so until the current situation is remedied. At what point is it good customer service to discontinue services due to no fault of said consumer base? Market history shows us that the single worse thing a company can do is to arbitrarily allow influences beyond the control of consumers to negatively impact services, determined by consumers to be status quo, without any warning or notification. If left unresolved and unaddressed, the current situation could set dangerous precedents for internet users across the country by allowing service providers to instantly discontinue provided services at the moment they feel that the services they provide are not being adequately compensated for from outside companies.
On a side note, I was listening to Howard Stern (oh no!) this morning and he said that his Time Warner internet connection at home didn't work. Howard then called a tech guy to come and fix the problem, only for him to call a help desk to figure out what happened. The help desk didn't even know what was wrong. It sounds like Level 3 just pulled the plug and didn't notify ANYONE. Or maybe it was Cogent, the point is nobody outside of that dispute KNEW what was going on.
This sounds like a good way to alienate your customers and/or ruin your business model. But that is just my opinion.
He who knows best knows how little he knows. - Thomas Jefferson
I think since Wednesday.
-----BEGIN PGP SIGNATURE-----
12345
-----END PGP SIGNATURE-----
So, it appears a big part of the Internet traffic is controlled by large companies like Cogent or Level 3. No big surprise. I think this highlights the need for a new approach to connecting people together. I know there's been talk of wireless mesh networks where everybody is both an end point and a router. This would work in populated areas but I'm not sure how well it would work for "long haul" connections which is what the issue is here. Can anybody think of (or know of) any alternatives that gives control and power of the Internet back to the people who use it?
Bradley Holt
At issue is a type of network connection called 'peering.'
In other news, the RIAA announced they've stopped an extremely large P2P network.
You are more than the sum of what you consume. Desire is not an occupation.
Internet cannot route when your providers do not want you to communicate.
Nothing can protect you in this case.
If on the other hand there was a natural calamity and every one was trying to get you access
then you would get it. Like it happened during Katarina.
This is not a natural calamity.
The best option is to ditch your provider if they are not a monopoly and if they are lobby to your government to create multiple providers.
But for easy karma, just go get a +5 comment in the other thread, and repost it here without attribution.
Not that I would ever do such a thing...
cheers
The Internet will IMVHO always be quite fragile. While the design lends itself to robustness the reality is that there is only money for a few very big connections and therefore a disaster that affects one of these connections is going to cause wide spread outages.
Take, for instance, the connections running between Europe and America. I bet most of them run in almost exactly the same place on the sea bed because it's the cheapest / shortest path to take. A fairly localized geological disaster (at least in geological terms) could cut all the cables at once; or at least enough to make to difference.
If we wanted the network to be robust we would need to run cables up over the north pole and round the equator and probably stick in some satelite links as well. There just isn't money for that. People are willing to accept the risk that it might fail in extreme situations.
FWIW I think the problem is worse on the global scale than the country scale. I imagine most developed countries probably have enough redundancy in their own country. It's the interconnects between countries that are probably the biggest problem.
I used to have a better sig but it broke.
If you make the Top Tiers a government-controlled service, expect long term problems like censorship, taxation and regulations on sub-level tiers.
Neither company involved in this dispute wants to do t is. They need to work it out, or other companies will find a solution and take the customers.
If you're desperate to provide data to multiple top tiers, pay for a host that is connected to multiple backbones.
There is zero need to mandate anything. Let the free market provide and we'll be safer in the long run. Let government provide and we'll see a slowly creeping tyranny online.
You would only notice if you are on one of these two networks. I am personally on UUNet at home and MCI at work, and my server is on SpringLink (via Schlund, who I am not familiar with). As a result, all of my traffic is completely unaffected. Customers on a single-homed connection through Cogent, or through L3 cannot see other single homed customers on the other network. The rest of us don't know the difference. The dumb thing that this article points out is that both Cogent and L3 are refusing to route packets destined for each other through the rest of the internet (probably for fear of fucking up other peering agreements by dumping too much traffic on their other peers). I believe there was a comment in the previous thread about this issue saying that traffic in one direction could be routed, but that even return packets were being null-routed at some point, preventing any type of connection from being established.
--Brandon
As I understood the problem, redundancy wasn't an issue. Level 3 was actively filtering out request to Cogent, however they came in. The redundancy was working, but Level 3 was playing NetNanny and blacklisting all Cogent IPs.
The problem with web services is that they need for the internet to be completely secure and completely reliable. The internet of today is neither.
Physicians trying to use the internet to take care of critically ill patients are already experiencing this. Radiologists sitting home reading films are seeing this as well.
Is 100% on neccessary? Hell, VoIP is making money like crazy over this unstable network of ours.
My suggestion is to test with people that will understand the limitations of your service. Then get a little VC money to spread your servers out.
It's very true, and anyone can see how a few big companies basically make the net work in north america. Simply do traceroutes to various big web sites, and you'll notice the packets always go across the same networks. The biggest one seems to be alter.net (MCI), with others including Level3, above.net, AT&T and UUnet. Basically you remove any of these and the North American part of the Internet would be in chaos. The problem is because most ISPs do the same thing. They pick a primary provider, and get a backup one. The problem is they all pick the same few primary companies, and their backup links are much smaller pipes.
No, the internet was never designed with anonymity in mind, but it was designed to be a communications network that would not experience systemic problems when individual nodes and connections went down.
The only time peering should involve an ongoing exchange of money for bandwidth should be when a network is primarily serving as an intermediary between other networks, such as long haul or backbone networks.
But if most of the traffic from other networks is going to customers that are connected and already paying for your network's service then it makes no sense and is simply wrong for a network to start charging other network providers. It breaks the end to end communication model and is providing your customers with less than the service they are paying for. People pay for internet connectivity so they can transfer data between other users on the internet, not just the ones on your company's network.
If money exchanging hands is at all appropriate in this case it might be for the actual installation of routing equipment which establishes the physical connection between networks.
Be sure to let the UN know about that - this is surely something they'll want to take care of when they take "control of the Internet" away from the US. :)
All this crap about it showing weakness in the internet is uninformed bs. They didn't just stop peering, but they are actively blocking traffic from cogent. If Level3 had just stopped peering the traffic would reroute around the problem. The only time you will see problems is if your a cogent customer trying to get a single homed computer on level3's network. We are a cogent customer and an internap customer, and to get around the problem I just reouted traffic destined for level 3 networks over one of our internap t's. This solved the issue for us.
Privatization strikes again. You put the infrastructure into the hands of a few powerful people and this is what you will get. Those big power outages happened for the same reason. We aren't holding those in charge responsible. There is no redundancy when there is only one provider. They can cut you off and what are you going to do? Only community services and coops can provide the necessary robustness. But it seems to be more convenient to just hand it over to corporate pirates.
What?
As a customer who has had Cogent inflicted on us (when Verio sold all their domestic internet lines to Cogent), we've had nothing but pain and bumbling inefficiency from them for the last six months.
I contacted Cogent's "premium" help desk last night when I found that I was suddenly no longer able to get to our networks in Australia. The tech had no idea that his own company was in the middle of a huge peering battle with L3. I had to tell them!
As soon as all that pesky arctic ice melts away, it'll be cheap enough to run cable across the pole.
As a bonus, Santa's new underwater toy factory can tap into it.
Woo-hoo, faster email to Santa! Hope the jolly old elf doesn't discover online pr0n or he'll never get those presents made on time.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Unfortunately, that is not the Internet that we have today. In the original Internet, every router knew about every network connected to the Internet. Most networks had connectivity to many other networks. Discovery protocols allowed alternative routes to be discovered if one failed.
Today, we don't have a (mostly) fully connected net, we have ISPs who don't know anything about networks which they don't "own", only that certain IP prefixes need to be passed to ISP x, y or z.
This makes the infrastructure much more fragile than it was originally intended to be. We ended up with this for a few reasons. First, the wimpy routers in use at the time had limited memory available to hold the network maps. The answer chosen was to no longer attempt to hold a full world view, but to divide the world into regions, certain IP prefixes would "belong" to those regions, and all any router would need to know about was networks in its region, plus how to route traffic to other regions, who would take care of routing within the region. This led to "backbone" connections - high capacity links needed because all traffic between regions now didn't "diffuse" through the network, but was channeled into specific connections. It also set the scene to allow the net to be commercialised, those regional centers were obvious "choke points" that an enterprising company could own and pretty much dictate the pricing to lower level enterprises who would do the dirty work of dealing with end-users.
Slowly but sureley the Internet evolved into a system dependent upon a few companies with high-speed links between them - prime candidates BTW, as locations for government control to be imposed. The self-healing nature of the original Internet was lost because all traffic HAS to pass via the top level companies infrastructure and over their interconnect backbone connections.
The "self healing" Internet is long gone.
About 4 months ago I got a call from a sales critter at Cogent saying "We will knock 50% off of the price you are paying for your L3 connectivity if you drop them and come be our customer." I was kind of surprised at the boldness of this proposition because they were specifically targeting current L3 customers. I was even more surprised to find out from others that this sales pitch from Cogent was company wide. Of course this pissed off L3 and that was the start of this pissing contest.
Ah I see so by giving control over to the UN it will magically put in place all the hardware, software and correctly configure the web to never fail? Hopefully this statement was made to be joke because otherwise it doesn't make a bit of sense. I picture the web now as a 100,000 foot long giant slinky that someone has twisted into oblivion. I don't even know if the web can be fixed at this point...
News Reporters Make Tasty Polar Bear Treats!
If Level3 didn't want to peer anymore with Cogent. That's understandable, it wasn't an even exchange of traffic anymore. They could have done the right thing and simply stopped the peering. Insted, they have decided to be vendictive and filter any traffic to/from Cogent's IP range, even if the traffic is coming through some other ISPs network that Level3 still has peering or paid relationships with.
One again, the internet routers are perfectly able to find routes, Level3 is just deliberately trashing the packets before they get there. The Internet isn't 'unstable'. Any ISP can filter packets entering or leaving their network, and Level3 has decided to do so in an bad way. This just means Level3 customers should be pissed. This is nothing for anyone to get their panties in a bunch over except Cogent and Level3 customers, who's ISPs are being dicks.
http://www.internetpulse.net/
I'm not affiliated with them in any way, and I'm sure there are other similar sites, but I thought it was worth mentioning.
Damage: Level3 won't accept Cogent traffic. Horrible hack: tunnel BGP traffic to Level3 customer who masquerades requests as local traffic.
You don't need to masquerade anything, if you're connected to Level3 and Cogent, just configure your router to advertise your route to the Level3 network on the Cogent side and vice-versa.
Then watch your router melt under the hundreds of gigabits of traffic -- that you'll have to pay for both ways. Congratulation, you're the new peering agreement between Level3 and Cogent!
Knowing that it pissed Howard Stern off and wasted some of his time, I now feel much better about this outage.
At the fringes there are really two types of internet service offered: upstream and downstream. Most consumers (individuals) need a lot of downstream and very little upstream. They typically are sold assymetric service that is heavily biased in this direction. My cable connection, for example, gives me ~5Mbps down and 768kbps up. On the flip side are the content providers who typically need a lot of upstream bandwidth and less upstream bandwidth. ISPs have found that these customer are willing/able to pay quite a bit more for their internet connections. Therefore, the law of supply and demand has increased the cost of connections with higher upstream capacity.
Several levels up the ISP heirarchy, however, there are mostly only symmetric lines (T3, OCx, ...) providing equal upstream and downstream bandwith. In order to maximize the use of this bandwidth, many providers try to balance the number of content providers with content consumers in order to use the upstream and downstream capacity equally. In theory, this usage should be well balanced by the time it reaches the Teir 1 providers.
The problem we are having right now is caused by Cogent not subscribing to that business model. They have found that the cost to support content consumers is much higher than the cost to support providers. (If for no other reason than there are far more of them.) So, their business model skews heavily towards the provider customers, reducing their operational costs. This, in turn, means that they are able to offer lower costs to those content providers -- in many cases undercutting the other big service providers such as Level 3
This, of course, makes the other providers unhappy because it cuts into their high-yield business. So, occasionally, one of them demands compensation for "transit" instead of providing free peering. They do this because they feel (rightly IMO) that Cogent is able to make more money on these high paying content providers by using an asset owned by the other service providers -- the online customer/consumer base. Basically, Level 3 is telling Cogent that because Cogent is making money by using that virtual asset owned by Level 3, Cogent owes Level 3 some sort of compensation. It is worth noting that several other Teir 1 providers already take this approach with Cogent and Cogent is forced to pay for "transit" service to those providers' customers.
As long as all the Teir 1 providers cooperate, the system works reasonably well. However, in this case, Cogent is trying to take advantage of that informal cooperation to make some extra money. So, they are being capatalists. In this case, capatalism is at odds with cooperation and the system is not working well.
Many people are calling for government regulation to prevent this sort of situation. I expect this to cause some major problems. The issue could be resolved if all the Teir 1 providers would realize that there is a different market value for ingress and outgress traffic. In a free market, I expect that the ingress traffic (corresponding to upstream traffic of content providers from the lower levels) would have substantially more value than the outgress traffic (downstream traffic to consumers). The outgress traffic might even have negative value (meaning that a service provider would charge to take care of it). In the case that two peers balance their traffic well (the ideal cooperative solution) no money needs to change hands. In the other cases (like this one) the ISP with excess outgress usage should probably be charging the one with excess ingress.
Unfortunately, there is no fluidity to the system between the true market (the upstream and downstream bandwidth consumers) and the core market (the Teir 1 providers). If there were, Level 3 could justify their demand for more money based on the value of the traffic they were accepting from lower down the food chain.
Wow, that's a nice idea. That'll mean that all I have to do is run a bit of Ethernet into a peering point and I'll get free connections to all the tier ones. Fabulous.
Oh - hang on, if someone else runs a bit of Ethernet in, do I have to connect to them? Damn.
If either Level3 or Cogent was buying a "default" service from a third party, their customers wouldn't have a problem. The moment the peering connection was cut the lower-priority BGP routes from the third party would have taken over and their traffic would have gone through the third-party link.
The reason these two jokers are having this problem is that they made a business decision to only move traffic with reciprocal peering and then failed to keep that peering alive. That's because they're both cheap-ass bastards; peering costs a heck of a lot less than buying transit.
Go buy from someone else who who isn't a cheap-ass. Someone who buys transit for anything they can't peer. You won't have a problem.
The only lesson here is that most time honored of lessons: you get what you pay for.
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
The pitch is even better now. If you are an L3 transit customer, Cogent will give you free service for a year. For L3's current customers this solves the immediate problem, and they wind up multi-homed, so they don't get bitten by this in the future.
Si vis pacem, para bellum
The only thing more annoying than a Libertarian is an (un|mis)informed Libertarian
The problem isn't soley with the business arrangements between the "big providers" - oh, certain, that does have impact, but the internet would be as robust as ever, if every participant on it could be a peer.
This is how the network was meant to be, a mesh comprised of stupid interconnects and smart nodes. Every node on the internet, from the largest colo to the smallest wireless handheld, should have the ability to be a true peer on the internet. In practice, this isn't really possible, but imagine a mesh network with a distributed p2p DNS system which many people could run if they wanted to - if only a fraction were running it, and were distributed enough, such outages might not occur (the traffic could continue to be routed, albeit at a slower pace).
Everyone should be able to be a peer on the network, everyone should be able to get at least one static IP, everyone should be able to run their own server(s) if they want to. Right now, the only way you can do it is by paying huge amounts of $$$ so you can get a garden hose instead of a straw. I am not saying access to the internet should be or could be free, but peering should be a natural right of being a part of the internet, not something you have to pay extra (a LOT extra) for.
Reason is the Path to God - Anon
I had a friend on Roadrunner who complained he couldn't connect to many sites. I think he happened to know that they used Level 3. Is there a way to determine what backbone your ISP or a particular site uses?
Fetch Text URL - Firefox Extension
HOW PATHETIC! Two major ISP are willing to piss off thousands of people just because they've thrown their toys out of the pram. Back at school they'd get told to shut up and get along, now it'll become a legal action. GET A FUCKING GRIP!!! I thought the world was too sensible for this kind of thing. I was wrong.
You feel sleepy. Close your eyes. The opinions stated above are yours. You cannot imagine why you ever felt otherwise.
*Sigh*. Why do you spew nonsense if you actually have not even found out how a clue looks like, not to mention ever aquired one ?
So you claim there are no Internet Exchange Points ?
pray tell, what is this thing ? Or that one, not to mention the middle one.
Oh, and what do you think those Guys do for a living ?
Nobody expects you to be a fucking genius or know everything. But why are some folks constantly touting stupid nonsense instead of keeping their mouths shut and learning something ?
Cogent COULD route around the damage - if they wanted to, but they don't.
If the peering point had been taken out by a bomb, the re-routing would have been performed in fairly short order. However, this is not the case here.
Level3 think that Cogent is taking the piss and is not a real peer. Level3 want Cogent to buy transit to reach Level3, either directly from them (or from someone else) because at the moment the peering is very lopsided, and costing Level3 a bucketload of money and giving Cogent a boatload of free bandwidth.
Cogent on the other hand doesn't want to pay for transit to Level3.
Right now, Cogent could route all their traffic for Level3 over transit they pay for. They don't want to do that because it won't force Level3 back into the peering agreement. So what they do is leave the link severed and do not re-route so that Level3 customers cannot get to sites hosted by Cogent. This means Level3 customers will grumble at Level3. Additionally, they offer a year's free transit to single homed Level3 customers just to raise the brinkmanship with Level3 a notch higher. Basically it's war between L3 and Cogent.
If Cogent re-routes their traffic, they are defeated and L3 will never re-peer. What Cogent are hoping is that enough angry customers on the L3 end will whine at L3 so L3 will be forced to re-peer.
For the rest of us in the peanut gallery (i.e. those of us who aren't single homed customers of Cogent or Level3) we can just watch the fun and games and throw peanut shells at the squabbling combatants because we don't see any black hole at all.
Oolite: Elite-like game. For Mac, Linux and Windows
FYI, smaller ISPs pay larger ISPs for bandwidth all the time. The larger ISPs have huge costs. Switches costing hundreds of thousands of dollars, filled with a bunch of cards in it that each cost hundreds of thousands. Lots of them. Lots of fiber and other costs. It gets real easy to have billions invested just in hardware. They offset those costs in part by selling bandwidth to smaller ISPs. That's the way the net works.
Try telling some small ISP that they should stop paying their upstream provider. That the upstream provider should give them bandwidth free so that the larger ISPs customers can access websites hosted by the smaller ISP. They will tell you you are living in a dream world. That's not the way the net works.
That's exactly what it looks like. And yes, the routers are set to find another path. The problem is when it finds a new path through some 3rd or 4th ISP to the Level3 network, as soon as the Level3 router sees the packet originated from a Cogent IP address, it null routes it. That's not a problem with fragility of the net, it's Level3 behaving badly. (Note: Cogent should have ponied up money for traffic to a larger provider to avoid this mess in the first place. There are no good guys involved in this.)
Is this why I can't read userfriendly or Something Positive this morning? Or is it just some weird coincidental webcomic blackout?
the problem has been solved. I can ping level3 from cogent and i have one connection. I don't know yet who flinched first......
The availability grid for the past 4 hours shows ~40% and the grid for the past 1 hour shows 100%. As noted by "Cally" below, I honestly have no idea how exactly this grid has been generated (hence my original disclaimer) but this certainly seems to indicate, from a practical standpoint, that the L3/Cogent issue has been very recently resolved. Indeed, from my (single-homed) L3 server I can now traceroute directly to a (single-homed) Cogent host.
Peering arrangements are different. Two networks that have a lot of traffic for each other will set up direct connections, split the direct costs of the connections, and not charge for accepting packets from the other carrier. But they'll only advertise the routes for their *own* customers. If two small ISPs peer with each other, typically they're each also buying transit service from big ISPs, but it's cheaper for them to dedicate a connection or put bits on a public peering point like MAE-West than to both pay their upstream ISPs.
The biggest ISPs in the US are called "Tier 1" ISPs, and they all peer with each other rather than buying transit, though they might buy transit for international connections, if they can't get the other side to buy transit from them. It seems flaky, but it makes business sense, or at least it did for a while. In some sense, being big enough that all the other Tier 1s will peer with you is what defines Tier 1, and aside from technical issues, it's a marketing thing - "See, we're one of the big players!" Peering and Transit don't mix very well - you either connect to a given carrier by peering, or by transit, or else you spend a long time hammering out custom arrangements about exactly which routes you'll accept and tweaking routing tables.
Cogent is a Wannabe-Tier-1. Their main business model is to put fiber into big multi-tenant office buildings and sell everybody 100-meg Ethernet for about the price other carriers charge for one or two T1s. If I were a customer, I wouldn't expect there to be enough upstream to really get that much bandwidth all the time, but I'd expect to get more than a T1 all the time, and a lot more than a T1 almost all the time. Level 3 has apparently decided they're not getting enough value out of the relationship (i.e. not sending Cogent enough packets to make it worth their while) to keep peering, and wants Cogent to either pay them for service or get transit from somebody else. They gave them about 50 days to make other arrangements, but Cogent decided to play chicken with them.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks