Slashdot Mirror
Blackout Shows Net's Fragility
It doesn't come easy wrote to mention a ZDNet article discussing a recent outage between Level 3 Communications and Cogent Communication. A business feud inadvertently highlighted the fragility of the Internet's skeleton. From the article: "In theory, this kind of blackout is precisely the kind of problem the Internet was designed to withstand. The complicated, interlocking nature of networks means that data traffic is supposed to be able to find an alternate route to its destination, even if a critical link is broken. In practice, obscure contract disputes between the big network companies can make all these redundancies moot. At issue is a type of network connection called 'peering.' Most of the biggest network companies, such as AT&T, Sprint and MCI, as well as companies including Cogent and Level 3, strike "peering agreements" in which they agree to establish direct connections between their networks. "
What I don't get is why one of them would suddenly want the other to pay up. What's changed now, and why does the smaller company have to pay the big one's bills?
Am I missing something here?
Clever signature text goes here.
A solution can be to make it mandatory that all connecters to any internet exchange point peer to each other. all of them...
or use satellite technology
The pr0n industry was designed to find alternative routes of delivery in case of Internet outages.
yesterday? I swear the entire net came to a crawl yesterday afternoon.
I've been hearing about this for days, but haven't noticed any slow links, or down sites... anywhere. I hit lots of major sites a day, minor too. I have a few webservers in various parts of the world... and I didn't have a single blip anywhere.
What exactly got blacked out? I know the issue, but just never saw it's effect anywhere.
Hey, I've found some interesting background info on this novel story here.
This statement popped up in some of my security readings. It's most "efficient" to have one path between two places, and it's most "efficient" to set up peering agreements to route packets. But these efficient measures can introduce single points of failure.
On a similar note, that's why there are 13 root DNS servers, and why most of us aren't supposed to use them. The DNS example though, is one where efficiency and robustness agree. It's more efficient, at least in terms of net bandwidth, to use a DNS server closer than the root servers.
The living have better things to do than to continue hating the dead.
fast as fast can be. you'll never catch me.
http://www.gamergod.com/article_display.cfm?articl e_id=329
Good article on this situation here
This situation has adversely affected various users of both companies' services. The inability of Level 3 to handle this situation in a fair and equitable manner to the consumers has alienated many customers and will continue to do so until the current situation is remedied. At what point is it good customer service to discontinue services due to no fault of said consumer base? Market history shows us that the single worse thing a company can do is to arbitrarily allow influences beyond the control of consumers to negatively impact services, determined by consumers to be status quo, without any warning or notification. If left unresolved and unaddressed, the current situation could set dangerous precedents for internet users across the country by allowing service providers to instantly discontinue provided services at the moment they feel that the services they provide are not being adequately compensated for from outside companies.
On a side note, I was listening to Howard Stern (oh no!) this morning and he said that his Time Warner internet connection at home didn't work. Howard then called a tech guy to come and fix the problem, only for him to call a help desk to figure out what happened. The help desk didn't even know what was wrong. It sounds like Level 3 just pulled the plug and didn't notify ANYONE. Or maybe it was Cogent, the point is nobody outside of that dispute KNEW what was going on.
This sounds like a good way to alienate your customers and/or ruin your business model. But that is just my opinion.
He who knows best knows how little he knows. - Thomas Jefferson
In theory, this kind of blackout is precisely the kind of problem the Internet was designed to withstand
Riiight...just like the Internet keeps everyone anonymous.
So, it appears a big part of the Internet traffic is controlled by large companies like Cogent or Level 3. No big surprise. I think this highlights the need for a new approach to connecting people together. I know there's been talk of wireless mesh networks where everybody is both an end point and a router. This would work in populated areas but I'm not sure how well it would work for "long haul" connections which is what the issue is here. Can anybody think of (or know of) any alternatives that gives control and power of the Internet back to the people who use it?
Bradley Holt
You answered your own question. It's just not possible without servers in different physical locations. You have to account for more then just network trouble, there's always natural disasters to consider.
Then again, if it's a hobby project out of your own pocket iut doesn't actually have to be 24/7. It's not like you're signing a contract to guarantee that kind of uptime.
At issue is a type of network connection called 'peering.'
In other news, the RIAA announced they've stopped an extremely large P2P network.
You are more than the sum of what you consume. Desire is not an occupation.
The RIAA's lawyers are currently working on documents to stop this new threat to the entertainment industry!
This is why you make sure that your ISP/provider peers with mutliple "Tier 1" providers.
We peer through 3 major providers, and our users didn't see an outage because of it.
This was reported two days ago. Don't believe the hype!
☠
Internet cannot route when your providers do not want you to communicate.
Nothing can protect you in this case.
If on the other hand there was a natural calamity and every one was trying to get you access
then you would get it. Like it happened during Katarina.
This is not a natural calamity.
The best option is to ditch your provider if they are not a monopoly and if they are lobby to your government to create multiple providers.
Everyone here at the company I work for was crying their internet was down because the msn homepage and some microsoft page were down and they didn't think to try another site. When your job as IT becomes to tell people to try a different website then the homepage then there's a problem.
Didn't notice anything different in the UK. Some of my torrents had less peers than usual. Shrug.
But for easy karma, just go get a +5 comment in the other thread, and repost it here without attribution.
Not that I would ever do such a thing...
cheers
The Internet will IMVHO always be quite fragile. While the design lends itself to robustness the reality is that there is only money for a few very big connections and therefore a disaster that affects one of these connections is going to cause wide spread outages.
Take, for instance, the connections running between Europe and America. I bet most of them run in almost exactly the same place on the sea bed because it's the cheapest / shortest path to take. A fairly localized geological disaster (at least in geological terms) could cut all the cables at once; or at least enough to make to difference.
If we wanted the network to be robust we would need to run cables up over the north pole and round the equator and probably stick in some satelite links as well. There just isn't money for that. People are willing to accept the risk that it might fail in extreme situations.
FWIW I think the problem is worse on the global scale than the country scale. I imagine most developed countries probably have enough redundancy in their own country. It's the interconnects between countries that are probably the biggest problem.
I used to have a better sig but it broke.
If you make the Top Tiers a government-controlled service, expect long term problems like censorship, taxation and regulations on sub-level tiers.
Neither company involved in this dispute wants to do t is. They need to work it out, or other companies will find a solution and take the customers.
If you're desperate to provide data to multiple top tiers, pay for a host that is connected to multiple backbones.
There is zero need to mandate anything. Let the free market provide and we'll be safer in the long run. Let government provide and we'll see a slowly creeping tyranny online.
As I understood the problem, redundancy wasn't an issue. Level 3 was actively filtering out request to Cogent, however they came in. The redundancy was working, but Level 3 was playing NetNanny and blacklisting all Cogent IPs.
I didn't even know anything about this until Howard Stern started talking about it this morning on his show. He was pretty pissed about not having email for 2 days. He also said that nobody informed him that it wasn't on his end. He was going nuts trying to get his computer fixed, calling every computer guy that he knew to come look at it.
DEAD DEAD DEAD DELETE ME
I don't believe in large government but this strikes me as one of those things that government is good for.
Why wouldn't it be a good thing for some governmental agency to regulate the physical location of various installations? It seems to me that many providers use the same colocations to house their equipment. It would seem smart that there would be some regulation to prevent all the Internet eggs in one basket.
Create several more physical IXP's that are located in geographically diverse areas with redundant connections. Then regulate that only a limited number of companies could colocate together within a certain number of IXP's.
This could prevent one companies "disagreement" with another from effecting the traffic being routed to an alternative link.
Does this make sense?
"What the hell is an aluminum falcon?"
Bandwidth isn't (and never will be) truly free as long as the hardware and admin labor has a cost. But if we seek way to deliver the most packets at the least cost, then market forces will drive the price down. The total system would route around both damage and inefficient (= high cost) parts of the network.
Two wrongs don't make a right, but three lefts do.
The problem with web services is that they need for the internet to be completely secure and completely reliable. The internet of today is neither.
Physicians trying to use the internet to take care of critically ill patients are already experiencing this. Radiologists sitting home reading films are seeing this as well.
Is 100% on neccessary? Hell, VoIP is making money like crazy over this unstable network of ours.
My suggestion is to test with people that will understand the limitations of your service. Then get a little VC money to spread your servers out.
It's very true, and anyone can see how a few big companies basically make the net work in north america. Simply do traceroutes to various big web sites, and you'll notice the packets always go across the same networks. The biggest one seems to be alter.net (MCI), with others including Level3, above.net, AT&T and UUnet. Basically you remove any of these and the North American part of the Internet would be in chaos. The problem is because most ISPs do the same thing. They pick a primary provider, and get a backup one. The problem is they all pick the same few primary companies, and their backup links are much smaller pipes.
Time Warner Cable in upstate NY was unable to connect to several sites. SETI and Newsfeeds notably on Wed.
If the traffic is about even both ways, the peering aggrements are made with no cash exchange. If it's uneven, the network not bearing its fair share of traffic usually has to pony up some cash as part of the "peer" aggreement. If things don't turn out as expected, the network carrying an unfair burden will usually back out or renegotiate the peer aggreement. You usually don't find out what the actual network traffic is until you start peering.
The only time peering should involve an ongoing exchange of money for bandwidth should be when a network is primarily serving as an intermediary between other networks, such as long haul or backbone networks.
But if most of the traffic from other networks is going to customers that are connected and already paying for your network's service then it makes no sense and is simply wrong for a network to start charging other network providers. It breaks the end to end communication model and is providing your customers with less than the service they are paying for. People pay for internet connectivity so they can transfer data between other users on the internet, not just the ones on your company's network.
If money exchanging hands is at all appropriate in this case it might be for the actual installation of routing equipment which establishes the physical connection between networks.
No it can't. Think before you post.
All this crap about it showing weakness in the internet is uninformed bs. They didn't just stop peering, but they are actively blocking traffic from cogent. If Level3 had just stopped peering the traffic would reroute around the problem. The only time you will see problems is if your a cogent customer trying to get a single homed computer on level3's network. We are a cogent customer and an internap customer, and to get around the problem I just reouted traffic destined for level 3 networks over one of our internap t's. This solved the issue for us.
Privatization strikes again. You put the infrastructure into the hands of a few powerful people and this is what you will get. Those big power outages happened for the same reason. We aren't holding those in charge responsible. There is no redundancy when there is only one provider. They can cut you off and what are you going to do? Only community services and coops can provide the necessary robustness. But it seems to be more convenient to just hand it over to corporate pirates.
What?
The complicated, interlocking nature of networks
But when you choose to have a single critical link you don't have an interlocking web of connections.
even if a critical link is broken.
If it was a web there would be no critical links to break. The problem is that for various (technical/economic) reasons there is a backbone (or series of backbones) it isn't really an interlocking web.
when you hear Howard Stern complain about not getting emails and whatnot.
The Doormat
If you're not outraged, then you're not paying attention.
Back in the day in there was no 'peering' contract. It was all techies helping techies and peering arrangements would happen with a handshake over some good sushi. You take my routes, I'll take yours. Then all the lawyers got involved, its been a nightmare ever since.
Here is a solution, set up your wireless router to bridge with a friend or someone you know. Keep this up untill you have a realy big WAN. Hell after a while you could have "INTERNET 2.5 - Wow that latency is a BITCH. But the man can't keep me down!"
Hey, but you Euro/UN types go ahead an slap down your new root servers wherever you want....that certainly won't screw everything up.
/ 06/1241227&tid=95&tid=219
http://politics.slashdot.org/article.pl?sid=05/10
-Styopa
If I remember correctly, shouldn't network protocols be flexible enough to find another path to the server? Let's say that you are on one network and the desired server is on another network that have their direct connection killed. Can't you go and find an alternate route through a peer that is connected to the both of you? It would be slower but shouldn't this happen?
TANSTAAFL
To the moron mod, that was FUNNY! I'm pretty sure the author meant it as a joke, because I laughed when I read it. Flamebait? Sheesh.
If you were ISP in distress, wouldn't you find a way to circumvent this downed peering point? I'm sure other Tier 1s have jumped in and offered transit for a modest sum. If I were an evil ISP and I wanted to make sure that didn't happen, I'd blackhole all the netblocks (or any ASPATH with his AS) from the guy I wanted moolah from until he forked over the funds. Unless I'm missing something, that's the explanation for why networks who are purchasing transit from Cogent/L3 are having trouble.
As a customer who has had Cogent inflicted on us (when Verio sold all their domestic internet lines to Cogent), we've had nothing but pain and bumbling inefficiency from them for the last six months.
I contacted Cogent's "premium" help desk last night when I found that I was suddenly no longer able to get to our networks in Australia. The tech had no idea that his own company was in the middle of a huge peering battle with L3. I had to tell them!
nuttin here
As soon as all that pesky arctic ice melts away, it'll be cheap enough to run cable across the pole.
As a bonus, Santa's new underwater toy factory can tap into it.
Woo-hoo, faster email to Santa! Hope the jolly old elf doesn't discover online pr0n or he'll never get those presents made on time.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Maybe a :) would have made it easier for you to understand.
This "homemade" MP3 is quite fitting in this situation. A must have for all concerned. :-) http://www.wps.com.au/music/Rumors.mp3
Unfortunately, that is not the Internet that we have today. In the original Internet, every router knew about every network connected to the Internet. Most networks had connectivity to many other networks. Discovery protocols allowed alternative routes to be discovered if one failed.
Today, we don't have a (mostly) fully connected net, we have ISPs who don't know anything about networks which they don't "own", only that certain IP prefixes need to be passed to ISP x, y or z.
This makes the infrastructure much more fragile than it was originally intended to be. We ended up with this for a few reasons. First, the wimpy routers in use at the time had limited memory available to hold the network maps. The answer chosen was to no longer attempt to hold a full world view, but to divide the world into regions, certain IP prefixes would "belong" to those regions, and all any router would need to know about was networks in its region, plus how to route traffic to other regions, who would take care of routing within the region. This led to "backbone" connections - high capacity links needed because all traffic between regions now didn't "diffuse" through the network, but was channeled into specific connections. It also set the scene to allow the net to be commercialised, those regional centers were obvious "choke points" that an enterprising company could own and pretty much dictate the pricing to lower level enterprises who would do the dirty work of dealing with end-users.
Slowly but sureley the Internet evolved into a system dependent upon a few companies with high-speed links between them - prime candidates BTW, as locations for government control to be imposed. The self-healing nature of the original Internet was lost because all traffic HAS to pass via the top level companies infrastructure and over their interconnect backbone connections.
The "self healing" Internet is long gone.
About 4 months ago I got a call from a sales critter at Cogent saying "We will knock 50% off of the price you are paying for your L3 connectivity if you drop them and come be our customer." I was kind of surprised at the boldness of this proposition because they were specifically targeting current L3 customers. I was even more surprised to find out from others that this sales pitch from Cogent was company wide. Of course this pissed off L3 and that was the start of this pissing contest.
Cogent is dumping a lot of data on Level 3 in a peering arangment. Which means that the data is destined for customers on Level3's network. Since Level3 makes money from charging customers for bandwidth, this is a benefit. More data to/from their customers means more money. Level3 however, wants to make money on the bandwidth twice for each customer (once by charging cogent to talk to Level3 customers and again from Level3 customers).
The only way this would be a problem is if cogent is using the connection as a "transit" connection, routing data for other networks through Level3 which doesn't seem to be the case.
OK, I have a crazy idea. I am NOT a net admin and am largely blue skying here.
Damage: Level3 won't accept Cogent traffic.
Horrible hack: tunnel BGP traffic to Level3 customer who masquerades requests as local traffic.
Yeah, the real solution is tier 2 folks having more peerings, but as a nasty workaround is that hack feasible?
Can you tunnel BGP traffic in TCP or ssh or something?
-l
Help cure AIDS, cancer, and more. Donate your unused computer time to worldcommunitygrid.org. Join Team Slashdot!
nobody noticed because nobody uses cogent or level three.
Ah I see so by giving control over to the UN it will magically put in place all the hardware, software and correctly configure the web to never fail? Hopefully this statement was made to be joke because otherwise it doesn't make a bit of sense. I picture the web now as a 100,000 foot long giant slinky that someone has twisted into oblivion. I don't even know if the web can be fixed at this point...
News Reporters Make Tasty Polar Bear Treats!
Typically, when you're dealing with peering, it's the amount of traffic that you're pushing on the other guy, because people on your network want to connect to places on their network. (or use your network to get to another network that you peer with)
So, when a mom & pop ISP connects to one of the big guys, there's very little of interest on their network as an endpoint, and they probably don't advertise better routes than the tier1 already has, so they have to pay for the priviledge to be connected.
Now, in the case of Cogent, if I remember my ISPs correctly (I've not worked for an ISP for 6 years now), they were PSI, which merged with a few other networks -- the problem was that the networks weren't well connected, and so they were accused of what was known as 'hot potato routing' -- trying to get the traffic off their network as quickly as possible, even if it wasn't the shortest number of hops to the destination.
Some ISPs will also pad the routes that they advertise, so that they basically tell other people 'we'll take our own traffic, but we don't want to route other people's traffic through us'. (or it may be that their internal networks are so badly interconnected, that it really a whole lot of hops -- I mean, what good does it do to advertize that you connect to the moon, if you're going to route everyone by way of Jupiter?)
I'm guessing that Level 1 was montioring the traffic flowing between themselves and Cogent, and noticed that something seemed significantly amiss, suggesting that Cogent was taking advantage of their services.
I don't know the specifics of this case, however. so this is all conjecture, based on 6+ year old knowledge of ISPs. (and well, I can't find what happened to Boardwatch, so I can't pull up nice maps of the big players)
Build it, and they will come^Hplain.
This is it. We're coming to take over the net from you guy's. Your companies are obviously not able to run it reliably. All non US ISP's should be required to resolve DNS to un.un.
Because my ping times were crap yesterday to the servers I normally play games on.
http://www.internetpulse.net/
I'm not affiliated with them in any way, and I'm sure there are other similar sites, but I thought it was worth mentioning.
The Internet Health Report shows the interruption, and I noticed there is also the "Pending Assingment" as mentioned in these articles. I'm curious as to who this is.
- what is the definition of simultanagnosia?! I've been meaning to look it up!
I can't access my web host, textdrive.com, nor wikipedia.com from my resnet connection here at Waterloo University. Apparently they're relying exclusively on Cogent. It's quite frustrating because I can't can't do much. I have, however, found some anonymous browsing services and HTTP proxies that work well enough.
I've been in contact with the Information Services Technology here and it looks like some more important services are being distrupted:
On one hand it's good for me that official services are affected, as it should boost priority.I'm sitting off an ISP that plugs into Level-3 so this blackout is definately causing me problems. I can't get to Penny Arcade or Megatokyo (clearly this is a crisis). So I'm using Tor to get around it. Yeah, it's really slow at times, but it works.
( And as impossible as it sounds, there are more important sites I can't get to either, like the support site for a couple bits of software we use rather heavily at work. )
btw, this problem has already been resolved
6 verio.dfw03.atlas.cogentco.com (154.54.10.30) 7.682 ms 7.723 ms 7.283 ms
7 pop1-dls-p3-2.atdn.net (66.185.133.93) 7.515 ms 6.929 ms 6.793 ms
8 bb1-dls-p0-0.atdn.net (66.185.133.80) 8.176 ms 66.393 ms 13.965 ms
9 pop2-dls-p0-0.atdn.net (66.185.133.97) 7.083 ms 6.703 ms 7.292 ms
10 rr-houston.atdn.net (66.185.132.18) 7.571 ms 7.125 ms 7.943 ms
or at least it's been fixed for roadrunner cogent in Texas
The dispute between Cogent and Level 3 has been extended by Cogent itself. They are apparently blocking access to sites within their network based on the origin address. From a host on charter.com a traceroute ends at p12-0.core02.ord01.atlas.cogentco.com but from a host on ameritech.com the traceroute continues all the way to 66.28.205.104.
The Cogent management appears to be deliberately creating a DoS situation for publicity purposes.
Before I 'troll' more hateful comments, append a smiley to the first comment. It was a joke people, geez!!
Watch your investments "fail", having run off with all your money.
The real key to financial independence appears to be choosing your parents wisely.
IMHO this is unfortunate, it can give rise to phenomena like "Paris Hilton."
The living have better things to do than to continue hating the dead.
At the fringes there are really two types of internet service offered: upstream and downstream. Most consumers (individuals) need a lot of downstream and very little upstream. They typically are sold assymetric service that is heavily biased in this direction. My cable connection, for example, gives me ~5Mbps down and 768kbps up. On the flip side are the content providers who typically need a lot of upstream bandwidth and less upstream bandwidth. ISPs have found that these customer are willing/able to pay quite a bit more for their internet connections. Therefore, the law of supply and demand has increased the cost of connections with higher upstream capacity.
Several levels up the ISP heirarchy, however, there are mostly only symmetric lines (T3, OCx, ...) providing equal upstream and downstream bandwith. In order to maximize the use of this bandwidth, many providers try to balance the number of content providers with content consumers in order to use the upstream and downstream capacity equally. In theory, this usage should be well balanced by the time it reaches the Teir 1 providers.
The problem we are having right now is caused by Cogent not subscribing to that business model. They have found that the cost to support content consumers is much higher than the cost to support providers. (If for no other reason than there are far more of them.) So, their business model skews heavily towards the provider customers, reducing their operational costs. This, in turn, means that they are able to offer lower costs to those content providers -- in many cases undercutting the other big service providers such as Level 3
This, of course, makes the other providers unhappy because it cuts into their high-yield business. So, occasionally, one of them demands compensation for "transit" instead of providing free peering. They do this because they feel (rightly IMO) that Cogent is able to make more money on these high paying content providers by using an asset owned by the other service providers -- the online customer/consumer base. Basically, Level 3 is telling Cogent that because Cogent is making money by using that virtual asset owned by Level 3, Cogent owes Level 3 some sort of compensation. It is worth noting that several other Teir 1 providers already take this approach with Cogent and Cogent is forced to pay for "transit" service to those providers' customers.
As long as all the Teir 1 providers cooperate, the system works reasonably well. However, in this case, Cogent is trying to take advantage of that informal cooperation to make some extra money. So, they are being capatalists. In this case, capatalism is at odds with cooperation and the system is not working well.
Many people are calling for government regulation to prevent this sort of situation. I expect this to cause some major problems. The issue could be resolved if all the Teir 1 providers would realize that there is a different market value for ingress and outgress traffic. In a free market, I expect that the ingress traffic (corresponding to upstream traffic of content providers from the lower levels) would have substantially more value than the outgress traffic (downstream traffic to consumers). The outgress traffic might even have negative value (meaning that a service provider would charge to take care of it). In the case that two peers balance their traffic well (the ideal cooperative solution) no money needs to change hands. In the other cases (like this one) the ISP with excess outgress usage should probably be charging the one with excess ingress.
Unfortunately, there is no fluidity to the system between the true market (the upstream and downstream bandwidth consumers) and the core market (the Teir 1 providers). If there were, Level 3 could justify their demand for more money based on the value of the traffic they were accepting from lower down the food chain.
I have servers on the backbone of a European Telco (which I don t name because they have shown displease at my complaints) which appears to receive traffic from the US via Cogentco. While normal web navigation isn t affected, since early august all ftp traffic directed to our servers going tru Cogentco (to my limited knowledge it is unfortunately ALL traffic coming from the US) is capped at speeds ranging from 10 to 24 kB, which therefore has been the maximum upload speed to our servers for over 2 months. This Euro Telco isn t Level3, and the trouble isn t limited to the past few days. Maybe there s more than the story of a temporary peering breakdown between Level3 and Cogentco. IS anybody experiencing bandwidth problems on US Europe traffic involving Cogentco ?
Finest carrier? The same that censors internet access for large swaths of a continent based on moral concerns of some organisation in one country? (and I'm not talking about China - it's Europe and Germany)
Point taken. Thanks.
-l
Help cure AIDS, cancer, and more. Donate your unused computer time to worldcommunitygrid.org. Join Team Slashdot!
If either Level3 or Cogent was buying a "default" service from a third party, their customers wouldn't have a problem. The moment the peering connection was cut the lower-priority BGP routes from the third party would have taken over and their traffic would have gone through the third-party link.
The reason these two jokers are having this problem is that they made a business decision to only move traffic with reciprocal peering and then failed to keep that peering alive. That's because they're both cheap-ass bastards; peering costs a heck of a lot less than buying transit.
Go buy from someone else who who isn't a cheap-ass. Someone who buys transit for anything they can't peer. You won't have a problem.
The only lesson here is that most time honored of lessons: you get what you pay for.
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
There is a huge need to mandate.
Of course that need is generated by cmpanies that want the internet 'under control'.
So if you sit back and 'let the market decide' those who controll the mark will. It won't be for you.
The Kruger Dunning explains most post on
The pitch is even better now. If you are an L3 transit customer, Cogent will give you free service for a year. For L3's current customers this solves the immediate problem, and they wind up multi-homed, so they don't get bitten by this in the future.
Si vis pacem, para bellum
The only thing more annoying than a Libertarian is an (un|mis)informed Libertarian
If Level 3 doesn't want to peer with Cogent, that's fine. If they export incorrect routing information to gain a business advantage, that's fraud.
The problem isn't soley with the business arrangements between the "big providers" - oh, certain, that does have impact, but the internet would be as robust as ever, if every participant on it could be a peer.
This is how the network was meant to be, a mesh comprised of stupid interconnects and smart nodes. Every node on the internet, from the largest colo to the smallest wireless handheld, should have the ability to be a true peer on the internet. In practice, this isn't really possible, but imagine a mesh network with a distributed p2p DNS system which many people could run if they wanted to - if only a fraction were running it, and were distributed enough, such outages might not occur (the traffic could continue to be routed, albeit at a slower pace).
Everyone should be able to be a peer on the network, everyone should be able to get at least one static IP, everyone should be able to run their own server(s) if they want to. Right now, the only way you can do it is by paying huge amounts of $$$ so you can get a garden hose instead of a straw. I am not saying access to the internet should be or could be free, but peering should be a natural right of being a part of the internet, not something you have to pay extra (a LOT extra) for.
Reason is the Path to God - Anon
I had a friend on Roadrunner who complained he couldn't connect to many sites. I think he happened to know that they used Level 3. Is there a way to determine what backbone your ISP or a particular site uses?
Fetch Text URL - Firefox Extension
I can see both sides of the argument. Level 3 says they are losing money and Cogent says "hey we are sharing! WTF".
This has got stuff screwed up pretty bad though. The mesh topology I thought that existed apparently does not. I'm assuming DNS is all whacked out too. My company belongs on the Cogent side of this and our bank happens to be on Level 3's side (in networking terms). So, we are unable to do any banking (hourly and daily wire transfers, etc). I assume if the bank was multihomed this wouldn't be a problem but it looks like there is only one way in for now.
The oddest part...I did a WHOIS at networksolutions.com for our banks website and it says they are 1.1.1.1??? Anybody seen that kind of behavior before?
You know what? You're an asshole with no sense of humor. Think before you post, indeed.
Boycott everything - they're all trying to fuck you one way or another
HOW PATHETIC! Two major ISP are willing to piss off thousands of people just because they've thrown their toys out of the pram. Back at school they'd get told to shut up and get along, now it'll become a legal action. GET A FUCKING GRIP!!! I thought the world was too sensible for this kind of thing. I was wrong.
You feel sleepy. Close your eyes. The opinions stated above are yours. You cannot imagine why you ever felt otherwise.
Must... not..
All your internets are belong to UN!
Damn. Couldn't stop myself.
Management is a technology.
The business decisions affect the layout of the internet. The two are intertwined, and if there is an solution to this problem that involves a change in architecture, well then, there's something wrong with the current architecture.
Seems that buying up dark fiber might be a very lucrative investment. Your backbone in a pissing match? Jump on GoogleBone....
A backhoe took out Verizon's main cable into a few towns in Massachusetts. The reason this was an issue for us is it included the town that housed the hosting facility that had our servers. Our sites were down for most of a day.
One guy called up, demanding to know why out site was done, and he kept telling me to "Reboot the file sever" and "I don't care about your server. You don't need a server to have a web site. I have a web site and I don;t have a server."
Mind you, this jackass was a middle manager trying to take online courses, not an employee.
I went through the spiel that I'd been giving all the people who called up. The typical reply was along the lines of "SO it should be up tomorrow" or "Bad luck." A few people said "Don't you have another data center?"
This jackass said, "The Internet can't go DOWN," in the tone of voice normally used by high school students trying to humiliate someone with supposedly superior knowledge.
"Sir, I never said the whole Intern-"
I then heard him yelling to other people in his office, "The a**holes say the whole Internet's down. I can't believe they expect me to believe this kind of-"
Someone else took the phone and said to me, "The INTERNET isn't down, just your piss poor site."
"Sir, I never said the Internet was do-"
This new person yelled t the rest of his office, "They're changing their story."
Now I was pissed.
"Listen, I am NOT changing my story. The building that has our servers lost it's Internet connection. A cable was cut by a jackass laying cable. When they fix the cable, we will be back online. I never said the Internet was down."
"That's not what Bob said you said."
"In which case I question his listening skills and comprehension of technology." What the Hell, I was already job hunting.
Dead silence at the other end.
"Give me your manager."
"I AM the manger," I replied. The only person above me is the company's owner.
"Give me the owner."
"He left for the day when he found out our hosting center had lost it's Internet connection. I can send you to his voice mail if you like."
"Give me someone in sal-"
"They left too."
"Give me the owner' voice mail."
I did.
In the end, the owner had so many angry voice mail messages about the outage that he went through and deleted them.
He was very happy when I reminded him and the sales staff that we were still at 99.9% up time.
"Live Free or Die." Don't like it? Then keep out of the USA
I was just wondering, if an outage can cause such trouble, why don't antisocial elements such as terrorists target intercontinental cables and suchlike? Of course, they won't have the resources to hit them on the seabed, but couldn't they get them where they enter the water?
The reason that Cogent is still blacked out from Level 3 customers is that the traffic routing's not being changed. There's still connections between the two networks (through other networks), but those aren't being used by choice. Cogent COULD fix it if they wanted to, but they rather have the blackout and try to blame Level 3 for it.
- AMW
Yeah, it was designed so that when two parties don't want to exchange traffic anymore, magical InterWebGnomes charge in, armed to their teeth, to correct the problem and dicipline the offenders. In fact, the very ability to filter traffic at all means the network is broken! Oh, the horror!
;) Seriously though - more regulatory intervention needed perhaps?
OK, so Cogent and Level 3 don't want to talk. Fine. That's their business agreement between each other.
But what about the business agreements between them and their customers?
Have they violated their contracts with their customers? Do they owe them a refund for the full time the Cogent/Level 3 link is down? That could be a lot more expensive to both than accepting a more painful peering agreement, or buying transit from a third party for the duration.
Have they written the TOS so that they can do this without violating it? Then the customers are SOL. But in that case the customers have a strong incentive to hunt up another provider with better contract terms. So this could lead to a major, and ongoing, erosion of the customer base for both companies.
I suspect the invisible hand is about to give them both a MAJOR spanking. And give the net a push toward avoiding recurrences of this sort of "failure".
Internet service customers aren't paying to be pawns in this game. They're paying for connection to the internet - ALL of it. To the extent that Cogent and/or Level 3 convinced their customers that internet service was what they were buying, if they entered the contract while willing to take this move they engagaged in fraud.
Smart move for whomever DIDN'T initiate the line cut would be to cut a deal with a third party to pay them to carry their packets and deliver them TO the other. (The other can follow suit or not as it choses - but the willing player can't suck out packets the unwilling won't route.)
Then the willing player can say to its customers "I'm keeping my contract with you. I'm getting your packets delivered to their routers. Whether they go the rest of the way or not is the other guys operation. I'm accepting packets for you from his customers wherever they hit my routers - if they ever do - and delivering them to you. If the other guys won't deliver them to me there's nothing more I can do technically. I've tried to maintain a direct connection with him but he won't budge on reasonable contract terms so I'm paying extra to do everything I can without that to give you the service I sold you."
A solution of that form would keep a single bully from taking down anything but his own customers - and do it in a way that would be visible to any customers of the company he's fuding with: A traceroute would show that their packets go to the third party and stop at their peering router with the kermudgeon (which will probably be named appropriately) the first kermudgeon router, further in where filtering takes place if inbound packets are cut off, or all the way to the target. Meanwhile a traceroute from the target would show the packets never leave the kermudgeon's net. B-)
Of course if the kermudgeon decides to pay for the third party transport to keep from looking bad, the net is back up, while both sides have an extra cost of business that gives them an incentive to cut a cheaper deal.
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
This should be illegal. When you run an ISP, you basically say, "I'll take traffic and route it." along with the usual service providing to individuals or companies. Blocking any section for business reasons is just retarded.
What if I built a really popular road with my own labor and materials, and said, "Sure! You people can use it..no problem.."
And then I decide that only cars I SELL can drive down that road. Everyone else has to keep off. The government would smack me down and take it from me.
Job? I don't have time to get a job! Who will sit around and bitch about being broke and unemployed then?
But there is a perfectly viable and simple solution:
"Dear ISP,
as you might know we are your Customer, Contract Number .
According to our contract you are to provide internet connectivity to us for a monthly fee of . Unfortunatley you failed to do some since Wednesday Morning this week. I assume this is because you yourself are a Level3 customer, and Level 3 can't deliver that type if service any more.
Please correct this situation within the next 12 houzrs. Internet access is essential for your business and you are already at the borderline of your SLA. Personally I'd suggest you buy additional upstream from someone else than Level 3 (Cogent might seem a good Choce, sinmce they currently offer it for free to you) but of course the decision is yours, wahtever gets the job done.
Please be advised, though, that we definitely expect you to restore service no later than 12 hours after you have been served this notice. Failing to to so will force us to sue you for breach of contract in court, and since you elect not to restore our service despite of beeing able to do so and despite of us having fulfilled our part of the contract, we consider you acting in bad faith and consider the damage disclaimers in our contract void.
We look forward to your cooperation in this matter.
Joe. Q. Customert, Inc."
dereference, thanks for the link...everything is green except where these two meet
He who knows best knows how little he knows. - Thomas Jefferson
Good grief. :-)
...wait, I can't seem to get on bash.org today. Wonder why.
Ok, and peeps want to turn over the root DNS Servers from ICANN to the fUN*? WHAT?! This so reminds me of the old pissing contents back in the days over phone line bandwidth when I worked for smaller telcom companies. They would have to negotiate constantly with the big boys just to let their little traffic onto their networks. *sigh* When did things become so difficult?
Sorry man... the Internet pooped on me.
Is this why I can't read userfriendly or Something Positive this morning? Or is it just some weird coincidental webcomic blackout?
... who have kept their UUCP skills alive. For they are the only ones who will be getting their pr0n in the new age. ;-)
"Consistency is contrary to nature, contrary to life. The only completely consistent people are the dead." A. Huxley
is this the reason Penny Arcade has been site-non-grata for the past few days?
Fruit Fucker Fans wanna know!
With a "+1 Concise and to the Point"
Cogent went through exactly the same exercise with France Telecom in April this year - I was on the receiving end (hosting a service behind for lots of french customers) and it was a right royal PITA.
Interestingly enough, it was the same story, France Telecom had given Cogent 60 days notice to sign a new agreement. The deadline went by, Cogent wouldn't budge, never the faintest warning to the customers that there was a potential problem. In fact it was 12 hours after the peering went down before we got ANY information from Cogent, which was the following mail, basically a royal "passing of the buck".
From: NOC-EU
To: Thu, 14 Apr 2005 10:06:13 -0400
Subjact: Peer Problem with France Telecom HD271257
Dear Cogent Customer,
We are very sorry to hear you are experiencing problems with your
connection. We're aware of this issue and we are working to improve
things as quickly as possible.
The specific issue is with France Telecom. Since Cogent's inception,
we have maintained a robust peering relationship with them. Recently,
however, they made a unilateral decision to de-peer. The direct
result
of this action is the isolation of our two networks.
Cogent is very concerned by this action. Peering relationships such
as these are the foundation of a successful Internet community. We
are
working to better understand France Telecom's motivation. We
speculate
that this action is in direct response to Cogent's competitive
pricing
in the market. Cogent's goal is to have France Telecom re-instate the
connection.
We appreciate your patience as we work on this issue. We are closely
monitoring this situation in our support center. If you call back for
an update, please reference master ticket: HD271257 We encourage you
to also contact France Telecom so that they fully understand the
impact of their action. You may reach them at noc@opentransit.net
As always, we sincerely appreciate your business. We will keep you
up
to date on the situation.
Sincerely
regards / Mit freundlichen Gruessen
C.C.D. C o g e n t Communications Deutschland GmbH
Stephanstr. 3
60313 Frankfurt
Tel.: +49 69 299896 96
fax.: +49 69 299896 54
mail: noc-eu@cogentco.com
It seems to be Cogent's business philosophy to play a game "chicken" in these situations, to not warn the customers, and to take it's customers as hostages.
We migrated our entire uplink to another provider in 3 days, and of course the link came back up on day 5 - but we're NOT going back.
Not confused enough? http://translate.google.com/translate?u=www.slashdot.jp&hl=en&ie=UTF8&sl=ja&tl=en
...sabotage even the best infrastructure.
They wouldn't have been a transit until that link broke down.
Say you have 3 huge backbone providers, A, B, and C.
A peers with B
B peers with C
A peers with C
None of these providers is a transit provider for the others, because none of them need it as they all have peering relationships.
Now say the deal between A and C goes bad; the traffic can still get bwtween them by going through B. Now, in this situation, B would become a transit carrier, yes. I don't know if that would mean immediately that they would have to start paying higher fees or what, because that would not be their original contract
Fun!
As was pointed out numerous times, both of them have many other peers in common that could correct this with a few quick router commands. It would be expensive (multi-thousands per day), but in a disaster situation it would be done. We're talking about 7.5% of L3's customer base and 11% of Cogent's base having partial internet unreachability. This isn't a widespread OMG SKY IS FALLING event. Hell, we're all still posting this on slashdot VIA THE INTERNET WHICH IS OPERATING AS DESIGNED.
Beijing (AP) - In a move that has surprised the world, China has announched today that its new DVD format will be 100% Freedom-Free. "We want to make sure terrorists cannot attack the pride of the People's Republic of China," said President Hu Jintao. "China will not be hindered by other formats that could possibly include Freedom protocols," he concluded.
"We were just trying to stop those damned file sharers," said Mitch Bainwol, Chairman and CEO of the RIAA. "This time, China has gone too far. They can't expect to attack freedom and get away with it. Besides, how are we supposed to be the bad guys when China shows us up with this? We have an image to maintain."
Following the announcement, the RIAA is expected to respond later today with a Data-Free DVD format. "You can't steal what you can't see," said Bainwol.
Didn't realize I was on a reply for a different story. Got my FireFox tabs messed up.
the problem has been solved. I can ping level3 from cogent and i have one connection. I don't know yet who flinched first......
The availability grid for the past 4 hours shows ~40% and the grid for the past 1 hour shows 100%. As noted by "Cally" below, I honestly have no idea how exactly this grid has been generated (hence my original disclaimer) but this certainly seems to indicate, from a practical standpoint, that the L3/Cogent issue has been very recently resolved. Indeed, from my (single-homed) L3 server I can now traceroute directly to a (single-homed) Cogent host.
...of how capatilism doesn't work.
These providers have no obligation whatsoever to provide any level of redundancy so long as you keep paying their bills.
"Nine times out of ten, starting a fire is not the best way to solve the problem." - my wife
Peering arrangements are different. Two networks that have a lot of traffic for each other will set up direct connections, split the direct costs of the connections, and not charge for accepting packets from the other carrier. But they'll only advertise the routes for their *own* customers. If two small ISPs peer with each other, typically they're each also buying transit service from big ISPs, but it's cheaper for them to dedicate a connection or put bits on a public peering point like MAE-West than to both pay their upstream ISPs.
The biggest ISPs in the US are called "Tier 1" ISPs, and they all peer with each other rather than buying transit, though they might buy transit for international connections, if they can't get the other side to buy transit from them. It seems flaky, but it makes business sense, or at least it did for a while. In some sense, being big enough that all the other Tier 1s will peer with you is what defines Tier 1, and aside from technical issues, it's a marketing thing - "See, we're one of the big players!" Peering and Transit don't mix very well - you either connect to a given carrier by peering, or by transit, or else you spend a long time hammering out custom arrangements about exactly which routes you'll accept and tweaking routing tables.
Cogent is a Wannabe-Tier-1. Their main business model is to put fiber into big multi-tenant office buildings and sell everybody 100-meg Ethernet for about the price other carriers charge for one or two T1s. If I were a customer, I wouldn't expect there to be enough upstream to really get that much bandwidth all the time, but I'd expect to get more than a T1 all the time, and a lot more than a T1 almost all the time. Level 3 has apparently decided they're not getting enough value out of the relationship (i.e. not sending Cogent enough packets to make it worth their while) to keep peering, and wants Cogent to either pay them for service or get transit from somebody else. They gave them about 50 days to make other arrangements, but Cogent decided to play chicken with them.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Peering is an iterative prisoner's dilemma. The first company to overuse the other's bandwidth - ie to defect - shows a profit for having done so. As Robert Axelrod ( 'Evolution of Cooperation', 1984 ) and others have demonstrated, the best strategy in such a situation is a tit-for-tat, that is: I defect only after you do, but cooperate otherwise. Cogent defected, and then Level 3 defected. It will cost them both, but the lesson will not be lost on others. If most companies adopt TFT, those who defect prematurely will be culled out, and the the result will be a pool of better behaved companies. The resulting evolution is better for all of us.
Actually, avoiding a blackout like this one is not that difficult. All you have to do is make sure you are receiving your connection either directly from, or via a provider that receives a connection from, a level 2 service provider that has signed up with multiple level 1 providers. The only ones effected by this outage are those who only have one or the other of these two level 1s (well, maybe a 1 and a 1.5). Even those that have a different level 1 are unaffected.
09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0
The problem is really a matter of routing policy. Since the net is comprised of businesses, it all comes down to money. Technically, peers should announce all of their routes to peers. Practically, they only announce routes to their own customers to their peers. So, A, B, and C all peer. Peering between A and C goes down. However, B does not announce routes to either A or C even though they know them so traffic between A and C is cut off.
As long as the net is run by businesses that must make money to continue, and they are not paid to announce full routing tables, the net will be subject to this sort of dane brammage.