Slashdot Mirror
Internet Partitioning - Cogent vs Level 3?
slashmicah asks: "Internet partitioning and Tier 1 ISPs are something most people don't know much about (myself included). Today, however, some Slashdot readers might have run into some issues involving these two topics. Cogent Communications and Level 3, both Tier 1 ISPs, are apparently having some 'undisclosed' disagreements, causing an Internet partition by turning-off or deactivating their peering point. Cogent Co. has released a statement explaining their side of the problem, however they have no mention of when the problem will be fixed, or when they will sort it out. This partitioning is a problem because any [single-homed] computers that are connected through Cogent Co, can not connect to [single-homed] computers connected through Level 3. Having spent all day sorting out this problem, I ask Slashdot: Isn't there a better way that the issue of peering can be handled/regulated? If not, does the future hold a scenario in which the Internet is split into several separate networks, only to be connected at the whims of large corporations?"
While it is always fun to entertain such doomsday scenarios in ones' mind, I don't think that anything like this is possible. Current demands of most large corporations (Microsoft, Apple, any number of others) along with the internet-using public are for a universally-connected internet. Any company that simply creates its own network is going to face a huge revenue loss.
Level3 is threatened by Cogent's bandwidth pricing model, and is using it's weight to threaten that model, forcing Cogent to buy transit if it wants to reach its network. THat's how things work: you can't get free bandwidth from everyone, you're going to have to be willing to step up and pay for your link.
Video Phone Blogs send video messages straight to the web.
If not, does the future hold a scenario in which the Internet is split into several separate networks, only to be connected at the whims of large corporations?
A quote about censorship. Come on, we all know it. The internet will see that as damage and route around it. The very fact that you mention that this affects single homed computers on one or the other network means that even at the onset of this "partitioning" it is ineffective.
The customers on each one of the company's networks needs to call them and demand resolution. This is the fastest and most effective method of getting the company to pay attention and fix the problem. If the customers open trouble tickets on this issue it will get resolved. - Dan
At least it's not like UUNET more than one, some years ago, wanting to charge other Tier 1's per packet for transfer when peering while their traffic they wanted to pass for free. They were a big dog and were trying to make everyone pay. No one did and threatened to or did kill off traffic until UUNET got the sh*t together. But the did try to pull it off more than once.
I regret the lack of attribution on my above quote - it's from Geoff Huston, with full document available here
Video Phone Blogs send video messages straight to the web.
For something as essential to the nation as internet service, maybe it's time to consider implementing regulations similar to what electric, water, gas, and telecommunication companies have.
If my grandma can't check her email for a day, I don't really care that much. If my doctor is consulting with a cardiac specialist over using VoIP (V being either voice or video) concerning an acute health problem then I have a much larger problem with outages. As long as we have important economic or healthcare services running over the internet--which is the foreseeable future--this sort of thing needs to either be avoided or have a pre-planned workaround.
I guess this explains some of the unresponsive hosts I came across today. And here I was thinking it must be Bob's Worm of the Week.
---
According to the latest ruleset, this post should be modded as Vorpal Flamebait +5.
It's possibly on a par with the scenario of countries cutting others off their internet connections. Not that it can't be done, but the repercussions are akin to MAD. Although these days with the effective merger of state and corporate interests anything can happen...
insecurity asks the wrong question irritation gives the wrong answer
If large companies connect to multiple Tier 1s, then they pretty much don't have to care; their customers can still reach them.
Personally, there are several sites I can't get to from home now. I didn't have any problem getting them from work (UT Austin). I have effectively zero power to rectify this. Annoying.
Now, if Cogent offered me some way to connect to them for an additional $5/mo... would I?
Think... if the government allowed an additional $5/mo. for each Tier 1 my ISP (Time Warner) is connected to... my cable modem bill would instantly double.
That's a scenario that bothers me more than the dissolution of the Net does. Flip side, the Internet would get a whole lot more redundant really quick...
The Internet's insecure enough without introducing race conditions into it.
Besides, that's so 1990s. The 21st century equivalent is to yell "Tier 1 ISPs don't care about spam victims!"
we got an email this afternoon from our provider, who let us know that cogent will be reachable by their second link, which is WilTel. However, the link is slower than the Level3 link. There will also be more traffic being routed through less points, meaning congestion. (and obvious lack of redundancy, if the WilTel connection has problems, no Level3) We have had users complaining about sites being unreachable at random times this afternoon. One of our providers very big customers is the OSU Open Source Lab, home of Drupal, mozilla download servers, master.Kernel.org servers, and many, many others. If your having problems reaching these sites, that is probably why.
What are we going to do tonight Brain?
Incorrect. The company I work for is one of Cogent's largest clients (16 gigabits). None of our servers on the Cogent network can reach Time Warner, AOL, Verio, etc. I'm at home right now on my Time Warner connection and I'm unable to reach any sites on our network, ping any cogent routers, receive email, etc. We lost nearly 1/5th of our total Cogent traffic today due to this.
As a cogent customer, it's *really* not their fault, IMHO. I gather L3 pulled the same stunt with XO last week.
As to the notion by another poster of not expecting peering with someone bigger for free forever: 38.0.0.0/8 Class A is Cogent/PSI... how much bigger than being an entire Class A (and then some?!) does one have to be to be considered [ahem] "equal"?
It was a mutual arrangement: they both allowed transit for one another's packets... pretty fair given the size and stature of them both, I'd say.
Running 'Nix is like owning a Lightsaber. It's "a more elegant weapon for a more civilized time."
Everyone knows what a success MAE-East, MAE-West and the rest of the public peering points have been. Let's build a few more of them! Or, even better, encourage the federal government to get involved. Perhaps spending some of the federal budget on this problem would be a good idea. I think I recall a peering point clause in the constitution somewhere.
In all seriousness, these private companies will work it out when they realize that their paying customers are pissed and leaving because they're no longer selling very complete connectivity. Just like in the past, it won't take long. If TV has taught me anything, these problems are usually wrapped up pretty nicely in about 28 minutes.
Yeah I am just a network guy but I bet I know more about this than the "expert" "predicting" gas prices on CNN.
In places that do not constitute a major conjunction of backbone connections, a Tier 1 provider becoming a bit of a rogue would have a more noticeable effect on its downstream customers. Additionally, if too many providers pulled this kind of stunt, the resulting inefficiency from rerouting packets in an end run around the broken link could lead to the entire net being bogged down in a manner similar to what happened when Nimda, Blaster, or Sasser+Welchia hit.
As seen on any network, a sufficient degree of inefficiency will result in DoS. How many peerage agreements would have to be cancelled for this to happen? While I freely admit I couldn't compute a number for an effective local DoS vs regional Dos vs global DoS, I would still be extremely interested in making sure it won't happen.
---
According to the latest ruleset, this post should be modded as Vorpal Flamebait +5.
Ironically, because of the depeering, I can't get to it!
Viper is the preferred editor of the Emacs operating system.
I have been kicking around the fringes of the high-speed data stuff for a number of years, and there's one true lesson to be learned;
Telcos suck.
ALL of them do in their own special way.
Isn't there a better way that the issue of peering can be handled/regulated?
I've always favored shotguns at three paces.
"They redundantly repeated themselves over and over again incessantly without end ad infinitum" -- ibid.
Lets assume Cogent and Level 3 split up one city (and I know they have done it to at least one place) amongst themselves. Someone happens to be using voip to call 911 while on Level 3, while Cogent is maintaining the 911 system's voip call receiver, preventing the voip 911 call from ever reaching it...
wow they could both be sued for huge sums of money...
Viable Slashdot alternatives: https://pipedot.org/ and http://soylentnews.org/
SPF support for most open source mail servers can be found at libspf2.
"Recently, certain peers have been disconnected from their direct connection to the Level 3 IP network. Some disconnected peers may elect to block access to certain IP addresses as a result of the disconnection. If a peer elects not to restore connectivity to the Level 3 network through alternative means, customers seeking continued access to the Level 3 network should make alternate arrangements."
They're saying Cogent is intentionally not advertising routes to them via other providers, presumably because they're upset about not having a peering agreement in place. Anyone affected by this presumably needs to harass Cogent.
http://ws.arin.net/whois?queryinput=AS174
From Cogent's side, as linked in the summary:
"Cogent will offer any Level 3 customer, who is single homed to the Level 3 network as of October 5, 2005,
one year of full Internet transit free of charge at the same bandwidth currently being supplied by Level 3.
Cogent will provide this connectivity in over 1,000 locations throughout North America and Europe."
Not that I really know what that means, or whether their claim that Level3 cut things off really makes Level3 the bad guys. Anyone want to explain for those of us that don't get it?
Reminds me on an incident that happened in Australia a couple of years back. Telstra and Optus were pretty much owned all the links outside of Australia, but Telstra lost their major one in a shipping incident. (Sharp anchors?) With nowhere for their data to go they rerouted everything through Optus to let them handle it.
Optus didn't appreciate that and promptly blocked all data between themselves and Telstra. Customers with Telstra were pretty much screwed because they couldn't contact anything and with their network going nuts even sites within Telstra sucked a lot. Still, for a couple of days there, it was two halves of an internet available in here. Was amusing to watch really.
There's an easy technical fix to this problem: Start a nuclear war at the location of this peering point. Then by design the Internet will route around that area, and communications will be reestablished.
First, I think that Level-3 is within it's legal rights in terms of dealing with Cogent, but is probably in trouble with it's customers. I am a customer of Level-3 and of Cogent (in the same facility). When I buy IP transit from Level-3, I am not buying "part of the internet". This peering issue places 45+ Million IP addresses out of reach of the Level-3 network (and vice versa). Level-3 did not notify me that they were making this type of change. There is nothing on Level-3's website that even implies that everything is not hunky dory. If you buy a Level-3 line today, will they disclose to you that you are not connecting to the entire internet. I know I am being a little niave here, but not disclosing such a large change of policy is unconscionable.
Second, it is dishonest for Level-3 to blame Cogent for this exclusively. Level-3 had a peering arrangement with Cogent for a long time. If you look at Level-3's interconnection policy page:
http://www.level3.com/1511.html
It still looks like Cogent and Level-3 could peer under these terms. It was Level-3 that pulled the plug, not Cogent.
What is really annoying is that this is only traffic from Level-3 to Cogent, not to other parts of the internet. Level-3 wants money for Cogent customers to connect to Level-3's network but does not understand that this is a two-way connection and that Cogent's customers and Level-3's customer both benefit from this equally.
Up until this point, I was very happy with Level-3. They run an excellent network and I pay top-dollar to be on it. This blatent disregard for the impact on their customers is a diservice to their customers, to their reputation, and only begs for regulation.
The internet has no government, no constitution, no laws, no rights, no police, no courts. Don't talk about fairness or innocence, and don't talk about what should be done. Instead, talk about what is being done and what will be done by the amorphous unreachable undefinable blob called "the internet user base." -Paul Vixie
-- I have a private email server in my basement.
Peering is when you agree to send traffic destined to network X directly to network X via a direct connection between you and X. If you're using X's network to send traffic to Y, that's transit, and X will naturally expect you to pay for the privilege.
Everybody's a libertarian 'till their neighbour's becomes a crack house.
Translation: Cogent will let any Level 3 customer who is cut off use their service for one year at no charge.
This will eliminate any internet performance anomalies for those customers so that they are not affected in a bad way by this issue. It's also a good PR move that might let them grab a few Level 3 customers who are impressed by the goodwill gesture.
---
According to the latest ruleset, this post should be modded as Vorpal Flamebait +5.
I have reviewed all information available at this time, including discussion threads on many sites more specialised than Slashdot. This is bad. Very bad. Right now, there are millions of Internet users with partial connectivity.
But the action of Level3 is not merely an inconvenience to end users; it is hurting a great many small businesses, badly. There are thousands of small businesses that depend on single-homed Internet connectivity and that cannot afford dual-homing. There are dozens of low-cost datacenters that provide single-homed bandwith to tens of thousands of servers.
As we speak, the livelyhoods of thousands of entrepreneurs are being threatened. Many people depend on being able to offer internet services to any peer on the net. But today, Level3 has changed the rules of the game, and have split the Internet into two somewhat isolated internets.
This is happening on a very large scale. Sure, most of the affected people and businesses are going to get through it just fine. But given the sheer scale of the Internet, a small percentage of those depending on full connectivity will not escape this ordeal unscathed.
You can be sure that a few small businesses will close because of this, the reputations of a few persons will be damaged, and there will be a few bankruptcies - all because of Level3's evil actions. You won't hear about it in the media - nobody cares about such small-scale damage. But the damage is already done, and it is getting worse with every passing hour.
I urge you to join me in a five-minute hate against Level3 and all that their evil discriminative ways stand for. While Cogent is widely recognized for its shitty cut-rate network, they are the good guys here. In the past few years, Cogent has been a major driving force for lowering bandwith costs. Level3 is fighting back, and they long for the days where they charged 5000$/mbps. I say: down with Level3 !
The short version goes something like this:
.. well .. you get what happened today (I'm making an educated guess here based on what I know of the two carriers involved). A decides that spending 30 grand a month for what is a very lopsided bandwidth agreement is no longer economically feasible or reasonable. They go to B and say 'look, we're not doing this anymore, we're basically paying a hell of a lot of money every month for you to send a ton of traffic to us, and we don't send much of anything to you. You can either pay for all (or some larger portion of) the circuits, pony up some $$ per megabit, or we'll just cut it off at the stub and be done.'
Provider A and Provider B peer, be it public or private, normally they do this in several places and alternate who pays for the circuit, etc. Now, under normal circumstances, they both push enough traffic from one to the other to justify this mutual payment plan. However, in some cases, you find that B is either intentionally dumping traffic into A thinking A won't notice, or A discovers that its sending so little traffic to B in comparison to the amount B is sending to A that its not worth the continued cost.
When the first sort of thing happens, it usually gets resolved -REALLY- quick, that sort of behavior is not tolerated and will result in B getting de-peered by A (and potentially others once the abusive behavior is discovered and known) exceptionally quick unless B can show that it wasn't done knowingly or intentionally.
When the second instance happens
Based on Cogent's 'oh poor us' post from this morning, I'm leaning towards them having given L3 the finger when L3 said 'look, this isn't equitable, we're going to have to re-arrange the money'.
YMMV of course, but I'm betting I'm not terribly far off.
As a cogent customer, it's *really* not their fault, IMHO. I gather L3 pulled the same stunt with XO last week.
Yeah, Level 3 is really out of line in my opinion. It's not that they shut down the peering link. That wouldn't be that big of a deal. The traffic would just flow through other providers on less efficient routes. It's not as though every single backbone carrier peers with every other. But I just checked my BGP sessions, and Level 3 is not advertising the Cogent route at all. And you know for a fact that Level 3 is receiving the Cogent route from many of it's other peers. But it appears that they are intentionally filtering out the Cogent route. Which is pretty much not playing by the rules. It's one thing to shut down a peering agreement. It's something else entirely to refuse to accept that route from any of your other peers.
I wish that was true, but I am both a Cogent customer (100meg Link) and a Level3 customer (Multiple T1's) and the worst is true. My Level3 connection can not reach my Cogent connection, and my Cogent connection can not reach my Level3 circuts. Level3 is no longer BGP peering with Cogent so all routes normally advertised to Level3 no longer exist (from the prospective of Level3 customers). And although the request packet from Cogent might make it to Level3's network via another provider (i.e. route around the problem) with no route back to Cogent's network (BGP ASN 174) there is no way to get the traffic back to the user on the Cogent network.
The internet is made up by many layers of ISPs. Individual users connect to a user-level ISP, which must then connect to other ISPs to gain access to the rest of the net. The biggest ISPs tend to trade bandwidth between themselves without any payments in terms of money, while they charge lesser ISPs to connect to their networks.
In this case I think it's a fair guess that Level 3, which used to let Cogent connect for free, has decided that they are enough smaller (Cogent is about half Level3's size, controlling a 'mere' 23+ million IPs), that they ought to be paying to use their network. Cogent probably refused to pay, at which point Level3 cut them off as a negotiating tactic.
Now people on ISPs who connect to the rest of the internet through cogent, and only cogent, can't connect to anything that connects to the net through Level3 and only Level3.
Any reputable ISP ought to connect to many others, not just a single large provider, and thus see at worst a noticeable slowdown in some sites due to this depeering. But there are always some people who go the cheap and easy way, set themselves up with a single point of failure, and get bit in the ass by events like this.
You can call:
720-888-2518 (Level3 Investor Relations)
and complain.
Or call 877-453-8353 (Main customer service number).
That's called transit and you pay for it. Peering connections are intended to reach the peer's directly connected customers, which would include the peer's transit customers.
Anyone want to explain for those of us that don't get it?
What they are essentially saying is: "We haven't done anything. We haven't made any changes on our side" Level 3 have terminated their connection to Cogent "Without cause". Now, that's probably legal speak on Cogent's side for we haven't got the letter in triplicate yet telling us what the reason is. Or otherwise whoever put up the notification about it doesn't know.
Now, Cogent may have tried to change the peering arrangement, or Level 3 may have too, one side probably didn't agree, or otherwise an agreement ran out and the switch got flipped. This has happened previously with Cogent in their peering arrangement with AOL.
What Cogent are trying to do is get business from Level 3 customers because Level 3 stopped the connection. Cogent is offering them connections to the Cogent network (And subsequently Cogent's customers) for a year with no fee on the amount of data they put through. That connection itself they will obviously have to pay for, but the customer can connect into (presumably) the closest of any of 1000 points across North America and Europe.
Now some people are already connected to both Cogent and Level 3. These people won't have any problems as they will be able to go direct into either ISP. These people would probably have never have used the interconnect between Cogent and Level 3 either, unless one of their connections into either Cogent or Level 3 went down.
I understand this is still rather technical, for a simpler version, take a look through the document that I linked to.
Curiosity was framed; ignorance killed the cat. -- Author unknown
I called Cogent regarding this. They stated that there is too much traffic to switch to another peer.
I can't believe I'm hearing a call for more regulation, even U.N. control. The lack of rational thought and disregard of unintended consequences amazes me.
The Internet has flourished without much control, run by Both large and small businesses for one reason: profit. Information is free yet its distribution is profitable.
If we give government control (taxation, censorship and worse(, we'll see less freedom.
Why did this jinx happen? Because the top tier providers aren't making a profit. But their calls for support go unheard, so they found a way to make it news.
When businesses that rely on the infrastructure paid for by private industries, they have high expectations. But they're not paying for that infrastructure!
Trust me, no one wants to bifurcate the Internet. Its a ploy to show a problem that needs to be solved. You will Never see it done for control, censorship or monopoly powers. You'll only see it when consumers don't pay for what they use. See California's old electric company that was forced to sell energy at a loss. They went bankrupt.
We once had to backhaul a huge number of routes because of a peering issue between PSInet and @home I think. Fiber had been pulled all the way to the mark outside the building. All PSI had to do was literally connect up the line. It turned into a pissing match between the two CEO's and just sat rotting for MONTHS.
I remember back when NO ONE would peer with Quest because they were all datacenters and not dialup, so they had no advertising value. So no one would peer, not even if Quest paid. (depending on who bennifits most, the determines things like who pays how much etc). So Quest went around buying up little backwater mom&pop ISPs only for the peering. If they wanted a peering point with a certain net in a certain area, just find who has one, and buy them out. It was funny.
You can call: 720-888-2518 (Level3 Investor Relations) and complain.
Or call 877-453-8353 (Main customer service number).
ISPs want common carrier status when the RIAA sues them, but they don't want it when it means they must carry traffic from all other networks.
These guys suck. May capitalistic pressure force them out of business.
-ted
But that's exactly what the Internet is (well, sometime's they're connected at the whim of educational institutions, but the whole point of the internet is that it's a network of networks).
Apparently, someone should inform Alex Jones that this problem is not the Illuminati finally coming after him.
Clarification, Cogent Communications Group, Inc. which is the Cogent being discussed has only a ~$240 million market cap. The Cogent you refer to as a $2.1 billion market cap company is Cogent Inc which does biometrics, not telco services.
Folks on the NANOG list are discussing this rather vigorously at the moment. You can follow the thread here: http://www.merit.edu/mail.archives/nanog/2005-10/
Sorry, but with modern BGP routes if there is no way through a Level 1 peering point then the data is unlikely to get through except for if your upstream is multihomed with each of the parties that severed their peering point(s). There really isn't as much redundancy of routes as many people think, that mostly went out after MAE stopped being a common peering point for all the carriers and private peering points took over most of the inter carrier traffic.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
About who is big in a peering relationship:
Has nothing to do with size of IP address range.
Has everything to do with how much travels from Cogent to L3 for L3's customers and how much travels from L3 to Cogent for Cogent's customers. As long as those amounts are roughly the same (someone I consider a network guru once told me 4:1 or closer) then it makes sense for both peers to allow the traffic for the other's customers without charging anything for that service.
I don't know if this is why L3 cut off the connection, but if it is I think they are in the right. In peering, each side controls a toll road and are letting the others' customers use their road so that their customers can use the other's road. If Cogent's customers are using L3's road but L3's aren't using Cogent's, why should L3 continue the peering relationship? If L3's customers complain, then L3 will have to do something about it. Otherwise, Cogent will have to make a deal with L3 to reopen the pipe - I imagine huge amounts of money may be involved.
Level 3 is not advertising the Cogent route at all.
I'd bet L3's argument is that they will not provide transit across their AS to Cogent. It's a play that's been made several times before. The first time I know of it being done was in 1995 when Sean Doran pulled this at the CIX-W router, preferring to take commercial traffic via NSFNET or Sprint reseller service. Not only didn't it work, but it caused some immediate political backlash as Sean's action (presumably made without his boss's approval, who was the chairman of the CIX board and took some political grief for Sean's latest stunt) caused several state's to literally drop off the map.
If my memory's right, I think this got pulled again around 1998 timeframe on Exodus by someone like Genuity (I may be wrong about the culprit), only for the higher ups at the culprit to discover they couldn't see half of the world's worthwhile websites and search engines. Much of this was in the transit battle - e.g. if you had consumers, you felt your eyeballs were the value of the Internet and all other ISPs should pay you to get to your consumers, while if you were a content provider, you had the stuff all those consumers were paying their ISP to get to and someone had better pay you for that content.
What can you do about it? Let your ISP know you're not paying them for 80% of the Internet. When UUNET considered pulling this stunt around 1997, I worked for a small software shop that had a couple bonded UUNET T1's and we let them know we were going to drop them the moment they were only selling partial Internet. Then follow through if they do (UUNET backed off). Bilateral agreements are weird things in the world of settlement-free IP exchange, so unless you want a settlement-driven Internet (which will have unusual effects you might not want, like driving a per-packet pricing model), just expect this occasionally and drop those who don't play well with others. When L3 drops customer base, even the Denver boys will figure out their customers aren't happy.
*scoove*
I worked there and its a deathmarch shop for IT and network people; they treat their peopel like crap, and leaving there was the best thing I ever did.
The arrogance of Jim Crowe [workign on his 7th manion and 8th large layoff at Level 3] and Kevin O'Hara (President, CEO) is only matched by the jailbird Bernie Ebbers. They only reason they have yet to decalre bankruptcy and liquidate thier debt (and clear away their bad business model with a fresh debt-free start) is that all their Omaha cronies have tied up money in the company stock, which would be flushed.
That they would resort to stunts like this against companies that undermine their pricing is not surprising. Level 3 have amassed BILLIONS in debt that they cannot service at current pricing levels, while Cogent and other more nimble competitors can sustain operations and drain Level3 into bankruptcy. So Level 3 execs do what arrogant desperate people do: lash out.
Level 3 is playing the "Sampson" card - if they cant make people price it their way, they will take the internet down with them.
And they did this trying to kill XO and now Cogent. Watch for more until they finally admit their business model is a failed one, and they declare bankruptcy, wipe the debt, and then begin to price lower and rake in the profits that their debt service is now eating.
In this case Cogent has: as Cogent does, you're surely, surely not a small ISP!
The point is: as most of us are non-multi-homed end users of ISPs--even major ones like Cogent--we're now all subject to the whims of *other* ISPs as to whether or not we can see customers who aren't even hosted by them?!?! Grrrr.
For instance: right now no one on Comcast, Road Runner, or Verizon can see our sites or those of our customers. How does L3 get off doing that?
Running 'Nix is like owning a Lightsaber. It's "a more elegant weapon for a more civilized time."
That either network corporation allowed this to occur is without pardon.
What I'm afraid of, is when this is all over and people realize how singificant it was, the solution to mangers will be "buy service to each, so we never have to worry about being partioned". Which is exactly what both companies would like to see.
-Malakai
A Dragon Lives in my Garage
It's more complicated that you make it out to be. Even if you connect to a large ISP (like NYC Time Warned Road Runner) you are shit out of luck right now. Not because they don't have a way of routing around the break, but beacuse they aren't prepared to implement such a drastic change. It's not all automatically controlled like people think. Most companies, like Time Warner RR would need to modify and reload hundreds of routers to effectively use some other connection point to get around the current 'block'.
And it's been about 8 hours and they still haven't.
At this stage, you'd be better off with a smaller ISP, because they have fewer connection points to update with the new routing table rows.
-Malakai
A Dragon Lives in my Garage
Depends on where you're connecting from. I was able to hit Penny Arcade from Long Island, NY but not from Pennsylvania or Virginia.
I've seen a few other sites (ucomics) that are on the "other side of the rift" that I haven't been able to get to today. Fortunately, the office was not one of them (or I'd have had to drive the 5 hours into work).
Wolde you bothe eate your cake, and have your cake?
Ummm.. Was I sleeping when Cogent became a Tier-1 ISP?
:-)
Several things:
1) The size of your ISP's ARIN/APNIC/RIPE netblock allocation is not exactly related to the concept of peering parity. (more on this in a moment..)
2) If you use the Internet for "mission critical" applications, YOU should a) be multi-homed on multiple ISP backbones, verify that they have good peering with backbones you need to transit and have your own BGP AS OR b) leverage a single providers' network to the extent possible, thereby elininating problems like these.
3) You buy from Cogent - YOU GET WHAT YOU PAY FOR.
4) Peering. From dictionary.com "Peer" - 1: a person who is of equal standing with another in a group. Notice the use of the word "Equal"? Its there for a reason. If you take the time to read the fine print of peering agreements (http://global.mci.com/uunet/peering/) , you will see that section 1.2 states:
Traffic Exchange Ratio. The ratio of the aggregate amount of traffic exchanged between the Requester and the MCI Internet Network with which it seeks to interconnect shall be roughly balanced and shall not exceed 1.8:1.
Translation: If you dump more traffic on us than we dump to you, then we have an asymmetric relationship. You are not worthy of being my peer because you take more than you give, which lumps you in with the rest of my customers who must PAY for access.
The above peering language is similar for all major Internet backbones.
5) I would expect that Cogent is present in at least one of the public peering points (Mae+pick your favorite ordinal direction) - so their BGP reachability information should be flowing through the MAE's.... Should...
Let this be the lesson - if you build a network on $10/mbit access, you get $10/mbit access. Usually, it's great, but sometimes bad things happen. Even to good fiscally responsible people, like yourselves. Good luck in your next job..
Cogent seems to have a problem policing it's network. That traffic keeps hitting L3 and they are tired of it. That's overly simplified, but yeah, cogent is not the best for policing it's network...I would have a hard time peering with them.
Now, since there is apparently some lack of understand of what peering is by the author of the article.... Peering is when 2 companies run a line between themselves. They aren't selling bandwidth to each other, but they share the line cost and traffic between them only goes between them. It helps speed up routes and in many cases helps avoid bottle necks at some of the central hubs.
First one who can post output from traceroute before and after the partition gets a free '5, Informative'!
This sig rocks the casbah.
I can't believe nobody has yet bothered to explain the difference between transit and nontransit service.
When you buy Internet bandwidth from your ISP, you are getting transit service. This means that you can use the link to send traffic to that ISP and to other ISP's upstream from it.
Nontransit service means that the link is to be used exclusively for sending traffic to that one ISP.
All of the Tier 1 ISP's provide nontransit service to each other, because at tier 1 there is no such thing as "upstream." This is not people playing stupid, this is how it's done at the top. It's the reason why the major peering points exist.
Any ISP who wants to shut off a peering arrangement for stupid business-o-political purposes is creating a hole in its own connectivity, and therefore shooting itself in the foot, plain and simple.
Tired of FB/Google censorship? Visit UNCENSORED!
Important production servers need dual homed, highly reliable connectivity. Public facing servers are a commodity. The commonality of blade servers and big data center technology are escallating this.
Case in point: I've run my own data center for 12 years (18 if you could dial up bbs crap). This week, I'm shutting it down. I need more reliability for an important application, and it will be cheaper for me to outsource the public facing side to a data center (In my case, linux boxes at ServerBeach -- I can plug them, they've made me happy).
This is coming from someone with 13 years running his own shop; who owns good firewall, routing, and standby power equipment; as well as servers. Still, it will be cheaper from month 1 to outsource today. For less money, I don't have to buy (or maintain) hardware, get more bandwidth, multi-homed servers, way more reliable power and facilities, and a lower power bill.
The market is changing. More and more consumer broadband utilities (which is what they are) will have to drop out of the single homed dedidcated circuit market. Dissagree? Time Warner doesn't. Why do you think they're building state of the art colocation facilities and datacenters in the markets they serve?
Because soon public facing servers for any serious purpose will live primarily in big datacenters. The only companies to host their own, will be hosting them in their own big corporate data centers.
The problem with quotes on the internet, is that nobody bothers to check their veracity. -- Abraham Lincoln
When I asked for an explanation of this, it had to do with a corporate silent-treatment of sorts; because Paltel/Jawwal (the Palestinian telco) was suing Cellcom for licensing infringement and illegal operation, the Cellcom network decided to boycott the Palestinian phone carriers. This caused all sorts of problems for Palestinian society, and the effect was that everyone in Palestinian areas were ditching the local telco and getting Israeli Cellcom cell phones. Jawwal was facing dire times, after their offices were raided by Israeli military and tech imports were prevented because of blanket security concerns.
For folks on the ground, this was just one more manifestation of the intifada/occupation, even the corporations were going at it.
More background available here, here and here.
I work for Time Warner Cable and we use Level 3. I can't talk about the details of what's going on but the notes on the ticket opened for this issue and the conference call/bridge have covered alot. Alot of people in high up positions are working for find us a work around until a solution can be found. Level 3 is still routing traffic TO Cogent but they are not routing it back at this time. I was working on this issue all night. My suprise to come home and find it on Slashdot. I sure hope they can come to some kind of agreement soon (contract was terminated at 5:30am 10/5/05) but from what's been said thus far it's not looking like it's going to be a quick fix.
Soylent Green is peoplicious!
lack of resiliency in BGP routes has nothing to do with "level 1" peering points, whatever those are. The vast bulk of private interconnections are richer, more geographically diverse, and generally better managed than the legacy MAE peering points. However, there hasn't been the sort of "peering of last resort" available since well before those days either: the CIX was the peering point of last resort, and eventually both Sprint and UUNet withdrew from it, fundamentally changing the Internet topology from a star to a partial mesh.
Read about it here - warning: it's a 120-page pdf...
-David
Need Geek Rock? Try The Franchise!
um, no.
BGP really *IS* automated, and you clearly have never worked for a large ISP. Disclaimer: I've worked for both of the two largest ISPs, and had backbone access at each, within the last five years. I am not currently employed by either of them.
If you're connected to an ISP who has connections to both Cogent and L3, you're fine. By definition, that includes any actual Tier 1 ISP (UUNet, AT&T, Qwest, etc)
If you're a customer of an ISP who is a customer of one of those particular Tier One providers, you're okay. Your packets will route to either L3 or Cogent as appropriate
The real problem is if you're either a customer of or a customer's customer of Cogent or L3 - at that point there's a disconnect. Both L3 and Cogent are significant wholesale dialup providers, so a lot of dial customers are affected.
-David
Need Geek Rock? Try The Franchise!
Ah, there isn't any routing info that has to change at the modem end. The way routing works is that you don't know the whole route from end to end, it's figured out as you go.
From your computer there's only 1 path it can take, out the modem and to the ISP (unless you have multiple connections to your home network). Where to goes from there depends on the destination and how the ISP has their network setup. A pretty typical situation would be something like this:
So, there's nothing your modem has to "know" about the routing, it just sends it to the ISP's routers for them to figure it out. If it's taken them that long to fix the problem then they probably don't have a good multi-homed setup or they have a lot of static routing that needs to be changed or Level 3 and/or Cogent are still advertising the route as valid but are blocking the traffic.
TWRR can change their routing tables all the time. They could completely change backbone providers for that matter. The only effect the end user would see could be changes in response times and transfer speeds to various sites since it's taking a new route. There can be some disruption in the initial change over until the routing change propogates but that should be fairly quick on the local network.Listen, You seem to keep responding while ignoring what I'm actually saying, so i'm going to spell it out to you.
Customers of some ISPs that have routes out both to the L3 side and Cognent side CAN NOT access any Cognent controlled networks (AS174). In some cases it has to do with not knowing another route to that network. In other cases it has to do with Cognent blocking a path they just don't want used. Case A is Level3's issue, Case B is Cognents. Either way, the downstream guy is screwed.
Look. Here's me trying to get to Level3 side:
And here's me trying to get to something on the Cognent side:
The fact that RoadRunner is sending my packets via ATDN via Verio to get to AS174 shows me that the pinned route RR previously had (ie, all traffice for cognent side, haul via Verio which Cognent bought) is still up, but Cognent is actively blocking the traffic. If they didn't block it, we wouldn't know they were depeered and this would be a non-story. Now, I can't tell you that previously the data was backhauled via the AS3356 (Level3) network, but this is my guess. I just don't have any tracerts from then.
:
But not that Cognent is the only bad guy in this, Level3 has no advertised routes to AS174. Check http://www.level3.com/LookingGlass/
And from what I read on NANOG they are filtering advertisments of the AS174 routes from reaching anyone on their side. So even if you could route through L3 to Sprint to get to Cognent, you wouldn't know.
-Malakai
A Dragon Lives in my Garage
That's not how BGP works. BGP requires TCP state, and the routes would have been cleared out within a few minutes. Your way would work if everyone used static routes, which they don't.
There have been a lot of mistaken things said about this: neither side is actually manipulating the routing table - L3 just removed the only way for it and Cogent to exchange routes.
-David
Need Geek Rock? Try The Franchise!
Cogent's business model is to sell large bandwidths for a low price, usually in multi-tenant office buildings. So they'd drop a fiber into the basement, and sell 100 Mbps ethernet connections to businesses in the building for about the price other carriers would charge for a T1 (that was back when a T1 was typically $1000 instead of $300; I haven't followed Cogent's prices in the last year or two.) Could you expect to get 100 Mbps consistently all the time? Not realistically, but you *could* expect to get lots more bandwidth than a T1 almost all the time, so it was a pretty competitive deal.
But at the end of the Interent boom, every carrier's finances looked pretty unstable, and a very aggressive business model that depends on getting free peering from big carriers while stealing their business customers looked extremely volatile :-) So does it make business sense for a Tier 1 provider to peer with Cogent as opposed to charging them money for Transit? Maybe, maybe not, and it looks like Level3 used to give them free peering but has changed their mind about it. Not the first time something like that has happened to Cogent - they've been back and forth on this with one or more carriers over the last few years. L3 seems to have decided that there's not enough reason to care about Cogent customers to give them free service.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
You utter stupid rubbish and have no clue yourself.
You confuse peering and transit. Cogent sending their L3 traffc via, say Verizon, would violate their peering agreement with Verizon. They would have to pay Verizon for transit/upstream. OTOH, in their case, L3 would not have to pay anything since they still peer with Verizon, and since now Cogent pays Verizon as upstream, Cogent would now be considered part of Verizons network for peering purposes with L3.
So, if Cogent blinks first, they would be forced to leave tier1 and raise their own traffic cost by paying for upstream, while L3 would'nt pay anything.
Right now L3 cutting the peering in fact does stop all traffic between their two network clouds, until one of them starts shelling out the money to get traffic flowing again.
According to an article on NANOG, L3 gave Cogent 50 days advance warning to make other arrangements. Cogent didn't, preferring to play chicken and hope it made L3 look worse than Cogent so they'd back down. At this point, both drivers are barrelling down the road at each other, blindfolded, tossing spare steering wheels out the window, but unfortunately for Cogent, L3 is driving a bulldozer...
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks