Internet Partitioning - Cogent vs Level 3?

slashmicah asks: "Internet partitioning and Tier 1 ISPs are something most people don't know much about (myself included). Today, however, some Slashdot readers might have run into some issues involving these two topics. Cogent Communications and Level 3, both Tier 1 ISPs, are apparently having some 'undisclosed' disagreements, causing an Internet partition by turning-off or deactivating their peering point. Cogent Co. has released a statement explaining their side of the problem, however they have no mention of when the problem will be fixed, or when they will sort it out. This partitioning is a problem because any [single-homed] computers that are connected through Cogent Co, can not connect to [single-homed] computers connected through Level 3. Having spent all day sorting out this problem, I ask Slashdot: Isn't there a better way that the issue of peering can be handled/regulated? If not, does the future hold a scenario in which the Internet is split into several separate networks, only to be connected at the whims of large corporations?"

Comments

Reminds me of...

[mr_z_beeblebrox]

If not, does the future hold a scenario in which the Internet is split into several separate networks, only to be connected at the whims of large corporations?

A quote about censorship. Come on, we all know it. The internet will see that as damage and route around it. The very fact that you mention that this affects single homed computers on one or the other network means that even at the onset of this "partitioning" it is ineffective.

To get the problem fixed

[Dan Ferguson]

The customers on each one of the company's networks needs to call them and demand resolution. This is the fastest and most effective method of getting the company to pay attention and fix the problem. If the customers open trouble tickets on this issue it will get resolved. - Dan

OK, WTF time here

[EndlessNameless]

For something as essential to the nation as internet service, maybe it's time to consider implementing regulations similar to what electric, water, gas, and telecommunication companies have.

If my grandma can't check her email for a day, I don't really care that much. If my doctor is consulting with a cardiac specialist over using VoIP (V being either voice or video) concerning an acute health problem then I have a much larger problem with outages. As long as we have important economic or healthcare services running over the internet--which is the foreseeable future--this sort of thing needs to either be avoided or have a pre-planned workaround.

I guess this explains some of the unresponsive hosts I came across today. And here I was thinking it must be Bob's Worm of the Week.

Re:OK, WTF time here

[ProfaneBaby]

A doctor relying on an internet link for professional purposes better damn well have more than one provider, anyway.

Dual-homed networks are not affected by a simple depeering.

Re:OK, WTF time here

[EndlessNameless]

:::Seriously. People lose water, electricity, cable TV, etc, all the time. They don't all suddenly die. What exactly do you propose the government does when the internet goes down? Send in FEMA?:::

My point is that this service has reached a saturation level in this society such that it must have reasonably high availability and be reasonably priced in order for society to continue functioning normally.

Essentially, it is another piece of infrastructure that we have become dependent on. Yes, I can live with internet access for a week. I did when I moved because the sole local provider has its head up its rear. However, businesses and other entities will have problems functioning at their normal levels if the internet becomes unavailable or less suitable for their normal uses for a prolonged period of time. In the case of potable water, it is simple to predict what problems will arise if availability is reduced or eliminated. What effects can we expect if the effective capacity of the internet shrinks? You know as well as I do that there are many critical systems on the internet, and if the problem gets bad enough some of them will become unusable. How much spare capacity do we have to cover the overhead associated with routing around peerage points that are disconnected for no good reason (good as in technically sound basis for doing so).

This is the issue that concerns me: While the internet is designed to handle failures and remain operable, how many failure points or shutdowns will it take before the internet in this country is unable to meet the demands required by our infrastructure? I've read nothing anywhere that attempts to address this question. While this single cancelled peerage isn't enough (as proven by the fact that I'm responding to your post), I would rather know how much "give" the system has and impose clear regulations so that corporate hissy fits like this don't push things too far. Perhaps this one cancellation is virtually trivial. My concern is the possibility of a larger spate of peerage cancellations causing problems. The internet is as crucial as telephone service to keeping things running smoothly in the US. We have oversight and regulation of the telecommunications industry along with several other services that have been deemed critical to either personal health and safety or to the national interest. I believe the internet has become so integrated into the personal, business, and political spheres that it may need similar treatment.

The point would be to ensure that the system which is supposed to resistant being broken will not only remain functional but will also always be functional enough to support essential infrastructure. Perhaps my off-the-cuff examples of what is essential and what is not were not perfect, and I'm sure either one of us could provide better ones with a bit of thought. Refuting a specific example as being flawed does not, however, address the general concern. How much of this sort of bedwetting behavior can we allow from ISPs before the consequences become bad for a society that is becoming increasingly reliant on the internet?

Re:OK, WTF time here

[PostItNote]

It was on NANOG recently - lemme check - yup!

Article: http://www.physorg.com/news6940.html
Actual Paper: http://netlab.caltech.edu/pub/papers/Doyle-topolog y-PNAS-0508.pdf

And, to clarify and correct myself, it looks like they are claiming that the internet is not scale-free. Which is not quite the same thing as being power-law, but is generally related.

Re:Interesting scenario, though most likely untrue

[Anonymous Coward]

If large companies connect to multiple Tier 1s, then they pretty much don't have to care; their customers can still reach them.

Personally, there are several sites I can't get to from home now. I didn't have any problem getting them from work (UT Austin). I have effectively zero power to rectify this. Annoying.

Now, if Cogent offered me some way to connect to them for an additional $5/mo... would I?

Think... if the government allowed an additional $5/mo. for each Tier 1 my ISP (Time Warner) is connected to... my cable modem bill would instantly double.

That's a scenario that bothers me more than the dissolution of the Net does. Flip side, the Internet would get a whole lot more redundant really quick...

public peering!

[po_boy]

Everyone knows what a success MAE-East, MAE-West and the rest of the public peering points have been. Let's build a few more of them! Or, even better, encourage the federal government to get involved. Perhaps spending some of the federal budget on this problem would be a good idea. I think I recall a peering point clause in the constitution somewhere.

In all seriousness, these private companies will work it out when they realize that their paying customers are pissed and leaving because they're no longer selling very complete connectivity. Just like in the past, it won't take long. If TV has taught me anything, these problems are usually wrapped up pretty nicely in about 28 minutes.

Re:public peering!

[TeraBill]

Public peering points are fine, but some of this may go back to what Johna Till Johnson wrote about in this weeks issue of Network World. She says that the major backbone providers lose money on providing the Intenet backbone links and that the pricing model for this is in need of change. That may be some of what is driving Level 3 in this scenario, since they do carry substantial backbone Internet traffic. Peering points don't really address this since they generally provide a way for more regional people to interconnect and to connect to the backbone providers, but if you build an peering point and none of the big providers come, who carries stuff to the rest of the world? Most of the non-tier 1 providers don't have the pipes to carry that level of traffic. I've seen smaller peering setups built and they generally only serve to provide better access between the local providers, which does have its benefits.

And with the U.N. wanting to take a more active role in the Internet, maybe they need to start trying to manage such things. If it is a utility now, like phone service, maybe we need to do more to regulate how those relationships happen, since I'm sure that if Level 3 decided not to terminate calls from Sprint, someone at the FCC would be on it pretty quickly.

Re:public peering!

[raddan]

I think they're counting on the fact that it is extremely difficult for their to switch providers. We have Cogent-- it took months to get them in to light up our building. And the paperwork from our previous provider (Verizon, *shudder*) took months, too. Combine all of this with the fact that often the service agreement is written with your parent company (read: bureaucracy), and "just switching" isn't really an option. Our NY office is switching off of AT&T, but this is after, seriously, nearly two months of up-and-down, shitty service.

Outages piss me off, but I don't think I'd be motivated enough to deal with the hassle unless we we're talking about more than a week. We have direct links to all of our mission-critical stuff, and backup (T1 or slower, eugh) links to the internet, so we'd survive, but not without real irritation.

Re:Interesting scenario, though most likely untrue

[w1r3sp33d]

IF the backbones segmented everyone with dual links would keep their UUnet, people without UUnet would get one or pressure their provider to use the UUnet uplink, basically it would be survival of the strongest and UUnet would win.

Yeah I am just a network guy but I bet I know more about this than the "expert" "predicting" gas prices on CNN.

Re:Your statement is incorrect

[EndlessNameless]

In places that do not constitute a major conjunction of backbone connections, a Tier 1 provider becoming a bit of a rogue would have a more noticeable effect on its downstream customers. Additionally, if too many providers pulled this kind of stunt, the resulting inefficiency from rerouting packets in an end run around the broken link could lead to the entire net being bogged down in a manner similar to what happened when Nimda, Blaster, or Sasser+Welchia hit.

As seen on any network, a sufficient degree of inefficiency will result in DoS. How many peerage agreements would have to be cancelled for this to happen? While I freely admit I couldn't compute a number for an effective local DoS vs regional Dos vs global DoS, I would still be extremely interested in making sure it won't happen.

Cogentco website problems

[Bananatree3]

For some absurd reason I cannot connect to www.cogentco.com. I have tried going to it with a web browser and pinging it, but it doesn't connect. The ping returns:

"Ping request could not find host www.cogentco.com. Please check the name and try again."

but, when I ping it from nwtools.com, it works just fine. I can connect to many other websites, but not to cogent. I am on a verizon DSL, if that makes any difference. Does anyone have any ideas as to what's going on?

This could spell problems

[gcnaddict]

Lets assume Cogent and Level 3 split up one city (and I know they have done it to at least one place) amongst themselves. Someone happens to be using voip to call 911 while on Level 3, while Cogent is maintaining the 911 system's voip call receiver, preventing the voip 911 call from ever reaching it...

wow they could both be sued for huge sums of money...

Seen it before in Australia

[Airconditioning]

Reminds me on an incident that happened in Australia a couple of years back. Telstra and Optus were pretty much owned all the links outside of Australia, but Telstra lost their major one in a shipping incident. (Sharp anchors?) With nowhere for their data to go they rerouted everything through Optus to let them handle it.

Optus didn't appreciate that and promptly blocked all data between themselves and Telstra. Customers with Telstra were pretty much screwed because they couldn't contact anything and with their network going nuts even sites within Telstra sucked a lot. Still, for a couple of days there, it was two halves of an internet available in here. Was amusing to watch really.

No Rules.

[Nethead]

The internet has no government, no constitution, no laws, no rights, no police, no courts. Don't talk about fairness or innocence, and don't talk about what should be done. Instead, talk about what is being done and what will be done by the amorphous unreachable undefinable blob called "the internet user base." -Paul Vixie

Not surprised. Used to do this for a living.....

[Anonymous Coward]

We once had to backhaul a huge number of routes because of a peering issue between PSInet and @home I think. Fiber had been pulled all the way to the mark outside the building. All PSI had to do was literally connect up the line. It turned into a pissing match between the two CEO's and just sat rotting for MONTHS.

I remember back when NO ONE would peer with Quest because they were all datacenters and not dialup, so they had no advertising value. So no one would peer, not even if Quest paid. (depending on who bennifits most, the determines things like who pays how much etc). So Quest went around buying up little backwater mom&pop ISPs only for the peering. If they wanted a peering point with a certain net in a certain area, just find who has one, and buy them out. It was funny.

Conspiracy

[zecg]

Apparently, someone should inform Alex Jones that this problem is not the Illuminati finally coming after him.

Calm down. Anomalys happen. No biggie.

[Anonymous Coward]

As an eBGP transit engineer for an undisclosed tier1 transit ISP (neither of those mentioned in the article), allow me to calm down those of you that are excitable and correct those of you who are predicting future doomsday scenarios. This is simply an anomaly occurance in a world of less-than-formal business partnerships that comprise the internet (AKA, 'a connection of peering agreements between transit isp's and their customers). Peering agreements have long been fairly informal so long as the traffic engineers for each entity can verify that the bi-directional flow is mostly equal in either direction; as to avoid providing welfare services to another network without a mutually beneficial amount of reciprocation (traffic flowing the OTHER way). As traffic-engineering expands beyond what used to be comprised mostly of MRTG and NetFlow with a higher-level granularity; where disputes only usually consisted of whether various broadcast types, headers and other types of overhead should be included in the netflows. NOW, there are packages so granular being put into place on each peering point, the reports being generated now are much more pomp and circumstance of the administrative type, that since they are on a much more high level of zoom; what USED to be a normal directional fluctuation of 1% for a few days now shows up on a fresh-face VP's desk in the form of "WE ARE PROVIDING HUGELY DISPARATE ADVANTAGES TO OUR PEER AT THIS POINT" and these VP's start shit without getting the proper context from the traffic engineers. I'm not saying this is the CAUSE for this instance, but this is why disagreements have taken an upward swing. The problem is now gaining vision since these trigger-happy execs are being put back into context by the engineers, and everything will even out. This particular instance in the news just happens to be an ANOMALY, and IS NOT A HARBINGER OF THINGS TO COME. so STOP the doomsday crap unless you have inside information; otherwise your claims are baseless and alarming. There is TOO MUCH INVESTED now in the internet as a transit for business operations that corporate america will NEVER, EVER allow a segmentation of the net even in the worst case scenarios, which is NOT what this is. It's a one-off, so just chill. Even if Cogent NEVER peers with L3 anywhere ever again, there WILL be alternate paths and capacity buildouts to take up the slack created by these two organizations acting like b****es.

FURTHER, the author is not entirely correct in saying that single-homed computers downstream of cogent cannot talk to single-homed computers downstream of L3. While it's true the AS_PATH such that the traffic cannot currently cross directly from cogent to L3, it's NOT true that single-homed cogent customers cannot talk to single-homed L3 customers. FALSE. There are almost definetely SEVERAL other [AS_]paths to practically ANY network downstream of cogent OR L3. The lowest-cost path may be currently inoperable, and may forever be inoperable, but another AS_PATH will take it's place, and the capacity shift WILL be dealt with by the infrastructure engineers at whichever isp(s) have assumed the next best path. The author ought to correct this as his premise violates the very technique by which the internet offers redundancy.... alternate paths in the case of severed links.

Chill peeps

G's up Backhoes down

Re:Interesting scenario, though most likely untrue

[malakai]

It's more complicated that you make it out to be. Even if you connect to a large ISP (like NYC Time Warned Road Runner) you are shit out of luck right now. Not because they don't have a way of routing around the break, but beacuse they aren't prepared to implement such a drastic change. It's not all automatically controlled like people think. Most companies, like Time Warner RR would need to modify and reload hundreds of routers to effectively use some other connection point to get around the current 'block'.

And it's been about 8 hours and they still haven't.

At this stage, you'd be better off with a smaller ISP, because they have fewer connection points to update with the new routing table rows.

Re:Question from the clueless

[WuphonsReach]

Depends on where you're connecting from. I was able to hit Penny Arcade from Long Island, NY but not from Pennsylvania or Virginia.

I've seen a few other sites (ucomics) that are on the "other side of the rift" that I haven't been able to get to today. Fortunately, the office was not one of them (or I'd have had to drive the 5 hours into work).

Actually, the problem is because....

[scronline]

Cogent seems to have a problem policing it's network. That traffic keeps hitting L3 and they are tired of it. That's overly simplified, but yeah, cogent is not the best for policing it's network...I would have a hard time peering with them.

Now, since there is apparently some lack of understand of what peering is by the author of the article.... Peering is when 2 companies run a line between themselves. They aren't selling bandwidth to each other, but they share the line cost and traffic between them only goes between them. It helps speed up routes and in many cases helps avoid bottle necks at some of the central hubs.

Actually, for small business the economics change

[CFD339]

Important production servers need dual homed, highly reliable connectivity. Public facing servers are a commodity. The commonality of blade servers and big data center technology are escallating this.

Case in point: I've run my own data center for 12 years (18 if you could dial up bbs crap). This week, I'm shutting it down. I need more reliability for an important application, and it will be cheaper for me to outsource the public facing side to a data center (In my case, linux boxes at ServerBeach -- I can plug them, they've made me happy).

This is coming from someone with 13 years running his own shop; who owns good firewall, routing, and standby power equipment; as well as servers. Still, it will be cheaper from month 1 to outsource today. For less money, I don't have to buy (or maintain) hardware, get more bandwidth, multi-homed servers, way more reliable power and facilities, and a lower power bill.

The market is changing. More and more consumer broadband utilities (which is what they are) will have to drop out of the single homed dedidcated circuit market. Dissagree? Time Warner doesn't. Why do you think they're building state of the art colocation facilities and datacenters in the markets they serve?

Because soon public facing servers for any serious purpose will live primarily in big datacenters. The only companies to host their own, will be hosting them in their own big corporate data centers.

Corporate Silent Treatment

[Jazzinjake]

In Israel and the Palestinian Territories, this issue has been an ever-present one with regard to the cell phone companies there. While I was spending time there, many activists who were working in Palestine would get two SIM cards, (Cellcom, and either Orange or Jawwal), because Cellcom (Israeli) and Jawwal (Palestinian) wouldn't talk to each other. Furthermore, you couldn't call a Cellcom phone from a Palestinian (Paltel) land line.

When I asked for an explanation of this, it had to do with a corporate silent-treatment of sorts; because Paltel/Jawwal (the Palestinian telco) was suing Cellcom for licensing infringement and illegal operation, the Cellcom network decided to boycott the Palestinian phone carriers. This caused all sorts of problems for Palestinian society, and the effect was that everyone in Palestinian areas were ditching the local telco and getting Israeli Cellcom cell phones. Jawwal was facing dire times, after their offices were raided by Israeli military and tech imports were prevented because of blanket security concerns.

For folks on the ground, this was just one more manifestation of the intifada/occupation, even the corporations were going at it.

More background available here, here and here.

Re:Level 3's official statement

[xsonofagunx]

They're saying Cogent is intentionally not advertising routes to them via other providers, presumably because they're upset about not having a peering agreement in place. Anyone affected by this presumably needs to harass Cogent. I think you read that wrong, or maybe I am. What I'm seeing is We [L3] disconnected someone [Cogent]. Cogent, you might be a little pissed, and decide not to allow our traffic to go over your lines (go figure...). If a peer [Cogent, again] doesn't find another peer to route their traffic through, then their customers won't be able to access our network. If you're a customer of that peer [Hi, Cogent!] you're screwed, so hook up with us, or one of our partners. I'm reading it as L3 trying to dick Cogent around. Maybe that's just me. I don't care either way so far, haven't been denied access to anything yet. We should take this as a note that we should start the internet over, as a totally distributed network. Every country has a main link (of course, there are several of these for every decently sized country, or you'd kinda be defeating the distributed part...), and every state/province has their own peering to that link, and cities down from there etc. - all government owned, and agreed internationally (UN perhaps... tho they've been kinda missing with Bush in office...) that under no condition can they ever intentionally sever the link. I would even think that, government owned and funded, it would be almost a trifle for 10mbit internet connections to be had by all. Of course, the current providers would still be offering residential and business service, but the money would be going to the government, to help cover the costs of the network (along with federal subsidies), instead of Tier1 ISPs.

Inexcusable

[OldManAndTheC]

A poster on Broadband Reports said it best:

If hackers or terrorists caused this kind of disruption in the internet the government and media would freak out.

Re:This is bad. Very bad.

[malakai]

Your missing the point that a very large amount of home users via cable modems and DSL are affected. I understand you are not, and that's great, but put yourself in the shoes of 10s if not 100s of thousands of home users that can't hit sites they check on daily. Or can't VPN to their work/clients (I can't hit 3 of my clients). Should I be required to bear the cost of an extra DSL line on top of my business cable line? The cable provider promised me access to the internet, the whole internet. What L3 is doing by filtering out all the route advertisements for alternate paths is preventing many setups from even routing around the break. That's uncalled for.

The fact this effects 95% of NYC cable modems is going to piss off a lot of execs at many different companies and bring more light to this situation then L3 or Cognent can imagine. I've gotten calls from lots of clients who want to know if they should be calling Time Warner board members because they can't VPN into their office from home.

Having to explain to them it's not really a Time Warned RR issue and who's 'isssue' it is should not be any of ours job. This should not happen. Dirty pool is being played and it's crippling the Internet for a large number of users.

And there's not a damn thing* anyone can do about it.

(* actually, i saw a suggestion by someone to download the Google WiFi beta VPN client, and use it to add a second route to your home PC, via the Google datacenter pathways to the fractured side of the net. That this is the only recourse is very scary)