ISP Dispute Causing Connectivity Issues for Customers

I Don't Believe in Imaginary Property writes "A peering dispute between Telia and Cogent is causing routing and connectivity problems for many internet users. Cogent shut down their connections to Telia over what they described as a 'contract dispute' over the size and location of their peering points. Telia attempted to route around the problem, but Cogent blocked that, too. This has caused a lot of trouble for sites which are not multi-homed. Groklaw, for example, is on a Cogent network (MCNC.demarc.cogentco.com), so any Europeans connecting via Telia can't get through."

Comments

That's what happens...

[Doug52392]

This just goes to show you what happens when the money obsessed CEOs of corporations argue: The customers lose!

First post btw :)

Re:That's what happens...

[xstonedogx]

The thought of them arguing is much less frightening to me than the thought of them holding hands and skipping through a field of daisies together. ...for a couple reasons.

How much for only half an Internet?

[davidwr]

If I'm paying $50/month for Internet access, do I get half of that back if I can only get to half the Internet?

This isn't a silly question:
If YOU are the ISP, and YOUR actions are causing ME to not be able to get to SOMEONE ELSE, then my lawyers will try to hold YOU responsible.

Stupidity like this will cause both companies problems with their customers in court and in the marketplace.

Re:How much for only half an Internet?

[bagboy]

Do you people even read your TOS? You are not guaranteed anything without an SLA.

Re:How much for only half an Internet?

[Spazmania]

After the Cogent/Level 3 spat a few years ago, smarter network engineers realized it wasn't safe to use either Cogent or Level 3 as their sole Internet provider. Second provider? Sure. But not sole.

After this Cogent/Telia spat, no one with a brain will pick Cogent as their sole Internet provider.

This won't hurt Cogent too deeply. They charge so little for bandwidth that it's hard to resist picking them as your #2.

Re:How much for only half an Internet?

[Brian Gordon]

All ISPs take you to the same internet, so why pay more than you have to! :)

Re:How much for only half an Internet?

[rucs_hack]

If YOU are the ISP, and YOUR actions are causing ME to not be able to get to SOMEONE ELSE, then my lawyers will try to hold YOU responsible.

Are you a coder? It's just that your post resembles an SQL statement.

Re:How much for only half an Internet?

[fm6]

Stupidity like this will cause both companies problems with their customers in court and in the marketplace. I don't think a few disgruntled Swedish users are going to have much of a legal or economic impact on Cogent. Telia certainly will suffer, but they're not the ones that pulled the plug. According to Cogent, this is all Telia's fault for not being a good peering partner. But there really ought to be a better way to settle this than disrupting Internet access for millions of people.

What really has me concerned is that Cogent is choosing to punish Telia beyond simply shutting down the peering points. They've blocked all traffic that originates from Telia's network even if it comes through a third network. Doesn't that violate their peering agreements with the third networks? And isn't it dangerously like censorship? Perhaps someone should ask the FCC.

Re:How much for only half an Internet?

[Anonymous Coward]

I would say it's not safe to even use Cogent or Level 3 period after more than 5 years of dealing with them both extensively. Too many peering issues coming out of nowhere.

Re:How much for only half an Internet?

[budgenator]

Well at least with European Grade Broadband you can get nowhere really fast!

Re:How much for only half an Internet?

[cgenman]

They charge so little for bandwidth that it's hard to resist picking them as your #2.

Coincidentally, they've also chosen you for their #2.

Re:How much for only half an Internet?

[Minupla]

Yes, but since the customers of these companies tend towards the type of customers who do pay for SLAs (ISPs, companies rather then home users) I think the point is valid. Personally I've never used either of them as a provider, so I don't know how their SLAs are written, and they probably don't provide any assurances beyond their boundary, but I think an argument could be made that since the problem is demonstrably an issue within their control (a contract dispute) that the SLA should hold.

Min

Re:How much for only half an Internet?

[Detritus]

The TOS won't always get them off the hook. Claims made in ads can be considered part of the contract, even if they are disavowed in the TOS.

Re:How much for only half an Internet?

[Anonymous Coward]

I think it's time you showed those turds who's the boss.

Re:How much for only half an Internet?

[Neil Hodges]

SELECT * from SLASHDOTTERS where SQL_KNOWLEDGE = 1;

There, fixed it for you.

Re:How much for only half an Internet?

[adolf]

SLA == Service Level Agreement. Most Slashdotters seem to think that an SLA means something like "OMG! Wonderful fat pipe all for me!!!", but it's just a contract, much like the TOS that the same Slashdotters seem to blame for everything including world hunger.

In fact, since they're both just contracts, either one can be good for the customer, or bad for the customer. The only innate differences are three words at the top of the page, which is about as insignificant a distinction as I can think of.

Re:How much for only half an Internet?

[lth]

Spot on. Until now we've been relying on Cogent as our sole internet provider, which we've been cursing ourselves for the last couple of days.

We're a pretty large IT cooperation between colleges and business colleges in denmark, and this bit of fun has just meant that around 20% of our users can't reach our servers over the internet.

And what awfull timing, almost ruining easter holidays. Lot's of overtime setting up a new internet connection parallel with the one we've already got and the internal routing hell that then ensues when you're not multi homed.

Well. We've learned our lesson and are taking steps to become multi homed as quickly as we possibly can. I don't know if we'll consider Cogent as a partner in the future. They may be cheap but this kind of idiocy is hardly confidence inspiring.

Again?

[Constantine XVI]

Didn't this happen a few years back? Level3 and Cogent, IIRC

Re:Again?

[Cervantes]

Didn't this happen a few years back? Level3 and Cogent, IIRC Wow. It's almost like you read the article or something...

Yep

[iminplaya]

Quite a house of cards our fragile infrastructure has become. Somebody says "bomb" in San Francisco, and your flight from Mobile to Nashville will be grounded. A disagreement over the price causes droughts and blackouts in California. And our super robust internet can cut off whole countries with the snip of a cable or a flip of a switch. That's no way to run a circus, I say.

This message was brought to you by... BIGCO...where the nose meets the grindstone.

Re:Yep

[morbiuswilters]

"Super robust Internet"? Good God, you must be one of those people who think the Internet was originally designed by the military to survive a nuclear attack. The Internet has always been fragile and highly dependent on centralized routing. It's a shame these two companies can't work together, but there are plenty of providers who have more respect for their customers. This isn't a conspiracy to undermine your rights, it's the inability of two for-profit businesses to act in the best interests of the customers who pay their bills. It sucks but it happens and we move on.

Re:Yep

[QuantumRiff]

You'll notice that none of these are the faults of the technology, but the faults of the Humans (or lawyer/accountant equivalents).

Re:Yep

[iminplaya]

Never said anything here about conspiracies or rights. This is merely the result of the proverbial "too many eggs in one basket". or conversely, "too many chefs..." It's why we need good, efficient government services to prevent these companies from taking down the whole thing. We could have that if we simply demanded it. And these piddly arguments would pass unnoticed outside of the belligerents' offices. If the service is critical enough, then the government should step in and tell them to turn the switch back on. Just like when it orders strikers against an airline back to work, but never orders the company to pay the workers what they demand, or when it bails out the bank to prevent economic disaster, but never zeros out a person's mortgage. Funny thing that, the merchants' interests always take precedence over all else, and we're stuck with the lousy service and high prices.

It seems that according to the summary and the article, that there aren't plenty of providers to take up the slack. We're guessing it's because Cogent eventually slammed the door shut on these alternate paths to their network from Telia, since none of Cogent's customers accessed Telia via alternate routes during this time. We shouldn't permit this to happen.

Re:Yep

[fingusernames]

Um, I used to use this Internet thingy when it was ARPAnet, before the advent of private backbones. I remember HOSTS.TXT and the real InterNIC. And yes, it was originally designed to route around major failures. That was one of the reasons DARPA, e.g. the military, funded it. It may have not done it perfectly, it may not have been able to survive a full-on nuclear conflict, but it was certainly designed and funded in good part as a research project into network robustness in the face of catastrophe.

Ever since the backbones went private though, all bets are off. You are entirely correct as of the early 90s. As we all know, it's "my network, my rules." Hence this peering spat, and the ones before, and the ones to come.

Larry

This doesn't seem too crazy to me...

[morbiuswilters]

The Internet is built on cooperation. If two companies can't agree on how they will connect, then it seems they have that right. Just like their customers have the right to move to a different provider. Personally, if I was seriously affected by this I would never do business with either of the involved parties again. Hopefully people will leave and that will push them to negotiate, but I don't think they should be forced to work together if they don't want to.

Re:This doesn't seem too crazy to me...

[QuantumRiff]

You know, thats not true. In my area, I can choose Qwest DSL, charter cable, Clear-wire, small ISP's, etc. Every single one of them uses Qwest's fibers out of town. If Qwest gets into a spat with somebody, I can't access the internet, regardless of which ISP I am using locally. Keep in mind, I sit in a town that is on a main fiber route for williams, level 3, and a few others along the west coast, but none of them will sell any access locally. (were apparently too small of fish, which is a shame, williams cable has a set of buildings holding equipment about 100yards from where I am now sitting)

Internet is vital now...

[JustinOpinion]

There was a time when the Internet was more like a novelty or hobby project. Those of us using it were on the fringe, and nothing that we did on the 'net was vital.

That is no longer the case. The Internet has grown to become a vital infrastructure. Just about every business relies on the Internet to get their work done. It is an indispensable tool for students and academics. It has risen nearly to the status of roads or electrical power in terms of being depended upon by billions of people.

What's my point? My point is that with respect to most utilities (roads, water, electricity, phone) we wouldn't tolerate much interruption in service... and we certainly wouldn't accept companies squabbling as a decent excuse for degrading the infrastructure. Can you imagine driving to work one day and finding roads blocked because of a contract dispute?

I'm not sure what the answer is. Turning the Internet into a government utility has its own problems. Similarly, laws which require certain norms for the utility may be over-reaching or impotent. But, ultimately, we need to push for this critical infrastructure to no longer be treated as a best-effort hobby/entertainment service. We need companies (and possibly legislators?) to acknowledge that the Internet is critical, and that this means that uptime/bandwidth/QoS must be maintained at a high-level.

Re:Internet is vital now...

[morbiuswilters]

Um, the Internet is surely important, but I wouldn't suggest it is more critical to survival than roads or food, both of which seem to be handled quite fine by private enterprise. And I take it you have never been involved in a traffic jam, because this kind of crap happens all the time in the real world. Yeah, it bites, but there are plenty of businesses who may hundreds of thousands a month of connectivity that will not be amused by this. I expect repercussions for the involved ISPs. The "answer" to me is to realize that sometimes people or organizations get into stupid disputes and it inconveniences people, but that people will find a way to work around it. This cannot turn out well for Cogent or Telia.

Re:Internet is vital now...

[ScrewMaster]

Um, the Internet is surely important, but I wouldn't suggest it is more critical to survival than roads or food

I would, because the organizations which provide us with food and other necessities are dependent upon the Internet. I doubt the average interstate trucking company would have any idea how to operate without the Internet and GPS. The entire supply chain is utterly dependent upon modern communications, from production to delivery. The tech just makes everything so damned efficient that we've largely forgotten how to get along without it. I think we're starting to see how dangerous that can be, given the caliber of the folks running said communications.

In any event, the way to handle the likes of AT&T/SBC, Comcast and the rest is very simple: it's called standards. That worked very well for the phone system for a hundred years: AT&T (the old AT&T) built out the most reliable communications system on the planet, but that's because they were a heavily-regulated monopoly which had enforced quality-of-service standards. Comcast and the rest can provide almost no service at all for what we pay them and they get away with it.

Unfortunately, the government itself is so corrupt that it's unlikely Congress would ever be able to implement any kind of ISP regulation that has teeth to it, much less enforce it. Hell, they fucking gave away some hundreds of billions of dollars to these assholes, and never bothered to ask for an accounting of where the hell it went.

Re:Internet is vital now...

[mi]

Can you imagine driving to work one day and finding roads blocked because of a contract dispute?

Why, yes I can — the government-owned New York subway was gripped by just such a problem recently (in 2005). Millions of people were affected — getting to work was a nightmare...

In more Socialist countries (such as France) subway and other vital infrastructure is routinely shut down due to strikes (which are contract disputes between workers and employer). I was actually hit by such a strike myself — on that one week I was in Paris — and had to walk through the streets smelling of rotting garbage, because garbage collectors were on strike too — no kidding...

If people don't want to do their job for some reason, there is no way to force them. It was already illegal for New York transit to strike, but they did it anyway. For another example, when the policemen feel, they aren't treated nicely, they strike too. Although it is illegal for them to strike (obviously), you can not stop them from calling in sick (the special term is "Blue Flu"). For yet another example, flight controllers can't strike either, yet they had to make Reagan famous by striking — and disabling an even more important part of the country's (world's!) infrastructure...

These things will happen...

Re:Internet is vital now...

[nogginthenog]

Happens all the time in France.

Here in the UK we even have a special car park for when the French port workers strike:
http://en.wikipedia.org/wiki/Operation_Stack

Operation Stack is the codename used by Kent Police and the Port of Dover in England to refer to the method of using sections of the M20 motorway in Kent to park lorries when the English Channel or Dover ports are blocked by bad weather or industrial action. It has been implemented over 75 times since its inception 20 years ago.

Re:Internet is vital now...

[glitch23]

I would, because the organizations which provide us with food and other necessities are dependent upon the Internet. I doubt the average interstate trucking company would have any idea how to operate without the Internet and GPS.

You say that like those companies didn't exist prior to the Internet and GPS capability. They have existed for decades and did just fine. They are only more efficient now, as you said, with the technology available. If it went away they would just have to adjust by going back to the way they did business in the past. They wouldn't like it but they would survive because every other company would have to do the same so it wouldn't be like one company would go back to being less efficient than another. They would still be on equal footing as far as costs are concerned. If anything, the smaller companies who may not be able to afford some of the technology that the bigger companies can afford would have a better chance of survival.

Re:Internet is vital now...

[glitch23]

*cough*retard*cough* I guess all the people who starve while the trucking companies that ship food to the grocery stores in the cities adjust back to paper are just a little business hiccup.

People did not die just because old fashioned paper/pencil was used. Companies were not incompetent just because they had to do things without computers. They are incompetent for other reasons. If you are going to sling names you should so with your real username too; it might just make your high school name calling a little more credible.

Re:Internet is vital now...

[fingusernames]

Note what you wrote:

"Comcast and the rest can provide almost no service at all for what we pay them and they get away with it."

Note, "what we pay them." We pay them prices based on competitive forces, where reliability is just one factor. Granted, Comcast may not be the best example. But think in general.

The way the phone network got so reliable was because we granted a monopoly, and granted guaranteed, predictable profits. If it cost X to get the standards required, fine -- it was paid for, and there were *always* profits on top. That is key. We cannot have our cake and eat it too. Fast, good, cheap. We all know it. Look at the power companies. We have politicians pushing populist agendas to freeze rates, limit profits -- and the result is that the private companies running the power grid simply cut back elsewhere, and we have power outages and very little new power generation (of course that is also to be blamed on NIMBY opponents).

Perhaps we should have some more oversight of the Internet today, some sort of oversight board. But even if we did have that, would it or could it prevent the peering spats? Should it? What would a review board do in the case of an American company and European company with a contract that wasn't being honored? Would we need some sort of United Nations entity? How would this impact innovation and interest if decisions had to be brought before regulatory entities, subject to public comment, so on. If companies simply cannot depeer and make it actually hurt (what Cogent did by blocking traffic), then where do the incentives come from to provide the peering agreed upon? If we regulate it and mandate reliability, will we also regulate and mandate paying the true cost, along with a healthy enough profit to make sure a private entity remains interested in maintaining the network and providing for future growth and capacity? What will lack of competition do to cost, and market penetration? Will regulation drive away the private investors who fund these companies in hope of turning a healthy profit? Will we all pay for it via higher service fees or taxes? Is it perhaps ok to have these occasional spats, if the end result is a reasonably robust network at the "best" price? Or should the whole thing just be one big government funded and controlled system paid for by taxes and usage fees? Do we trust the government? Where would the innovation come from?

Perhaps what we have is the best of many imperfect possibilities?

Larry

Re:Internet is vital now...

[Dan541]

I don't think you understand what your talking about.

That was THEN this is NOW.

There is a big difference, the systems we use now would not cope without the Internet because it is now an integral part of the system, you cannot simply flick a switch and change the way companies operate.

Change takes time!

~Dan

Also affects WoW players...

[WinterSilence]

Also no one playing World of Warcraft using Cogent as ISP can connect to any WoW servers, since Blizzard use Telia's backbone...
This is listed in-game in WoW currently at the login screen.

Re:Also affects WoW players...

[Omnedon]

Entropia Universe uses Telia and I am sitting here in Michigan with an idle machine that I bought specifically to play that game. Not just Scandinavia "unhappy".

Tell it like it is: whoever's wrong, get over it

[postbigbang]

It's like an old telecom SS7 spat. Tell them to get over it. In three more days, we pull all our servers from and move on. Can't get to what we need? As ISPs, they have precious little time to figure it out, then we split. Go ahead, try and enforce that five-9's contract. Providers are everywhere, drooling for business. Bye-bye, blackout. Hello loneliness.

Third Parties

[EverlastingPhelps]

Sounds like Verizon and Blizzard need to fire up the old legal teams and start filing tortious interference suits on Cogent.

Death throws?

[davolfman]

In my limited experience de-peering like this usually precedes an ISP death. Other people have probably figured this out so it wouldn't surprise me if this is having a negative effect on stock prices. It makes you wonder why anyone would ever consider it a valid option if they aren't just a rat jumping ship. It just looks bad.

Re:Route around?

[dave562]

You're missing the fact that at the upper tiers of the internet, there are only so many routes available. There are simply somethings that can't be routed around because the ONLY route to where you want to go involves passing packets across the network you are trying to route around. Consider a smaller example. You want to route traffic to a Verizon DSL customer. Verizon has decided it doesn't want to pass your packets to the DSL customer. No matter how you try to route it, since Verizon sold the DSL service and controls the last few hops in the route, you simply can't route to the customer any other way.

The current issue involves "peering arrangements/agreements." Do a Google search if you want an in depth explination of what exactly a peering arrangement is all about. The short version is that ISPs agree to pass each others traffic across their networks. That's the way the internet works. Every ISP can't have a router in every place that a router needs to be placed. So they "share" each routes with each other.

Also affects email traffic in the US & Europe

[vinsci]

Since Cogent actively drops any traffic that's been even just in transit anywhere on the pretty big TeliaSonera International Carrier network (it's a tier 1 net that covers all of the US and Europe), your email messages will just be held at some random backup email server for a couple of days until you'll get a return notice saying your message hasn't been delivered yet. If you're lucky that is.

For any important/urgent emails, you now need to make a follow-up phone call, just to see if the message was delivered. (Yes, you could request a receipt when the message is opened, but it's optional for the receiver to send the receipt and many don't).

I hope that ibiblio & the internet archive (archive.org) are moved away from their current hosting on Cogent's network, urgently.

Great timing to send urgent business email, normally delivered within seconds, only to find out that it has never been received. I do wonder if this active sabotage of 3rd party Internet traffic might be class-actionable. Of course e-mail is just a tiny part of the overall losses that 3rd parties suffer from this.

The need for BAPPs (Big-Ass Peering Pipes)

[1sockchuck]

According to Wired, Cogent felt Telia didn't provide "fat enough pipes." The capacity of peering connections is becoming a point of tension in a growing number of peering relationships. Video traffic is driving strong demand for 10 gigabit Ethernet connections for peering, but some major ISPs are apparently reluctant to upgrade, asserting that the financial benefits of big-pipe peering don't offset the short-term expense of network upgrades needed to support 10gigE. The economics of peering is a tricky business sometimes, and video traffic is complicating the equation.

Re:Route around?

[jroysdon]

Cogent's customers need to sue Cogent over this. It's fine if AS174 (Cogent) doesn't want to accept routes that include AS1299 (Telia). However, if AS174 announces AS81's prefixes to its peers, which in turn peer with AS1299, then it must accept all traffic to AS81, as they have a contract agreement (customer or peer) with AS81 (where groklaw.net is hosted) and with the intermediate peer. It doesn't have to give AS81 any routes to AS1299, and AS81 has other peers that can route the traffic to AS1299, so the return traffic doesn't even need Cogent.

Cogent is breaking things by announcing a prefix and then blocking traffic to it (in AS81's case) if it comes from an AS they don't like. Or, it may be that the downstream customers are just using default routes and blindly sending traffic for AS1299 which AS174 is just dropping.

However, if Cogent is sending a default to customers, they have an obligation to learn all prefixes available from any peer they have, no matter the originating AS.

Shame on Cogent. Play by the rules. You don't have to peer with Telia, but honor the peering agreements you have for other customers to transit to any peer that has a peering agreement to get to Telia.

Works fine...

[swehack]

I'm in Sweden on two connections, one bahnhof, which rents most of it's fiber from Telia, and one IP-Only which has it's own atlantic cable, both work fine against Groglaw for me. Which is funny really because i know Telia owns most of the fiber in Sweden and that Bahnhof for example rents most of it's fiber and equipment from Telia.

Re:How to get by these silly commercial blockades

[knarf]

Replying to my own posting here: Garden Networks' GTunnel works with wine on Linux so if you don't feel like setting up a Tor node and don't want to hunt for anonymizing proxies on the web you can use that instead. If you add the Switchproxy or (preconfigured for GTunnel etc.) GProxy extension to Firefox you can switch between your normal net connection (with or without proxy) and the anonymizer.