Slashdot Mirror
ISP Dispute Causing Connectivity Issues for Customers
I Don't Believe in Imaginary Property writes "A peering dispute between Telia and Cogent is causing routing and connectivity problems for many internet users. Cogent shut down their connections to Telia over what they described as a 'contract dispute' over the size and location of their peering points. Telia attempted to route around the problem, but Cogent blocked that, too. This has caused a lot of trouble for sites which are not multi-homed. Groklaw, for example, is on a Cogent network (MCNC.demarc.cogentco.com), so any Europeans connecting via Telia can't get through."
This just goes to show you what happens when the money obsessed CEOs of corporations argue: The customers lose!
:)
First post btw
If I'm paying $50/month for Internet access, do I get half of that back if I can only get to half the Internet?
This isn't a silly question:
If YOU are the ISP, and YOUR actions are causing ME to not be able to get to SOMEONE ELSE, then my lawyers will try to hold YOU responsible.
Stupidity like this will cause both companies problems with their customers in court and in the marketplace.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Didn't this happen a few years back? Level3 and Cogent, IIRC
"I think an etch-a-sketch with an ethernet port would beat IE7 in web standards compliance."
Look at the comment below this...
:)
Sorry, I was the First Poster
HA HA HA lol
This reminds me of that time I let Billy borrow my baseball bat. Only, he wouldn't give it back so I called him a doo-doo head, and kicked him in the shins. He then told his parents, and they decided to tell my parents what a poor sport I was being. My dad put a stop to that quick and demanded the bat back and told them to stop being a bunch of cheap bastards and buy Billy a baseball bat. Once they finally ponied up and bought one, we started hanging out again and were best friends forever!!!
The End.
/sigh...
Quite a house of cards our fragile infrastructure has become. Somebody says "bomb" in San Francisco, and your flight from Mobile to Nashville will be grounded. A disagreement over the price causes droughts and blackouts in California. And our super robust internet can cut off whole countries with the snip of a cable or a flip of a switch. That's no way to run a circus, I say.
This message was brought to you by... BIGCO...where the nose meets the grindstone.
What?
The Internet is built on cooperation. If two companies can't agree on how they will connect, then it seems they have that right. Just like their customers have the right to move to a different provider. Personally, if I was seriously affected by this I would never do business with either of the involved parties again. Hopefully people will leave and that will push them to negotiate, but I don't think they should be forced to work together if they don't want to.
I have come here to chew memory and kick ass... and malloc() is returning a null pointer.
There was a time when the Internet was more like a novelty or hobby project. Those of us using it were on the fringe, and nothing that we did on the 'net was vital.
That is no longer the case. The Internet has grown to become a vital infrastructure. Just about every business relies on the Internet to get their work done. It is an indispensable tool for students and academics. It has risen nearly to the status of roads or electrical power in terms of being depended upon by billions of people.
What's my point? My point is that with respect to most utilities (roads, water, electricity, phone) we wouldn't tolerate much interruption in service... and we certainly wouldn't accept companies squabbling as a decent excuse for degrading the infrastructure. Can you imagine driving to work one day and finding roads blocked because of a contract dispute?
I'm not sure what the answer is. Turning the Internet into a government utility has its own problems. Similarly, laws which require certain norms for the utility may be over-reaching or impotent. But, ultimately, we need to push for this critical infrastructure to no longer be treated as a best-effort hobby/entertainment service. We need companies (and possibly legislators?) to acknowledge that the Internet is critical, and that this means that uptime/bandwidth/QoS must be maintained at a high-level.
I hope they settle this dispute soon, because it has affected me several times in the past week.
I live in Europe, and am the co-administrator of Phantasy Star Cave. One day I couldn't access it for hours, so I traced the domain, and telia was the node it stopped at. So when I saw this story I was like "That's it! That was the problem!".
Also no one playing World of Warcraft using Cogent as ISP can connect to any WoW servers, since Blizzard use Telia's backbone...
This is listed in-game in WoW currently at the login screen.
What kind of dog barks "BOFH! BOFH!"? A rootweiler of course...
It's like an old telecom SS7 spat. Tell them to get over it. In three more days, we pull all our servers from and move on. Can't get to what we need? As ISPs, they have precious little time to figure it out, then we split. Go ahead, try and enforce that five-9's contract. Providers are everywhere, drooling for business. Bye-bye, blackout. Hello loneliness.
---- Teach Peace. It's Cheaper Than War.
Sounds like Verizon and Blizzard need to fire up the old legal teams and start filing tortious interference suits on Cogent.
Isn't the Internet designed to route around blockages/outages (which, really, are more or less the same thing from the network pov..)?
Am I missing something?
I am the maverick of Slashdot
In my limited experience de-peering like this usually precedes an ISP death. Other people have probably figured this out so it wouldn't surprise me if this is having a negative effect on stock prices. It makes you wonder why anyone would ever consider it a valid option if they aren't just a rat jumping ship. It just looks bad.
You said "whoever's wrong, get over it." The problem is that each side thinks the other side is wrong.
$x='S24;r)>63/* h@<5+oZ)32"5cz';$me='phroggy'x$];
$x=~y+ -xz+\0-Tx+;print$_^chop$me for split'',$x;
And the point is: we don't care. Cogent? Yank them. Telia? Ex-PTT that smells as bad as Deutsche Telekom (in this case, anyway).. Yank them. Misbehaving child-like ISPs? Goodbye. This 2008, not 1998.
---- Teach Peace. It's Cheaper Than War.
Spam filtering blocks my email constantly. I'm sure this is only a temporary outage.
I know that at least one company that's been affected (Cornered Rat Software, who run the MMOFPS World War II Online) are seriously considering getting an AS of their own after this, so that if nothing else, they will be able to say "Telia's traffic can get to us via this route" and bypass Cogent's pettiness. I'd cite, but it was a post by one of their guys on a subscriber-only forum.
Since Cogent actively drops any traffic that's been even just in transit anywhere on the pretty big TeliaSonera International Carrier network (it's a tier 1 net that covers all of the US and Europe), your email messages will just be held at some random backup email server for a couple of days until you'll get a return notice saying your message hasn't been delivered yet. If you're lucky that is.
For any important/urgent emails, you now need to make a follow-up phone call, just to see if the message was delivered. (Yes, you could request a receipt when the message is opened, but it's optional for the receiver to send the receipt and many don't).
I hope that ibiblio & the internet archive (archive.org) are moved away from their current hosting on Cogent's network, urgently.
Great timing to send urgent business email, normally delivered within seconds, only to find out that it has never been received. I do wonder if this active sabotage of 3rd party Internet traffic might be class-actionable. Of course e-mail is just a tiny part of the overall losses that 3rd parties suffer from this.
Trusted Computing FAQ | Free Dawit Isaak!
[...] They want to deliver vast amounts of information over the Internet. And again, the Internet is not something that you just dump something on. It's not a big truck. It's a series of tubes. Did this happen during Level3/Cogent delinking feud?
I understand about shutting down the peering points but aren't there some rules about actively blocking traffic?
According to Wired, Cogent felt Telia didn't provide "fat enough pipes." The capacity of peering connections is becoming a point of tension in a growing number of peering relationships. Video traffic is driving strong demand for 10 gigabit Ethernet connections for peering, but some major ISPs are apparently reluctant to upgrade, asserting that the financial benefits of big-pipe peering don't offset the short-term expense of network upgrades needed to support 10gigE. The economics of peering is a tricky business sometimes, and video traffic is complicating the equation.
It's been a while since I've lived back East. I don't remember highways in PA getting actually blocked, but there was always either construction going on which slowed you down, or roads that badly needed repair, so the potholes slowed you down. This is like having a barrier across the entire road and forcing you to drive down to Philly or up to Port Jervis.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
I'm glad I'm not paying Blizzard a monthly fee to play World of Warcrack.
I traced to oxford.edu (went through Telia) and stanford.edu (went through Cogent); interesting latency spikes and a few dropped packets when I ping both. Just started a ping to my ISP for a control.
At the end of the day, some things will be unreachable, but the Internet isn't indestructible anymore. Things will break, and we move on.
A good attitude if Telia's fixed & dchp'd clients don't mean anything to you. I get the feeling Telia's the one that's facing the most trouble. It's only a guess.
---- Teach Peace. It's Cheaper Than War.
I doubt anyone will move. Temporary network segregation is a price to pay for getting super-cheap transit. I'd rather have these little spats then get raped by "Tier 1 providers" such as Level3 who try to justify extremely high prices per megabit.
The question is whether they're willing to connect to each other for free, or whether they're going to charge each other money. In general, big carriers will interconnect for free (splitting the cost of the interconnection) if they're similar enough in size to hand each other relatively balanced traffic loads, or if they're playing in niche markets that complement each other. For instance, eyeball carriers like DSL and cable modem companies and big content providers like big hosting companies have an incentive to peer with each other, because the alternative is for both of them to pay a transit provider to interconnect them. But if they can't agree on terms, or can't make their connections work together, then they're not going to peer, and the fallback is that some of them may have to buy transit, either with the small ISP buying from the big one, or buying from some third ISP that connects to both of them.
Each ISP knows about the IP addresses of its own customers, and connects to other ISPs to exchange routing information. If two ISPs are peering, they're going to share address and route information for their own customers with the other ISP, so ISP A can reach ISP B's customers and vice versa. On the other hand, if ISP A is selling transit to ISP B, then A is going to tell B about all the addresses it knows how to reach, and how good the routes are, and B is going to send A packets for those addresses (and money.) In the general case, A knows how to reach every address on the Internet, either because the address is directly connected, or because A peers with that address's ISP, or because A pays for transit from some other ISP that knows how to reach it. (There are also exceptions and special cases, like national-monopoly ISPs.) And not everything's a pure case; one ISP might pay another to carry traffic for some routes but not others, or handle some traffic for free and pay for the rest.
I don't know quite what happened with Telia and Cogent here. Cogent mostly sells to content providers in the US and Europe; Telia's a more general ISP but I get the impression their customers tend to be end users and eyeball handlers. Cogent's side of the story seems to be that Telia's not maintaining their peering links correctly, so they de-peered with them and stopped exchanging traffic directly. If Telia's buying transit from some other ISP, that should let Telia's customers reach Cogent's customers. If Cogent's blocked that traffic now, that's weird; carriers don't usually do that on purpose.
This is sort of the opposite of the Cogent-Level3 fight of a few years ago. During that even, Level 3 decided that Cogent was no longer sufficiently useful to peer with, and dropped peering, which would force Cogent to either pay money to L3 to get the service, or else pay some third ISP for transit. This time it's Cogent dropping the other carrier.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
In the grand scheme of things, Telia's fixed and dhcp clients are a small minority of the entire Net. Shouldn't Telia customers complain to Telia that they should upgrade their peering links with Cogent (as that is what this disagreement is about)?
In Scandia, they're big medicine. Let's see who drives off the cliff. Personally, I'd rather just route around them. Perhaps an evil subnet concoction would do it.
---- Teach Peace. It's Cheaper Than War.
Thought so. Great timing to send urgent business email, normally delivered within seconds, only to find out that it has never been received. You rely on email for critical communication ? Not a good idea. I do wonder if this active sabotage of 3rd party Internet traffic might be class-actionable. You suing ?
Thought so. Of course e-mail is just a tiny part of the overall losses that 3rd parties suffer from this. And if you have a contract with either level3 or teliasonera that includes sla provisions protecting you from this, call them. If you don't, switch your ISP.
I'm in Sweden on two connections, one bahnhof, which rents most of it's fiber from Telia, and one IP-Only which has it's own atlantic cable, both work fine against Groglaw for me. Which is funny really because i know Telia owns most of the fiber in Sweden and that Bahnhof for example rents most of it's fiber and equipment from Telia.
What we're dealing with here is actually a "First Fail!"
Congratulations on inserting just a little more entropy into the internet. Scream to the tune of the background noise.
For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
I hope this finally teaches people to get with the times and get themselves multihomed... ISPs and corporate connections likewise.
The Internet works by everybody paying their own share of the costs. I'm not paying for your connectivity costs, you're not paying for mine. Note that if you don't pay enough to cover your ISP's real costs, your ISP will start behaving like Cogent does and you're then part of the problem.
Of course I do and so does everybody else. Don't be silly. It's no different from relying on your fax machine (which will just scan the documents and which might succeed at sending them at first opportunty the receiver is able to receive it; or having relied on the Telex network way back when - what did you do when the Telex line went dead? Hint: there was nothing you could do). The only real alternative the Internet today for any significant amount of data that needs to be transmitted intact is courier dispatch, but you don't use that unless the amount of data is huge.
I'll be quite happy to join any class-action lawsuit against Cogent, thank you.
Keep in mind that unless Cogent both 1) claimed to route traffic from A to B and 2) then actively dropped traffic from A to B nothing bad would have happened. Now they knowingly mis-configured the routers to lie to other Internet carrier companies, thus causing this mass-outage. Surely it can't be too long before some entrepreneurial law firm picks up on this way of actively causing trouble for 3rd parties.
Trusted Computing FAQ | Free Dawit Isaak!
I hope that ibiblio & the internet archive (archive.org) are moved away from their current hosting on Cogent's network, urgently.
You paying ? Thought so.
The Internet works by everybody paying their own share of the costs. I'm not paying for your connectivity costs, you're not paying for mine. Note that if you don't pay enough to cover your ISP's real costs, your ISP will start behaving like Cogent does and you're then part of the problem.
So that is a no to you paying ibiblio and archive.org's move to a different provider then, got it.
Great timing to send urgent business email, normally delivered within seconds, only to find out that it has never been received.
You rely on email for critical communication ? Not a good idea.
Of course I do and so does everybody else.
Internally, MAYBE. Externally ? Well, they must've some shoddy IT department then if they can't make it clear that email is a best-effort service.
Don't be silly.
I'm not. You have no way of knowing whether your mail reached its intended recipient or whether it went into a spamfolder, got lost on the way, or is sitting in some mail server queue (as per RFC when transient connectivity problems happen). If you rely on it for urgent and critical business purposes, you damn well better have a backup-plan.
It's no different from relying on your fax machine (which will just scan the documents and which might succeed at sending them at first opportunty the receiver is able to receive it; or having relied on the Telex network way back when - what did you do when the Telex line went dead?
We are not way back when. You generally KNOW when a fax does not get sent (by way of a busy signal or such). You often also have out-of-band communication about the fax.
If your phone line dies, at least you know it did. And even then you can use a backup -- the plain old postal service.
Hint: there was nothing you could do). The only real alternative the Internet today for any significant amount of data that needs to be transmitted intact is courier dispatch, but you don't use that unless the amount of data is huge.
Or, you know, it's actually business critical and you have no way of knowing whether it would reach its intended target otherwise. Hint : email is not that kind of service.
I do wonder if this active sabotage of 3rd party Internet traffic might be class-actionable.
You suing ? Thought so.
I'll be quite happy to join any class-action lawsuit against Cogent, thank you.
But not actually bothered enough to explore it yourself.
Keep in mind that unless Cogent both 1) claimed to route traffic from A to B
and 2) then actively dropped traffic from A to B nothing bad would have happened. Now they knowingly mis-configured the routers to lie to other Internet carrier companies, thus causing this mass-outage. Surely it can't be too long before some entrepreneurial law firm picks up on this way of actively causing trouble for 3rd parties.
Uhm. They actually just dropped peering connections and dropped traffic to and from TeliaSonera's AS (by any route) at the border of their own network. "Third parties" affected are customers of Cogent -- and they should look at their contract language on whether there has actually been a breach of contract (usually, there isn't). Also affected are TeliaSonera customers trying to reach Cogent's network -- but then again, Cogent is not required to send or receive ANY traffic it does not deem fit for its network -- nothing illegal about that.
I agree it's a hell of a dickish move by Cogent, and they are known to seek confrontation. Then again, when you buy transit from Cogent, you either know that or you did not do your homework. If you don't want to put up with their antics, choose a different transit prov
Rebel Without A Pause
(disclaimer: Translated by me (poorly)
/translation
source: http://www.telia.se/privat/kundservice/telia_cogent.page?sl=teliase_aktuelltprv1
"Vad är det som har hänt?
Torsdag 13 mars kl 23.00 stängde Cogent, en amerikansk Internet Service Provider (ISP), förbindelserna mellan TeliaSonera International Carrier (TSIC) och dem. Frånkopplingen innebär att del av Telia kunder inte kommer att nå en del av Cogents kunder."
What have happened?
Thurs. 13th of March ar 11.00 PM Cogent, an american ISP, closed the connection between TSIC and them. The disconnection results in that some of Telia's costumers won't be able to reach some of Cogent's costumers.
Varför har problemet uppstått?
Det rör ett så kallat peering-avtal.
Why have the problem come about?
It has to do with a so called peering-agreement.
När blir problemet löst?
Vi kan ännu inte säga när problemet blir löst, men den viktigaste trafiken kommer gå via alternativa vägar.
When will the problem be solved?
We cannot say when it will be solved, but the most important traffic will go by alternate routes.
Vilka kunder drabbas och hur?
Du som kund kan drabbas om du försöker besöka en sajt som inte har någon "alternativ väg", det vill säga den nås endast via Cogent eller Telia. Detta är ett mycket begränsat antal webbplatser.
Which costumers are being affected and how?
You as a costumer can be affected if you try to visit a site that does not have an "alternate route", in other words it can only be reached by way of Cogent or Telia. This is a very limited number of websites.
Har ni styrt om trafik eller kommer ni att göra det?
TeliaSonera International Carrier har ingen möjlighet att styra om så kallad peering-trafik.
Have you re-routed traffic or will you do so?
TSIC has no possibility to re-rout so called peering-trafic.
Är det första gången som Telia har detta problem med Cogent?
Ja, det stämmer.
Is this the first time that Telia has this problem with Cogent?
Yes, that is correct.
I spent five days trying to find out why archive.org was down. Found out more by accident once an online friend told me his connection was fine. So now I rewrite all addresses that don't work with coralcdn.org's proxy. Maybe I shouldn't have mentioned this. Now the bastards will probably try to cut that connection, as well.
Could we please somehow replace these monolithic networks with peer to peer wireless connections or some such? There has to be some way of freeing our internet from the grasp of monopolists.
I didn't even notice. When I didn't get thedailywtf (which also seems affected), I just added .nyud.net and tried again - voila!
That'll work at least until cogent starts blocking every net that hosts someone who caches stuff, which in its turn can be served to teliasonera customers. But they can't be that silly, can they? [Touches wood]
Yes, I am a biological organism. All rumors to the contrary are just that, rumors.
How many times can they get away with lighting the drama bomb before it blows up in their face?
I remember a couple years back when this happened with Level 3 and Cogent. I was the network admin for a small company and we were unable to access our bank to make wire transfers. The bank was right across town. Can you imagine trying to explain to the higher ups that this was not our issue? GET THE CONNECTION FIXED NOW! Um, I can't? I didn't break it!!!!!
from Groklaw:
This posting comes to you through an anonymizing proxy. Not because I'm somewhere behind the Great Firewall of China or on the Microsoft campus in Redmond... but because Ibiblio's carrier (Cogent) has decided it does not want to peer with TeliaSonera anymore. So they blocked all traffic coming from or destined to TeliaSonera. When they found out that those pesky routers did what they were designed to do - route traffic around damaged nodes - they advertised some cheap routes and subsequently dropped traffic, thereby sealing the leaks. Leaving me, and many with me, without access to a substantial part of the internet. OK, everyone who can not reach Groklaw, please post here :-)
As I am actually posting here it is clear that there are ways around these commercial blockades, just as there are ways around political blockades [1]. Anonymizing proxy servers can be used by those hit by Cogent's last temper trantum until either Cogent and TeliaSonera make up or (preferrably) traffic is routed around Cogent.
If this type of behaviour is to be the future for the commercialised internet the need for services like those provided by Garden Networks or the Tor Project will grow. But the real question of course is whether this type of behaviour should be tolerated from a carrier. It essentially boils down to censorship, something which is not allowed in a common carrier as far as I know. If they had just refused to peer with TeliaSonera they would be in the right. Now that they actively attract and subsequently drop traffic they have crossed a line. If I were to be a Cogent customer I would seriously consider to move my business elsewhere or at least consider to relegate Cogent to the role of backup carrier. So Ibiblio, if you are reading this message from behind the wall...
[1]as predicted in many a cyberpunk novel the differences between politics and commerce continue to dwindle until they are all but indiscernible...
--frank[at]unternet.org
Telia is a very large transit provider in Europe with quite some content providers using them. It's a lot of end customers in Sweden, Danmark and Finland, plus endless amounts of datacenters and large customers all over Europe. They are actually the largest transit network in Europe, with their own net extending to Asia and the US.
In the Swedish press releases Telia stated that they didn't agree with Cogent on peering upgrades (peering sessions were running full).
This can mean anything from Cogent who didn't want to pay anything at all to Telia who didn't want to pay (because it's most likely that Cogent is SENDING traffic much more than receiving traffic from Telia network, since they have almost only content).
Neither TeliaSonera nor Cogent have stated who is not paying what, but two things are clear:
1. Cogent has been in peering disputes with Level3, AOL and FranceTelecom. All cases were caused by Cogent sending much more traffic than receiving and not wanting to upgrade and/or pay for traffic.
2. Telia stated clearly here that they wanted to upgrade the session but the costs were a problem here. If you can't agree on upgrade costs, why not drop the peering and play transit instead? Instead Cogent blocked them, whatever way traffic came in. Like in blackmail.
The answer to it all: money. Cogent doesn't have the bucks to pay for transit to/from TSIC and wants to play it cheap. If you sell 100mbit access for 1000USD, how can you have the money to seriously invest in things? Or to use transit to reach your destinations?
But it's not really that important this split. Anyone who still trusts Cogent as their sole bandwidth provider has been living on another planet for the last couple or years. It works both ways: TSIC customers can't reach Cogent. Cogent customers won't get any business from Europe.
--- Woohooo!
But who knew just how low one end of that spectrum is?
I guess thanks is in order to you, AC, for giving us all a reality check. Keep your minds in working order everybody--what you see above has happened to what was once an actual human being! Frightening.
It's affecting my MUD (hosted in Russia) playing too... My packets make their way over Cogento's network then suddenly hit a brick wall at the end of their space.