Slashdot Mirror
ISP Dispute Causing Connectivity Issues for Customers
I Don't Believe in Imaginary Property writes "A peering dispute between Telia and Cogent is causing routing and connectivity problems for many internet users. Cogent shut down their connections to Telia over what they described as a 'contract dispute' over the size and location of their peering points. Telia attempted to route around the problem, but Cogent blocked that, too. This has caused a lot of trouble for sites which are not multi-homed. Groklaw, for example, is on a Cogent network (MCNC.demarc.cogentco.com), so any Europeans connecting via Telia can't get through."
This just goes to show you what happens when the money obsessed CEOs of corporations argue: The customers lose!
:)
First post btw
If I'm paying $50/month for Internet access, do I get half of that back if I can only get to half the Internet?
This isn't a silly question:
If YOU are the ISP, and YOUR actions are causing ME to not be able to get to SOMEONE ELSE, then my lawyers will try to hold YOU responsible.
Stupidity like this will cause both companies problems with their customers in court and in the marketplace.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Didn't this happen a few years back? Level3 and Cogent, IIRC
"I think an etch-a-sketch with an ethernet port would beat IE7 in web standards compliance."
Look at the comment below this...
:)
Sorry, I was the First Poster
HA HA HA lol
Quite a house of cards our fragile infrastructure has become. Somebody says "bomb" in San Francisco, and your flight from Mobile to Nashville will be grounded. A disagreement over the price causes droughts and blackouts in California. And our super robust internet can cut off whole countries with the snip of a cable or a flip of a switch. That's no way to run a circus, I say.
This message was brought to you by... BIGCO...where the nose meets the grindstone.
What?
The Internet is built on cooperation. If two companies can't agree on how they will connect, then it seems they have that right. Just like their customers have the right to move to a different provider. Personally, if I was seriously affected by this I would never do business with either of the involved parties again. Hopefully people will leave and that will push them to negotiate, but I don't think they should be forced to work together if they don't want to.
I have come here to chew memory and kick ass... and malloc() is returning a null pointer.
There was a time when the Internet was more like a novelty or hobby project. Those of us using it were on the fringe, and nothing that we did on the 'net was vital.
That is no longer the case. The Internet has grown to become a vital infrastructure. Just about every business relies on the Internet to get their work done. It is an indispensable tool for students and academics. It has risen nearly to the status of roads or electrical power in terms of being depended upon by billions of people.
What's my point? My point is that with respect to most utilities (roads, water, electricity, phone) we wouldn't tolerate much interruption in service... and we certainly wouldn't accept companies squabbling as a decent excuse for degrading the infrastructure. Can you imagine driving to work one day and finding roads blocked because of a contract dispute?
I'm not sure what the answer is. Turning the Internet into a government utility has its own problems. Similarly, laws which require certain norms for the utility may be over-reaching or impotent. But, ultimately, we need to push for this critical infrastructure to no longer be treated as a best-effort hobby/entertainment service. We need companies (and possibly legislators?) to acknowledge that the Internet is critical, and that this means that uptime/bandwidth/QoS must be maintained at a high-level.
I hope they settle this dispute soon, because it has affected me several times in the past week.
I live in Europe, and am the co-administrator of Phantasy Star Cave. One day I couldn't access it for hours, so I traced the domain, and telia was the node it stopped at. So when I saw this story I was like "That's it! That was the problem!".
Also no one playing World of Warcraft using Cogent as ISP can connect to any WoW servers, since Blizzard use Telia's backbone...
This is listed in-game in WoW currently at the login screen.
What kind of dog barks "BOFH! BOFH!"? A rootweiler of course...
It's like an old telecom SS7 spat. Tell them to get over it. In three more days, we pull all our servers from and move on. Can't get to what we need? As ISPs, they have precious little time to figure it out, then we split. Go ahead, try and enforce that five-9's contract. Providers are everywhere, drooling for business. Bye-bye, blackout. Hello loneliness.
---- Teach Peace. It's Cheaper Than War.
Sounds like Verizon and Blizzard need to fire up the old legal teams and start filing tortious interference suits on Cogent.
Telia tried to route through other carriers. Cogent blocked this after half a day.
Disgusting if you ask me.
In my limited experience de-peering like this usually precedes an ISP death. Other people have probably figured this out so it wouldn't surprise me if this is having a negative effect on stock prices. It makes you wonder why anyone would ever consider it a valid option if they aren't just a rat jumping ship. It just looks bad.
You said "whoever's wrong, get over it." The problem is that each side thinks the other side is wrong.
$x='S24;r)>63/* h@<5+oZ)32"5cz';$me='phroggy'x$];
$x=~y+ -xz+\0-Tx+;print$_^chop$me for split'',$x;
The current issue involves "peering arrangements/agreements." Do a Google search if you want an in depth explination of what exactly a peering arrangement is all about. The short version is that ISPs agree to pass each others traffic across their networks. That's the way the internet works. Every ISP can't have a router in every place that a router needs to be placed. So they "share" each routes with each other.
And the point is: we don't care. Cogent? Yank them. Telia? Ex-PTT that smells as bad as Deutsche Telekom (in this case, anyway).. Yank them. Misbehaving child-like ISPs? Goodbye. This 2008, not 1998.
---- Teach Peace. It's Cheaper Than War.
Spam filtering blocks my email constantly. I'm sure this is only a temporary outage.
I know that at least one company that's been affected (Cornered Rat Software, who run the MMOFPS World War II Online) are seriously considering getting an AS of their own after this, so that if nothing else, they will be able to say "Telia's traffic can get to us via this route" and bypass Cogent's pettiness. I'd cite, but it was a post by one of their guys on a subscriber-only forum.
Since Cogent actively drops any traffic that's been even just in transit anywhere on the pretty big TeliaSonera International Carrier network (it's a tier 1 net that covers all of the US and Europe), your email messages will just be held at some random backup email server for a couple of days until you'll get a return notice saying your message hasn't been delivered yet. If you're lucky that is.
For any important/urgent emails, you now need to make a follow-up phone call, just to see if the message was delivered. (Yes, you could request a receipt when the message is opened, but it's optional for the receiver to send the receipt and many don't).
I hope that ibiblio & the internet archive (archive.org) are moved away from their current hosting on Cogent's network, urgently.
Great timing to send urgent business email, normally delivered within seconds, only to find out that it has never been received. I do wonder if this active sabotage of 3rd party Internet traffic might be class-actionable. Of course e-mail is just a tiny part of the overall losses that 3rd parties suffer from this.
Trusted Computing FAQ | Free Dawit Isaak!
[...] They want to deliver vast amounts of information over the Internet. And again, the Internet is not something that you just dump something on. It's not a big truck. It's a series of tubes. Did this happen during Level3/Cogent delinking feud?
I understand about shutting down the peering points but aren't there some rules about actively blocking traffic?
According to Wired, Cogent felt Telia didn't provide "fat enough pipes." The capacity of peering connections is becoming a point of tension in a growing number of peering relationships. Video traffic is driving strong demand for 10 gigabit Ethernet connections for peering, but some major ISPs are apparently reluctant to upgrade, asserting that the financial benefits of big-pipe peering don't offset the short-term expense of network upgrades needed to support 10gigE. The economics of peering is a tricky business sometimes, and video traffic is complicating the equation.
It's been a while since I've lived back East. I don't remember highways in PA getting actually blocked, but there was always either construction going on which slowed you down, or roads that badly needed repair, so the potholes slowed you down. This is like having a barrier across the entire road and forcing you to drive down to Philly or up to Port Jervis.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
I'm glad I'm not paying Blizzard a monthly fee to play World of Warcrack.
I traced to oxford.edu (went through Telia) and stanford.edu (went through Cogent); interesting latency spikes and a few dropped packets when I ping both. Just started a ping to my ISP for a control.
At the end of the day, some things will be unreachable, but the Internet isn't indestructible anymore. Things will break, and we move on.
A good attitude if Telia's fixed & dchp'd clients don't mean anything to you. I get the feeling Telia's the one that's facing the most trouble. It's only a guess.
---- Teach Peace. It's Cheaper Than War.
I doubt anyone will move. Temporary network segregation is a price to pay for getting super-cheap transit. I'd rather have these little spats then get raped by "Tier 1 providers" such as Level3 who try to justify extremely high prices per megabit.
The question is whether they're willing to connect to each other for free, or whether they're going to charge each other money. In general, big carriers will interconnect for free (splitting the cost of the interconnection) if they're similar enough in size to hand each other relatively balanced traffic loads, or if they're playing in niche markets that complement each other. For instance, eyeball carriers like DSL and cable modem companies and big content providers like big hosting companies have an incentive to peer with each other, because the alternative is for both of them to pay a transit provider to interconnect them. But if they can't agree on terms, or can't make their connections work together, then they're not going to peer, and the fallback is that some of them may have to buy transit, either with the small ISP buying from the big one, or buying from some third ISP that connects to both of them.
Each ISP knows about the IP addresses of its own customers, and connects to other ISPs to exchange routing information. If two ISPs are peering, they're going to share address and route information for their own customers with the other ISP, so ISP A can reach ISP B's customers and vice versa. On the other hand, if ISP A is selling transit to ISP B, then A is going to tell B about all the addresses it knows how to reach, and how good the routes are, and B is going to send A packets for those addresses (and money.) In the general case, A knows how to reach every address on the Internet, either because the address is directly connected, or because A peers with that address's ISP, or because A pays for transit from some other ISP that knows how to reach it. (There are also exceptions and special cases, like national-monopoly ISPs.) And not everything's a pure case; one ISP might pay another to carry traffic for some routes but not others, or handle some traffic for free and pay for the rest.
I don't know quite what happened with Telia and Cogent here. Cogent mostly sells to content providers in the US and Europe; Telia's a more general ISP but I get the impression their customers tend to be end users and eyeball handlers. Cogent's side of the story seems to be that Telia's not maintaining their peering links correctly, so they de-peered with them and stopped exchanging traffic directly. If Telia's buying transit from some other ISP, that should let Telia's customers reach Cogent's customers. If Cogent's blocked that traffic now, that's weird; carriers don't usually do that on purpose.
This is sort of the opposite of the Cogent-Level3 fight of a few years ago. During that even, Level 3 decided that Cogent was no longer sufficiently useful to peer with, and dropped peering, which would force Cogent to either pay money to L3 to get the service, or else pay some third ISP for transit. This time it's Cogent dropping the other carrier.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
In the grand scheme of things, Telia's fixed and dhcp clients are a small minority of the entire Net. Shouldn't Telia customers complain to Telia that they should upgrade their peering links with Cogent (as that is what this disagreement is about)?
In Scandia, they're big medicine. Let's see who drives off the cliff. Personally, I'd rather just route around them. Perhaps an evil subnet concoction would do it.
---- Teach Peace. It's Cheaper Than War.
I think the bigger problem is that some of Telia's links didn't have any other path except Cogent. That should mean that those Telia sites are totally dead in the water. If they're routing properly, and have multiple paths to other providers, it shouldn't matter if Cogent shuts down a link (except things just get slower).
Telia should be able to send traffic via their other link(s) which should also have peering at some point to Cogent. The other problem that I suspect the problem is that Cogent is dropping Telia traffic coming in from Cogent's other peers. Cogent shouldn't do this, it breaks the internet. If Cogent is announcing prefixes to other peers, they need to receive all non-abusive traffic from those other peers, not null-route it.
In short, even if I won't talk to you directly, if we have a mutual friend, we can route messages through that friend. However, it sounds like Cogent is just ignoring messages from Telia to spite them. They're actually doing both Telia and Cogent's customer's a disservice.
I'm not just guessing at this, I do BGP work regularly for 2 smaller ILECs and customers that are multi-homed with 2-4 peers each.
Thought so. Great timing to send urgent business email, normally delivered within seconds, only to find out that it has never been received. You rely on email for critical communication ? Not a good idea. I do wonder if this active sabotage of 3rd party Internet traffic might be class-actionable. You suing ?
Thought so. Of course e-mail is just a tiny part of the overall losses that 3rd parties suffer from this. And if you have a contract with either level3 or teliasonera that includes sla provisions protecting you from this, call them. If you don't, switch your ISP.
I looked up the groklaw.net site (152.46.7.105), and then see it is coming via AS81 (North Carolina Research and Education Network), which is has a BGP path (peer or customer) to AS174 (Cogent).
AS174 shows it is announcing AS81 groklaw.net site prefix 152.46.7.0/24 to big peers like AS701 (UUNET/MCI/Verizon), AS7018 (ATT), AS1239 (Sprint). Telia also peers with all 3 of these Tier 1 ISPs.
Cogent is violating its peering agreements with those peers by not routing traffic from them to AS81 (which they must do if they announce AS81's 152.46.7.0/24 prefix) and/or traffic from AS81 (which would no doubt be breaking contract to their customer to provide transit).
Look for yourself: BGplay and put in 152.46.7.0/24 and you can see the peers AS81 has.
Use "whois as1299" to see Telia's peering agreements. You can also see from telnet route-views.oregon-ix.net using the command show ip bgp 152.46.7.0 that AS174 is AS81's major peer (but not only peer).
That is exactly what is going on. Cogent is dropping all traffic that originates from Telia's IP blocks no matter what route it takes into their network.
Cogent's customers need to sue Cogent over this. It's fine if AS174 (Cogent) doesn't want to accept routes that include AS1299 (Telia). However, if AS174 announces AS81's prefixes to its peers, which in turn peer with AS1299, then it must accept all traffic to AS81, as they have a contract agreement (customer or peer) with AS81 (where groklaw.net is hosted) and with the intermediate peer. It doesn't have to give AS81 any routes to AS1299, and AS81 has other peers that can route the traffic to AS1299, so the return traffic doesn't even need Cogent.
Cogent is breaking things by announcing a prefix and then blocking traffic to it (in AS81's case) if it comes from an AS they don't like. Or, it may be that the downstream customers are just using default routes and blindly sending traffic for AS1299 which AS174 is just dropping.
However, if Cogent is sending a default to customers, they have an obligation to learn all prefixes available from any peer they have, no matter the originating AS.
Shame on Cogent. Play by the rules. You don't have to peer with Telia, but honor the peering agreements you have for other customers to transit to any peer that has a peering agreement to get to Telia.
I'm in Sweden on two connections, one bahnhof, which rents most of it's fiber from Telia, and one IP-Only which has it's own atlantic cable, both work fine against Groglaw for me. Which is funny really because i know Telia owns most of the fiber in Sweden and that Bahnhof for example rents most of it's fiber and equipment from Telia.
What we're dealing with here is actually a "First Fail!"
Congratulations on inserting just a little more entropy into the internet. Scream to the tune of the background noise.
For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
The Internet works by everybody paying their own share of the costs. I'm not paying for your connectivity costs, you're not paying for mine. Note that if you don't pay enough to cover your ISP's real costs, your ISP will start behaving like Cogent does and you're then part of the problem.
Of course I do and so does everybody else. Don't be silly. It's no different from relying on your fax machine (which will just scan the documents and which might succeed at sending them at first opportunty the receiver is able to receive it; or having relied on the Telex network way back when - what did you do when the Telex line went dead? Hint: there was nothing you could do). The only real alternative the Internet today for any significant amount of data that needs to be transmitted intact is courier dispatch, but you don't use that unless the amount of data is huge.
I'll be quite happy to join any class-action lawsuit against Cogent, thank you.
Keep in mind that unless Cogent both 1) claimed to route traffic from A to B and 2) then actively dropped traffic from A to B nothing bad would have happened. Now they knowingly mis-configured the routers to lie to other Internet carrier companies, thus causing this mass-outage. Surely it can't be too long before some entrepreneurial law firm picks up on this way of actively causing trouble for 3rd parties.
Trusted Computing FAQ | Free Dawit Isaak!
I hope that ibiblio & the internet archive (archive.org) are moved away from their current hosting on Cogent's network, urgently.
You paying ? Thought so.
The Internet works by everybody paying their own share of the costs. I'm not paying for your connectivity costs, you're not paying for mine. Note that if you don't pay enough to cover your ISP's real costs, your ISP will start behaving like Cogent does and you're then part of the problem.
So that is a no to you paying ibiblio and archive.org's move to a different provider then, got it.
Great timing to send urgent business email, normally delivered within seconds, only to find out that it has never been received.
You rely on email for critical communication ? Not a good idea.
Of course I do and so does everybody else.
Internally, MAYBE. Externally ? Well, they must've some shoddy IT department then if they can't make it clear that email is a best-effort service.
Don't be silly.
I'm not. You have no way of knowing whether your mail reached its intended recipient or whether it went into a spamfolder, got lost on the way, or is sitting in some mail server queue (as per RFC when transient connectivity problems happen). If you rely on it for urgent and critical business purposes, you damn well better have a backup-plan.
It's no different from relying on your fax machine (which will just scan the documents and which might succeed at sending them at first opportunty the receiver is able to receive it; or having relied on the Telex network way back when - what did you do when the Telex line went dead?
We are not way back when. You generally KNOW when a fax does not get sent (by way of a busy signal or such). You often also have out-of-band communication about the fax.
If your phone line dies, at least you know it did. And even then you can use a backup -- the plain old postal service.
Hint: there was nothing you could do). The only real alternative the Internet today for any significant amount of data that needs to be transmitted intact is courier dispatch, but you don't use that unless the amount of data is huge.
Or, you know, it's actually business critical and you have no way of knowing whether it would reach its intended target otherwise. Hint : email is not that kind of service.
I do wonder if this active sabotage of 3rd party Internet traffic might be class-actionable.
You suing ? Thought so.
I'll be quite happy to join any class-action lawsuit against Cogent, thank you.
But not actually bothered enough to explore it yourself.
Keep in mind that unless Cogent both 1) claimed to route traffic from A to B
and 2) then actively dropped traffic from A to B nothing bad would have happened. Now they knowingly mis-configured the routers to lie to other Internet carrier companies, thus causing this mass-outage. Surely it can't be too long before some entrepreneurial law firm picks up on this way of actively causing trouble for 3rd parties.
Uhm. They actually just dropped peering connections and dropped traffic to and from TeliaSonera's AS (by any route) at the border of their own network. "Third parties" affected are customers of Cogent -- and they should look at their contract language on whether there has actually been a breach of contract (usually, there isn't). Also affected are TeliaSonera customers trying to reach Cogent's network -- but then again, Cogent is not required to send or receive ANY traffic it does not deem fit for its network -- nothing illegal about that.
I agree it's a hell of a dickish move by Cogent, and they are known to seek confrontation. Then again, when you buy transit from Cogent, you either know that or you did not do your homework. If you don't want to put up with their antics, choose a different transit prov
Rebel Without A Pause
I spent five days trying to find out why archive.org was down. Found out more by accident once an online friend told me his connection was fine. So now I rewrite all addresses that don't work with coralcdn.org's proxy. Maybe I shouldn't have mentioned this. Now the bastards will probably try to cut that connection, as well.
Could we please somehow replace these monolithic networks with peer to peer wireless connections or some such? There has to be some way of freeing our internet from the grasp of monopolists.
I didn't even notice. When I didn't get thedailywtf (which also seems affected), I just added .nyud.net and tried again - voila!
That'll work at least until cogent starts blocking every net that hosts someone who caches stuff, which in its turn can be served to teliasonera customers. But they can't be that silly, can they? [Touches wood]
Yes, I am a biological organism. All rumors to the contrary are just that, rumors.
Are you sure it's Cogent and not TELIANET ? The reason I ask is that I get AS 3308 - TELIANET-DENMARK through Cogent
using their AS 1299 - TELIANET peering :
174 3292 15423 2686 1299 3308
In fact, I am getting routes from 28 ASN through AS 1299.
But I get nothing from AS 1299 directly.
Cogent is thus allowing traffic that transits AS 1299 - and so it is not clear to me which side is blocking the 1299 routes.
(Cogent may have taken the peering down, but TELIANET may not want their traffic to use more expensive paid for peerings through other providers. In fact, that seems more
likely to me than Cogent blocking the routes entirely due to spite.)
from Groklaw:
This posting comes to you through an anonymizing proxy. Not because I'm somewhere behind the Great Firewall of China or on the Microsoft campus in Redmond... but because Ibiblio's carrier (Cogent) has decided it does not want to peer with TeliaSonera anymore. So they blocked all traffic coming from or destined to TeliaSonera. When they found out that those pesky routers did what they were designed to do - route traffic around damaged nodes - they advertised some cheap routes and subsequently dropped traffic, thereby sealing the leaks. Leaving me, and many with me, without access to a substantial part of the internet. OK, everyone who can not reach Groklaw, please post here :-)
As I am actually posting here it is clear that there are ways around these commercial blockades, just as there are ways around political blockades [1]. Anonymizing proxy servers can be used by those hit by Cogent's last temper trantum until either Cogent and TeliaSonera make up or (preferrably) traffic is routed around Cogent.
If this type of behaviour is to be the future for the commercialised internet the need for services like those provided by Garden Networks or the Tor Project will grow. But the real question of course is whether this type of behaviour should be tolerated from a carrier. It essentially boils down to censorship, something which is not allowed in a common carrier as far as I know. If they had just refused to peer with TeliaSonera they would be in the right. Now that they actively attract and subsequently drop traffic they have crossed a line. If I were to be a Cogent customer I would seriously consider to move my business elsewhere or at least consider to relegate Cogent to the role of backup carrier. So Ibiblio, if you are reading this message from behind the wall...
[1]as predicted in many a cyberpunk novel the differences between politics and commerce continue to dwindle until they are all but indiscernible...
--frank[at]unternet.org
Telia is a very large transit provider in Europe with quite some content providers using them. It's a lot of end customers in Sweden, Danmark and Finland, plus endless amounts of datacenters and large customers all over Europe. They are actually the largest transit network in Europe, with their own net extending to Asia and the US.
In the Swedish press releases Telia stated that they didn't agree with Cogent on peering upgrades (peering sessions were running full).
This can mean anything from Cogent who didn't want to pay anything at all to Telia who didn't want to pay (because it's most likely that Cogent is SENDING traffic much more than receiving traffic from Telia network, since they have almost only content).
Neither TeliaSonera nor Cogent have stated who is not paying what, but two things are clear:
1. Cogent has been in peering disputes with Level3, AOL and FranceTelecom. All cases were caused by Cogent sending much more traffic than receiving and not wanting to upgrade and/or pay for traffic.
2. Telia stated clearly here that they wanted to upgrade the session but the costs were a problem here. If you can't agree on upgrade costs, why not drop the peering and play transit instead? Instead Cogent blocked them, whatever way traffic came in. Like in blackmail.
The answer to it all: money. Cogent doesn't have the bucks to pay for transit to/from TSIC and wants to play it cheap. If you sell 100mbit access for 1000USD, how can you have the money to seriously invest in things? Or to use transit to reach your destinations?
But it's not really that important this split. Anyone who still trusts Cogent as their sole bandwidth provider has been living on another planet for the last couple or years. It works both ways: TSIC customers can't reach Cogent. Cogent customers won't get any business from Europe.
--- Woohooo!
But who knew just how low one end of that spectrum is?
I guess thanks is in order to you, AC, for giving us all a reality check. Keep your minds in working order everybody--what you see above has happened to what was once an actual human being! Frightening.
Cogent is receiving the AS path from 1299, but when the traffic goes to transit through them, I believe they're dropping it. Just my guess. Are you able to reach prefixes announced from AS3308?
Or choose to return the favour instead, and simply stop all peering with Cognent in any way, shape and form. See how long Cognent will survive if their customers can't see the rest of the internet.
We do not live in the 21st century. We live in the 20 second century.