Transatlantic Cable Fault Disrupts Internet In UK
An anonymous reader submits "Web traffic between the U.S. and Europe has been hit after an undersea cable developed a major fault on Tuesday. Because the TAT-14 cable network is shaped like a ring, it should be able to cope with one such failure -- but unfortunately the consortium that owns it hadn't fixed an earlier problem, just off the U.S. coast. Just shows how systems with build-in redundancy can still go badly wrong...."
LINX, the London Internet Exchange, which carries nearly all UK Internet traffic and over half of Europe's Internet traffic
I guess the Echelon boys got to go home early that day.
Trolling is a art,
We have a link from the US to the UK.
It is redundant, unless we have 2 faults.
We have a single fault...but we don't repair it.
So then we have anouther one!
I would really like to ask if these guys ever thought of putting together a startup....because let me tell you, they already have the right frame of mind.
Neck_of_the_Woods
#/usr/local/surf/glassy/overhead
Imagine... some big cable that's thousands of miles long connecting continents...
That's just a weird idea. You gotta wonder who makes those things and how, exactly, they're maintained. Let alone set up in the first place. Do they just sit along the ocean floor? Are they suspended in mid-water? I have absolutely no idea. Just mind-boggling to me, the logistics of it.
i'm amazed that i survived - an airbag saved my life.
This really is a great loss for the Slashdot community.
the article, but here's the link to the linx (badum tsh) website, with another news site for the article.
When anger rises, think of the consequences.
Confucius (551 BC - 479 BC)
Honestly, this reminds me of the first transatlantic cable. They kept getting faults and they couldn't figure out what it was. Turns out the paying-out machine had the cable rubbing against some fine metal shavings which would occasionally get stuck in the casing and ground the cable to the sea-water.
I wonder what happened to this one?
- Sherman
this was all over the service provider lists yesterday...
l es .html
.. i heard days ... not hours :-(
The latest from the rumor mill....
FYI, for some history on the TAT's
http://davidw.home.cern.ch/davidw/public/SubCab
still seeing decent ping times. anyone detect an actual outage or issue? Best info we have is that there are two outages. One has existed
for the last 3 weeks or so between Tuckerton (New Jersey) and Bude (UK). It takes out the "southern path" across the atlantic.
There is a second outage between Bude (UK) and Katwijk (NL). For circuits that landed in London or France this (should have) taken out the redundant path for those circuits.
Circuits from Tuckerton (New Jersey) or Manasquan (New Jersey) to Katwijk (NL), Norden >(DE), or some city in Denmark who's name I
forget should still be up on the northern path.
> So, if you're in London or France your circuits are likely to be down, however some people in those locations used Contentinal capacity to link up to Katwijk, in which case they might still be operational.
I confirm that France is having some problem with TAT14.
France Telecom International Backbone (Opentransit) is currently running with
non TAT14 capacity (10G) and one oc48 direct to Copenhagen (that is ok).
We (Opentransit) are currently not experiencing any congestion but are implementing a new 10G circuit to secure our topology until TAT14 is back to life (one leg at least).
Both problems are undersea issues, so don't expect speedy resolution if you are down.
Yep
-Opentransit (France Telecom)
Just shows how systems with build-in redundancy can still go badly wrong...."
Um, the built in redundancy worked as it should, apart from the maintainers not fixing the first fault. Their maintenance is what went wrong. Nobody will ever be able to afford or build a system like this with so much redundancy that you aren't required to maintain it.
A computer without Microsoft is like ice cream without ketchup.
It was definitly noticable as our customer reported
1) Website traffic down at least 30%
2) Around 75% packet loss from the EU -> US
3) Slow delivery of email
Basically it caused a massive amount of headaches and you have to wondered WTF didn't they fix the first problem when it came up. Its like running a RAID Array on one disk.
Well least things seem to sort of be getting back to normal
Rus
Cheap UK and US VPS
"Just shows how systems with build-in redundancy can still go badly wrong...."
No, it shows how well designed redundancy can be overcome by bad management decisions! Engineering brought low by bean counters... Gee, when has that ever happened before?!
Any technology distinguishable from magic is not sufficiently advanced.
Any technology distinguishable from magic is insufficiently advanced. - Geek's corollary to Clarke's law
Man, the FBI is going to have to interview *every* *single* *fish* in the area for Al-Queda connections.
No one will even suspect the dolphins because they are supposed to be, like, higher mammals or something.
--- Ban humanity.
It probably was a startup, y'know, back in the ol' under-water-cable boom of 1999. But then the bubble burst, and all the people that dropped out of college to lay cables on the ocean floor had to find real jobs.
Expert in software patents or patent law? Contribute to the ESP wiki!
That's strange, I'm in the UK and SlashDot is hosted in America, so according to this story, I should be having problems -- but in fact, everything is working just fiFgfdgf3gf4h32hh%$$$424452
--
What short sigs we have -
One hundred and twenty chars!
Too short for haiku.
Have you considered the irony of posting such a comment in a web based discussion forum, considering that the creator of the web is British?
Why not do what my friend Mike does when he has a problem with a bad cable and just jiggle it a little? It works great for his monitor cable so why not for a giant bundle of fiberoptics/wires/whatever in the ocean? What could possably go wrong? Jiggling the cable has got to be cheaper than going down to BestBuy and buying a new cable and running it from the US to the UK. Don't get the extended warrenty though, it's not worth it!
Losing faith in humanity one person at a time.
One failure, occured on Oct 30, 2003, has existed for the last 3 weeks or so between Tuckerton (New Jersey) and Bude (UK). It takes out the "southern path" across the atlantic.
/.s since yesterday actually
http://davidw.home.cern.ch/davidw/public/SubCables .html
The new failure is between Bude (UK) and Katwijk (NL). For circuits that landed in London or France this (should have) taken out the redundant path for those circuits.
more info at
www.tat-14.com
http://www.kddiscs.co.jp/e/business/02_15.html
About the TAT-14 Cable Network
This transatlantic cable system is in full service, connecting the United States to the United Kingdom, France, The Netherlands, Germany, and Denmark.
The cable system is a dual, bi-directional ring configuration using DWDM multiplexing with 16 wavelengths of STM-64 per fiber pair. The system also utilizes reverse direction protection switching in the event of failure of the service fiber.
It has a dual route, transatlantic capacity of 640 Gbits on 2 service fiber pairs backed up by 2 protection fiber pairs. This configuration provides a capability of transporting 4,096 STM-1's or approximately 9,700,000 circuits across the ocean.
"The price good men pay for indifference to public affairs is to be ruled by evil men." ~Plato (427-347 BC)
Comment removed based on user account deletion
The USNS Zeus (ARC-7) is the Navy's cable laying and repair ship. The cable is laid mostly on the surface of the bottom, but at vulnerable points and at both ends (near shore) is its ploughed in to the mud/sand on the bottom. When a cut or fault occurs, the location of the fault is determined with a TDR or O-TDR, the same way it works with a land based cable. They know the cable length to the fault and have a survey map of where the cable was layed. It is physicaly located with side-scanning sonar and robotic submersibles, then hooked and brought on deck for repair (each end in case of a break). Once repairs are complete, the cable is unceremoniously shoved over the side, or re-ploughed depending on the location and mission of the cable.
Any technology distinguishable from magic is not sufficiently advanced.
Any technology distinguishable from magic is insufficiently advanced. - Geek's corollary to Clarke's law
Here's a map that I found which shows the "ring" of TAT-14...
TAT-14 Cable Route
There's more than one cable system linking US with Europe, it just happens that several carriers (Above.Net being one) only have capacity through TAT-14.
Other carriers have working circuits on TAT-14 and another link (e.g. Apollo, Tyco, AC-1, Gemini) and may have some degraded service (depending on whether their transatlantic links are less than twice the size of their peak demand). FranceTelecom OpenTransit is an example of one of them.
Interestingly, not many EU ISPs use TAT-14 North route, since it has a propagation delay of around 110ms (which is 40ms or so more than TAT-14 South from the UK and more than most other transatlantic cables)
Most ISPs in Europe that I can see are fine. Certainly the big international transit ISPs (Sprint, L3, C&W, MCI et al) aren't showing any more trouble than normal.
At the risk of being accused of Karma whoring, This page and This wired article from the late 90s are are good summary and a great story about undersea cables, respectively, despite being a little out of date.
Whose fault were these faults? Were these faults the fault of the oceanic faults? I've heard some people say that these faults were the fault of faulty maintenance, but it seems to me that you can't fault them for the faults. Now perhaps that's a faulty assertion, but I really believe that the fault of the faults lies squarely upon the techtonic faults and not the fault of this supposedly faulty maintainers. I really doubt that the faults are their fault.
...this post was all my fault. :-(
I'm sorry.
I can hear it now:
"Hey boss, half of the cable just failed. We need to get on this right away".
The cable's still working, right?
"Yeah, but if something else goes wrong, we're screwed".
Look, that cable hasn't failed in ten years; let's put off repairing it until January. That way it won't affect our 2003 budget.
But these things generally happen in pairs, and with no back up - well, we're taking an awful risk. If something else fails, most of Europe - well, I don't have to tell you the consequences. Plus, remember..the weather in January -
cuts him off-- Not gonna happen! Put it on the schedule for mid January!
Hopefully that manager is no longer employed....but don't be surprised if he winds up at Clear Channel! He sounds like just their kind of guy!You're new here, aren't you? Management is cheap, not lazy. Redundancy means that when something breaks they save money by not fixing it, not that they can keep running while they do fix it.
They delays in repair may also be due to the bids they have out to fix it: A Greek sponge diver, the "Polynesian" pearl diver from an unnamed Florida amusement park and a crew from Bangalore with no diving experience or equipment, but a willingness to follow the diving script. There's also a chance that an unnamed "muff diver" may be employed as well, but executives are downplaying it as part of their don't ask, don't tell policy.
Management originally wanted the crew from "Ghost Ship" because the chick was hot, but when they found out it was only a movie they had to look elsewhere.
As far as I'm aware the problem wasn't just limited to the UK but to the whole of Europe. One of our transit connection to the US using this Fiber, was disrupted. Following message we received from our transit provider:
We are currently experiencing a catastrophic failure on the fiber ring that is
affectively isolating Europe. We are researching the possibility of
alternative connectivity, and will update you as we get more information.
One more problem which was caused by this link outage is that our dns-servers (and those of multiple providers) where hit with a lot of dns lookups for lockdown.zonelabs.com (seems zonelabs firewall, queries that name). As the dns-server for that zone wasn't reachable anymore (no more traffic to the abovenet network in the US) the dns-servers had to do a query for each new lookup which caused a huge load. And effectively killing the customer dns servers, impacting traffic even more.
I've found the whole notion of undersea cables fascinating ever since I read Neal Stephenson's Mother Earth Motherboard
Reminds me of how we went through all this trouble at [UNNAMED CORPORATION] of making sure to mirror the root disks for all 3000+ servers, but nobody setup alerts or notifications of a disk failure. So even though all the disks were mirrored if one drive failed, nobody knew. So we ended up running most our boxes off one drive until the other drive went out. So sure, mirroring delayed a major problem, but the major problem still existed.
We also had a similar problem with Fiber Storage. For all the servers they had run two seporate fiber runs to each box that needed to use the "SAN". Each server would have two fiber cards installed. This way if one network went out, it would just fall back to the other card. Well, of course, both cables were plugged into the same switch.. Smart. Yes, we did have a fiber switch go out once.
Reminds me of the Brit newspaper headline - Storm Stops Ferries, Continent Cut Off!
I stole this
The thing is that he tended to run the projects himself, including getting funding. The strain was immense, solving technical issues, managing the projects as well as the finance.