Transatlantic Cable Fault Disrupts Internet In UK

An anonymous reader submits "Web traffic between the U.S. and Europe has been hit after an undersea cable developed a major fault on Tuesday. Because the TAT-14 cable network is shaped like a ring, it should be able to cope with one such failure -- but unfortunately the consortium that owns it hadn't fixed an earlier problem, just off the U.S. coast. Just shows how systems with build-in redundancy can still go badly wrong...."

Comments

Tin foil hat, please.

[grub]

LINX, the London Internet Exchange, which carries nearly all UK Internet traffic and over half of Europe's Internet traffic

I guess the Echelon boys got to go home early that day.

Re:Tin foil hat, please.

[bsharitt]

Does this mean that there will be packets washing up on the coast over the next few days?

Re:Tin foil hat, please.

[Doctor7]

Adding in a DNS server didn't do it for me - I tried using the Demon one and couldn't get anywhere (sticking with the default, I could still reach The Register for some reason, but nowhere else). The problem seemed to be that the ntlworld.com addresses were not resolving - mail, news, and web proxy - so maybe OpenNIC had them listed and Demon didn't

Re:Tin foil hat, please.

[Pieroxy]

From the story: Just shows how systems with build-in redundancy can still go badly wrong

Well, build-in redundancy is just there to let you some time to fix problems before disrupting activity. I mean, if I don't change HDD A on my RAID-1 Array when it is reported to be defective, there is no point in having a RAID-1 Array. The company in charge is responsible. The "build-in redundancy" did its job fine. They just shouldn't have installed a system with redundancy if they didn't plan on fixing non-disruptive problems.

Ok let me get this straight....

[Neck_of_the_Woods]

We have a link from the US to the UK.

It is redundant, unless we have 2 faults.

We have a single fault...but we don't repair it.

So then we have anouther one!

I would really like to ask if these guys ever thought of putting together a startup....because let me tell you, they already have the right frame of mind.

Re:Ok let me get this straight....

[bugbread]

My guess is that the initial problem may have been an undersea cable. Those generally take 2 or more weeks to fix, and if the weather is really bad, they have to pull the boats back in, delaying things further.

No evidence, of course, but it seems like the most logical reason. Cables like the TAT-14 don't stay unfixed just because someone in management is lazy.

Re:Ok let me get this straight....

[gorilla]

Remember that fixing faults in undersea cables isn't exactly an activity that you can do in 30 seconds. You've got to get a ship out to the location of the fault, hook the cable, and get it to the surface, and then fix it. There are going to be a limited number of cable ships which have the capability, and they might be busy elsewhere. Even once they start acting on the repair, they are going to take time to get to where the fault is (14 knots cruise speed isn't exactly the fastest ship around the QE2 cruises at 28 knots, and still takes a week to cross the atlantic. Remember a cable ship might be off in the other side of the pacific when it becomes free), and then time to get the cable and repair it. Therefore 'earlier this month' not being repaired is perfectly reasonable.

Re:Ok let me get this straight....

[dnahelix]

They don't go underwater to fix these cables. A boat follows the cable, pulling it up at one end of the boat, and dropping it down at the other. From what I understand, if a big storm or the like happens, they have to drop it and start over.

Re:Ok let me get this straight....

[Anonymous Coward]

The real problem is in the design of networks. Information networks are designed to be fault-tolerant (famously but erroneously attributed to a desire to withstand nuclear attacks) -- multiple connections and a "mesh" network mean that if nodes break, traffic is routed elsewhere and the network continues to function. This works great, and there's no problem with it. But the problem is, humans don't build networks this way, and economics is against doing so.

If you're buying a network connection, you buy it from the best provider available, which naturally means network connections become concentrated to a few suppliers, who in turn find economies of scale and provide lower prices, thus attracting more customers. Thus the economics of building networks naturally produces networks that have a few or even single points of failure: we noticed this on September 11th, when the knockout of the huge links through New York noticeably slowed transatlantic traffic, even to sites other than CNN and the other news sites that were being toasted by demand at that point. Centralisation is something that we naturally do because it's economically efficient, but centralisation leads to problems for networks.

In the energy sector, things are even less flexible, because energy connections are a lot more expensive to set up and difficult to maintain than information links. The US powercut was caused by the cascading failure of a daisy-chain of power stations around the great lakes. Nobody would build an information network that way any more, but it's still the natural way to build a power network. Italy's powercut was caused by a huge reliance on foreign power, supplied by JUST TWO LINKS to France -- one fell over, instantly overloading the second and knocking it out too.

Yes, we are critically reliant on these fragile networks. And yes, economic realities tend to cause these problems, but not because of privatization: it's simply because humans naturally tend to build poor networks, because those are cheaper -- no matter who pays the bills. To solve the problem, we need to pay more attention to networking theory when building all of our networks, and provide regulatory incentives to build better networks of both kinds.

Or one day, a critical failure will cause a cascading catastrophe, and it will be nobody's fault. We built the network to fail that way.

Re:Ok let me get this straight....

[billimad]

Cables like the TAT-14 don't stay unfixed just because someone in management is lazy.

no, but they do if it's up to the French to fix it.

Re:Ok let me get this straight....

[kiwimate]

Why the {expletive deleted} is this modded so high?

Well, let's see. Perhaps because the parent poster (i) did read the article; (ii) also read the thread here; and (ii) apparently has somewhat better English comprehension skills than someone else I could name.

You see, it's quite simple. He's talking about the initial problem. You're talking about the subsequent problem. The initial problem, from the article...

Unfortunately, a part of the cable near the US coast had already suffered a technical fault earlier this month, which meant there was no built-in redundancy to cope with Tuesday's failure. According to BT, the US-side fault should be fixed by the end of this week, which will bring the cable network online again.

Now, as you'll see, there is no mention as to what the actual problem was, just a guide as to when it'll be fixed which gives us an indication that it is taking some time. This, in turn, was the subject of discussion of the original post.

I hope this elucidation has clarified the issue for you. Oh yes, one more thing...you're a silly git.

Re:Ok let me get this straight....

[gorilla]

Finding where the fault occured is actually pretty easy. Each repeater can be remotely interregated for diagnostics, and they have a TDR built into them to detect exactly where the fault is. So you know it's say 1500m west of repeater #17. Go to that location, and pull it up.

Re:Ok let me get this straight....

[grozzie2]

no, but they do if it's up to the French to fix it.

Actually, if you read the article, you would know. The first fault occurred weeks ago, on the american side. The second fault on Tuesday, on the European side. If the first fault had been fixed when it happened, then this new one would not have brought the system down, so, it appears the facts disagree with your statement.

The statement should correctly read:
no, but they do if it's up to the americans to fix it.

Re:Ok let me get this straight....

[Roofus]

In case anyone was wondering, TDR=Time Domain Reflectometry.

http://www.tscm.com/tdr.html

Basically you just send a pulse using the cable which has a fault. At the point of the fault, the signal reverses its path. By timing how long it takes for a pulse to return, and by knowing the speed of the pulse in the cable, you can figure out how far along the cable the fault is.

Of course, it can suck if your cable doesn't travel in a straight line...

That's totally fuct

[siphoncolder]

Imagine... some big cable that's thousands of miles long connecting continents...

That's just a weird idea. You gotta wonder who makes those things and how, exactly, they're maintained. Let alone set up in the first place. Do they just sit along the ocean floor? Are they suspended in mid-water? I have absolutely no idea. Just mind-boggling to me, the logistics of it.

Re:That's totally fuct

[confused one]

It's literally one big, long, armored cable drapped across the ocean floor between the U.S coast and England.

hard to imagine, ain't it.

Re:That's totally fuct

[bugbread]

The sit on the ocean floor. In low water areas, they are cladded in steel, to prevent anchors, etc. from ripping them up. Recent ones are apparently treated to prevent sharks from chewing on them, which was an old problem. Fixing them involves sending out a big ship that hauls up cable from the ocean floor (they have a lot of slack so that this is possible), hanging the cable across the deck, fixing it, and lowering it back into the water.

Yes, I'm a WAN administrator, why do you ask?

Re:That's totally fuct

[gr8_phk]

How?

1) put a big spool of cable on a ship.
2) anchor the cable to shore.
3) set sail for the other side of the ocean.

I had no idea it could be so simple and obvious when I heard it either.

Re:That's totally fuct

[velo_mike]

You gotta wonder who makes those things and how, exactly, they're maintained.

Check out Global Marine Systems the company that laid it, and some of their cool toys (er, if you're into big assed boats).

Re:That's totally fuct

[pla]

That's really an impressive technology. Kinda creepy, too. All that information streaming through the dark sea bed...

Now you know why the high-pressure methane breathing aliens (the ones who live under the sea for convenience of maintaining "atmospheric" pressure in their domes) know all about us, but we know almost nothing about them.

We've just given them a high-bandwidth line that we have almost no ability to monitor between the two endpoints.


(For the humor-impaired... Laugh).

Re:That's totally fuct

[tie_guy_matt]

Read "The Victorian Internet" by I forget who wrote it. It is about the telegraph and how all the things we thought were new in the 90's had really been around for 100 years or so (chat rooms, compression, data security) and it talks about the first time some nut decided to put a big cable under the ocean to connect continents. It is a really cool book!

Re:That's totally fuct

[Noizemonger]

If you want to know more about this subject, there is an great article by Cyberpunk Writer Neal Stephenson on Wired in wich he describes the laying of a BIG intercontinental Communication-Cable. As a self proclaimed "Hacker-Tourist" he visits a lot of the construction sites all around the world giving colourful descriptions and a lot of well researched background infos about this rather arcane but fascinating subject. A great read. Find it on: http://www.wired.com/wired/archive/4.12/ffglass.ht ml

Re:That's totally fuct

[mikerich]

how in the world do you get that far down to reach the section of cable that is miles below the surface? holy crap, that must be one hell of a ship. it must hold a lot of cable to hoist that bidness up and then be able to hold the weight of it all as they work on it. do they just get it up high enough to allow divers to get to it? i agree with the parent of your post, this is absolutely amazing to me.

Yes you need a big hold and run the cable over the stern. These ships tend to have grapples to latch on to cables and haul it aboard for maintenance.

The first successful TransAtlantic cable was laid by Great Eastern designed by Slashdot's patron saint - Isambard Kingdom Brunel.

She was the biggest ship in the World for almost fifty years, and SIX times larger than any ship afloat (that was Brunel's Great Britain which was itself TWICE the size of any other ship).

Bearing in mind she was launched in 1857 here are the statistics:

• 680ft long
• 120ft over the paddle wheels
• gross tonnage - 18,914 tons
• displacement over 27,000 tons

She had a single screw, twin paddle wheels and six masts (her steam engines which were the biggest in the World (naturally) were still novel technology), a complete double bottom and double hull which was internally compartmentalised. She could have carried 4000 people from Britain to Australia and returned without refuelling.

What happened to her? Brunel could barely launch the ship, she had to be slid sideways into the Thames, rather than float her from a dry dock. It took three months to get her into the water. When she was afloat she had cost more than three times the original budget and the Eastern Steamship Company who commissioned her was on the edge of bankruptcy.

When she was fitted out, she was put to sea on trials. Brunel was aboard, but the effort of constructing Great Eastern had almost killed him. He suffered a massive stroke and was taken ashore. Shortly afterwards, off of the South Coast, Great Eastern suffered a massive explosion in one of the water jackets surrounding a funnel. Five men died and the ship had to be put in for repair. Brunel was told the news, and almost immediately lapsed into a coma, dying a few days later.

Finally she was put into service, not on the Australia run which had proved unprofitable, but on the North Atlantic. She never carried more than a tiny fraction of her passengers and was reknowned for rolling in heavy weather. One story does stand out, she hit a reef whilst travelling at full speed on the approach to New York. Her bottom was cut open along a greater length than that of Titanic, not only did she not sink, she continued her voyage without loss of life and arrived safely in New York where she was repaired.

Eventually the cost of running Great Eastern became too great and in 1864 she was sold to the Telegraph Construction Company for the purpose of laying the TransAtlantic cable. She was the only ship in the World capable of holding the entire cable - it took more than 5 months just to load the cable into her holds. The first attempt in 1865 was almost successful, but the cable broke in Mid Atlantic in more than 6000 feet of water.

So what did they do? They went back to Britain, picked up another cable and laid the first truly successful cable in 1866. Better than that, Great Eastern found the broken cable (no I have no idea how), spliced it and got that working as well.

Great Eastern's importance to the British Empire can't be underestimated. She laid the cables that joined Britain to the African colonies, the Eastern part of the Mediterranean, India and Australia. Without them, the British Empire could not have been governed.

And the ship? Well she was replaced by a custom built cable carrier Faraday in 1874 and laid up in Milford Haven, South Wales. She hung around there for twelve years before being t

Say it ain't so!

[Sanity]

With the Brits off the Internet, who are we going to rely on to correct our grammar? Who is going to tell us that every plot of every US scifi show was done in the 70's by Dr Who? Who is going translate Alan Cox' Welsh weblog for the rest of us? Who will fight the other side of the "Who invented the first computer?" debate?

This really is a great loss for the Slashdot community.

Re:Say it ain't so!

[Anonymous Coward]

Reports of us Brits being off the net have been greatly exaggerated

Re:Say it ain't so!

[arevos]

I'm from Britain, and I'm still here :)

As regards to who invented the first computer, I'd imagine it was more a debate between the German's Z3 and Britain's Colossus. The Z3 was Turing-complete, but only if you hacked it a bit, and it wasn't originally designed to do that, nor was it likely to have ever run that way (if I recall, it involved literally forking the punch tape, and taking advantage of a bug in the mechanical reader to similate an "if" function). Colossus was Turing complete computer, and was designed that way.

Z3 was finished on May 1941.
Colossus was finished in January 1944.
ENIAC was finished on February 1946.

The US created the second or third computer. Depending on how you figure it, it was Germany or Britain that created the first.

Personally, I'd vote for Colossus, as the Z3 was never intended to be Turing-complete, and probably never used that way. But then, I'm biased :)

Re:Say it ain't so!

[adrianbaugh]

Er, if the Brits are off the internet then you're not going to be able to read AC even in the original Welsh. As far as I know (living 12 miles from the border) Wales is still a part of Britain (despite the best-laid plans of the English to cut it loose and float it off into the Irish Sea ;-P)

Re:Say it ain't so!

[Chicane-UK]

It'll take more than internet disruptions to keep us Brits from correcting your grammar mistakes, and translating Alan Cox's blog.

Though I don't speak a word of welsh, and my grammar probably isn't that hot.. so you'll just have to make do with the company :D

Re:Kinda scary..

[grub]

Sorry, I've read that 5 times now and am unable to parse it. Also babelfish doesn't have Drooler->English translation. Please rephrase.

I was suprised this wasnt in

[Pingular]

the article, but here's the link to the linx (badum tsh) website, with another news site for the article.

First Cable

[General Sherman]

Honestly, this reminds me of the first transatlantic cable. They kept getting faults and they couldn't figure out what it was. Turns out the paying-out machine had the cable rubbing against some fine metal shavings which would occasionally get stuck in the casing and ground the cable to the sea-water.

I wonder what happened to this one?

Re:First Cable

[rcw-work]

This is borderline unrelated, but any long-haul fiber cable (that would need repeaters) will have power running through it - usually at a very high voltage.

Comment removed

[account_deleted]

Comment removed based on user account deletion

old news

[Anonymous Coward]

this was all over the service provider lists yesterday...

The latest from the rumor mill....

FYI, for some history on the TAT's

http://davidw.home.cern.ch/davidw/public/SubCabl es .html

still seeing decent ping times. anyone detect an actual outage or issue? Best info we have is that there are two outages. One has existed
for the last 3 weeks or so between Tuckerton (New Jersey) and Bude (UK). It takes out the "southern path" across the atlantic.

There is a second outage between Bude (UK) and Katwijk (NL). For circuits that landed in London or France this (should have) taken out the redundant path for those circuits.

Circuits from Tuckerton (New Jersey) or Manasquan (New Jersey) to Katwijk (NL), Norden >(DE), or some city in Denmark who's name I
forget should still be up on the northern path.

> So, if you're in London or France your circuits are likely to be down, however some people in those locations used Contentinal capacity to link up to Katwijk, in which case they might still be operational.

I confirm that France is having some problem with TAT14.

France Telecom International Backbone (Opentransit) is currently running with
non TAT14 capacity (10G) and one oc48 direct to Copenhagen (that is ok).

We (Opentransit) are currently not experiencing any congestion but are implementing a new 10G circuit to secure our topology until TAT14 is back to life (one leg at least).

Both problems are undersea issues, so don't expect speedy resolution if you are down.

Yep .. i heard days ... not hours :-(

-Opentransit (France Telecom)

The redundancy didn't go wrong..

[Keck]

Just shows how systems with build-in redundancy can still go badly wrong...."

Um, the built in redundancy worked as it should, apart from the maintainers not fixing the first fault. Their maintenance is what went wrong. Nobody will ever be able to afford or build a system like this with so much redundancy that you aren't required to maintain it.

Re:The redundancy didn't go wrong..

[Keck]

That's no different than single redundancy going wrong, but twice :) I think what I was trying to say was that people are expecting redundancy to be a sure-fire guarantee against bad things happening and taking you out, which is foolish at best. It's just a means of reducing your chances of a failure where the risk of downtime justifies the much larger cost.

Long story short, I thought the commenter was showing some level of ignorance in implying that "built-in redundancy" isn't supposed to "go wrong".

Noticable impact

[rf0]

It was definitly noticable as our customer reported

1) Website traffic down at least 30%
2) Around 75% packet loss from the EU -> US
3) Slow delivery of email

Basically it caused a massive amount of headaches and you have to wondered WTF didn't they fix the first problem when it came up. Its like running a RAID Array on one disk.

Well least things seem to sort of be getting back to normal

Rus

No...

[PSaltyDS]

"Just shows how systems with build-in redundancy can still go badly wrong...."

No, it shows how well designed redundancy can be overcome by bad management decisions! Engineering brought low by bean counters... Gee, when has that ever happened before?!

Any technology distinguishable from magic is not sufficiently advanced.

Well, that's consortiums for you.

[HarveyBirdman]

Can't they get some quantum wormhole tunneling dealy-widgets-whatzits going so we don't need cables anywhere anymore?

Man, the FBI is going to have to interview *every* *single* *fish* in the area for Al-Queda connections.

No one will even suspect the dolphins because they are supposed to be, like, higher mammals or something.

Re:Well, that's consortiums for you.

[tuffy]

Man, the FBI is going to have to interview *every* *single* *fish* in the area for Al-Queda connections.

I had no idea "Finding Nemo" was a crime drama...

one more thing...

[CaptainBaz]

You forgot the tea advocacy :-)

Statistics

[Etherwalk]

Anybody have hop count & RTT statistics?

probably was a startup

[ciaran_o_riordan]

It probably was a startup, y'know, back in the ol' under-water-cable boom of 1999. But then the bubble burst, and all the people that dropped out of college to lay cables on the ocean floor had to find real jobs.

Oh no the Internet almost broke?

[hellfire]

She added that the Internet was not broken, as traffic was rerouted through other networks.

I read this and I couldn't help but think of a CDW commercial:

Clueless pointy-haired boss to the camera: "Fred? I think I just crashed the Internet."

You'd Think

[The Dobber]

Because the TAT-14 cable network is shaped like a ring

That on a geeky, tech oriented site such as this, we could have a slighty better description.

That's Strange, I'm in the UK

[Mirk]

That's strange, I'm in the UK and SlashDot is hosted in America, so according to this story, I should be having problems -- but in fact, everything is working just fiFgfdgf3gf4h32hh%$$$424452

Re:Hmmmmm

[bugbread]

My network diagrams show:

• AC-1
• Gemini
• TAT-14CN
• TAT-12/13CN

Taking into account redundancy, that's 8 cables. There may be more, as my cable map is a few years old.

Re:Oh well

[arevos]

Have you considered the irony of posting such a comment in a web based discussion forum, considering that the creator of the web is British?

Re:Silly Limeys...

[Pingular]

Silly Limeys...
You relise you call us limeys because we used to eat limes to prevent getting scurvy, while your teeth fell out and you eventually died of Scurvy from lack of Vitamin C. Ironic that the very thing that saved many sailors lives is a (semi)insult you now use against us!

Just wiggle the cable!

[MrNybbles]

Why not do what my friend Mike does when he has a problem with a bad cable and just jiggle it a little? It works great for his monitor cable so why not for a giant bundle of fiberoptics/wires/whatever in the ocean? What could possably go wrong? Jiggling the cable has got to be cheaper than going down to BestBuy and buying a new cable and running it from the US to the UK. Don't get the extended warrenty though, it's not worth it!

Re:Just wiggle the cable!

[t_allardyce]

Just remember next time your on a plane - one of those 1000's of wires will have at one point stopped working and the first thing the engineer will have done is jiggle it, then when it worked they would have signed the form and moved on.

Re:Why only 2 cables?

[bugbread]

Because they're amazingly expensive. The TAT-14 cost 1.5 billion dollars to build.

actual information...

[Anonymous Coward]

One failure, occured on Oct 30, 2003, has existed for the last 3 weeks or so between Tuckerton (New Jersey) and Bude (UK). It takes out the "southern path" across the atlantic.

The new failure is between Bude (UK) and Katwijk (NL). For circuits that landed in London or France this (should have) taken out the redundant path for those circuits.

more info at

www.tat-14.com /.s since yesterday actually

http://www.kddiscs.co.jp/e/business/02_15.html
http://davidw.home.cern.ch/davidw/public/SubCables .html

For those who are too lasy to GooGle: TAT-14

[B5_geek]

About the TAT-14 Cable Network
This transatlantic cable system is in full service, connecting the United States to the United Kingdom, France, The Netherlands, Germany, and Denmark.

The cable system is a dual, bi-directional ring configuration using DWDM multiplexing with 16 wavelengths of STM-64 per fiber pair. The system also utilizes reverse direction protection switching in the event of failure of the service fiber.

It has a dual route, transatlantic capacity of 640 Gbits on 2 service fiber pairs backed up by 2 protection fiber pairs. This configuration provides a capability of transporting 4,096 STM-1's or approximately 9,700,000 circuits across the ocean.

Read The Article

[Ever Dubious]

It isn't like the US-side fault was just being ignored: "According to BT, the US-side fault should be fixed by the end of this week, which will bring the cable network online again." Given the logistics of repairing a fault, and without knowing when the US-side fault occurred, it is difficult at best to imply that the cable operators were somehow negligent in their actions.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Re:Kinda scary..

[bugbread]

Well, we haven't. As of a few years ago, we still had 3 other cables (6 other routes, as they are all ring topologies). We've just shut off some traffic, leased lines, etc. and forced more connections through the remaining lines, causing latency and connectivity for some people. The AC-1, TAT-12/13, and Gemini lines are up and running fine.

US Navy Cable Ship

[PSaltyDS]

The USNS Zeus (ARC-7) is the Navy's cable laying and repair ship. The cable is laid mostly on the surface of the bottom, but at vulnerable points and at both ends (near shore) is its ploughed in to the mud/sand on the bottom. When a cut or fault occurs, the location of the fault is determined with a TDR or O-TDR, the same way it works with a land based cable. They know the cable length to the fault and have a survey map of where the cable was layed. It is physicaly located with side-scanning sonar and robotic submersibles, then hooked and brought on deck for repair (each end in case of a break). Once repairs are complete, the cable is unceremoniously shoved over the side, or re-ploughed depending on the location and mission of the cable.

Any technology distinguishable from magic is not sufficiently advanced.

Re:US Navy Cable Ship

[travisd]

...actually when there's a non-break fault they frequently have to cut the cable to get it to the surface -- there's not enough slack when it's 2 miles below the surface. They cut and then splice new cable to give them enough slack.

Re:Time to

[gorilla]

Since this one is named TAT-14, it's not suprising that there are other TransAtlantic cables. There are currently active 8 different cables that AT&T use crossing the atlantic TAT-8 through TAT-14, and BUS-1. Cables TAT-1 through TAT-7 are retired.

Map of Trans-Atlantic Cable Network 14 (TAT-14CN)

[Anonymous Coward]

Here's a map that I found which shows the "ring" of TAT-14...

TAT-14 Cable Route

Re:Oh well

[Anonymous Coward]

Haha, check your facts...

Al Gore is not British.

Dammit!!!

[BurKaZoiD]

...web traffic between the U.S. and Europe has been hit...

I guess that means no underage swedish lolitas for me today...*sigh*

More than one cable system

[bvark]

There's more than one cable system linking US with Europe, it just happens that several carriers (Above.Net being one) only have capacity through TAT-14.

Other carriers have working circuits on TAT-14 and another link (e.g. Apollo, Tyco, AC-1, Gemini) and may have some degraded service (depending on whether their transatlantic links are less than twice the size of their peak demand). FranceTelecom OpenTransit is an example of one of them.

Interestingly, not many EU ISPs use TAT-14 North route, since it has a propagation delay of around 110ms (which is 40ms or so more than TAT-14 South from the UK and more than most other transatlantic cables)

Most ISPs in Europe that I can see are fine. Certainly the big international transit ISPs (Sprint, L3, C&W, MCI et al) aren't showing any more trouble than normal.

At the risk of being accused of Karma whoring, This page and This wired article from the late 90s are are good summary and a great story about undersea cables, respectively, despite being a little out of date.

Whose fault?

[iamdrscience]

Whose fault were these faults? Were these faults the fault of the oceanic faults? I've heard some people say that these faults were the fault of faulty maintenance, but it seems to me that you can't fault them for the faults. Now perhaps that's a faulty assertion, but I really believe that the fault of the faults lies squarely upon the techtonic faults and not the fault of this supposedly faulty maintainers. I really doubt that the faults are their fault.

I'm sorry.

...this post was all my fault. :-(

A fly on the wall would likely have heard....

[Newer Guy]

As TAT-14 is a dual, bi-directional ring of cable, a single serious fault should not be enough to break it, as traffic would still be able to flow between the countries on the ring. Unfortunately, a part of the cable near the US coast had already suffered a technical fault earlier this month, which meant there was no built-in redundancy to cope with Tuesday's failure.

I can hear it now:

"Hey boss, half of the cable just failed. We need to get on this right away".

The cable's still working, right?

"Yeah, but if something else goes wrong, we're screwed".

Look, that cable hasn't failed in ten years; let's put off repairing it until January. That way it won't affect our 2003 budget.

But these things generally happen in pairs, and with no back up - well, we're taking an awful risk. If something else fails, most of Europe - well, I don't have to tell you the consequences. Plus, remember..the weather in January -

cuts him off-- Not gonna happen! Put it on the schedule for mid January!

Hopefully that manager is no longer employed....but don't be surprised if he winds up at Clear Channel! He sounds like just their kind of guy!

Re:Hmmmmm

[gorilla]

Here are some more. Still not complete though.

Cheap management, not lazy management

[swb]

You're new here, aren't you? Management is cheap, not lazy. Redundancy means that when something breaks they save money by not fixing it, not that they can keep running while they do fix it.

They delays in repair may also be due to the bids they have out to fix it: A Greek sponge diver, the "Polynesian" pearl diver from an unnamed Florida amusement park and a crew from Bangalore with no diving experience or equipment, but a willingness to follow the diving script. There's also a chance that an unnamed "muff diver" may be employed as well, but executives are downplaying it as part of their don't ask, don't tell policy.

Management originally wanted the crew from "Ghost Ship" because the chick was hot, but when they found out it was only a movie they had to look elsewhere.

this affected the whole of Europe

[viktorVaugh]

As far as I'm aware the problem wasn't just limited to the UK but to the whole of Europe. One of our transit connection to the US using this Fiber, was disrupted. Following message we received from our transit provider:

We are currently experiencing a catastrophic failure on the fiber ring that is
affectively isolating Europe. We are researching the possibility of
alternative connectivity, and will update you as we get more information.

One more problem which was caused by this link outage is that our dns-servers (and those of multiple providers) where hit with a lot of dns lookups for lockdown.zonelabs.com (seems zonelabs firewall, queries that name). As the dns-server for that zone wasn't reachable anymore (no more traffic to the abovenet network in the US) the dns-servers had to do a query for each new lookup which caused a huge load. And effectively killing the customer dns servers, impacting traffic even more.

Optical Time Domain Reflectometry

[Fighting.Cephalopod]

It's like TDM, but for a fiberoptic line. Tells you within inches of where the break/fault is and can provide details such as what the nature of the problem is (air gap, short, loss of signal, etc).

Look up OTDR (some made by HP) for further info.

Mother Earth Motherboard

[elliotj]

I've found the whole notion of undersea cables fascinating ever since I read Neal Stephenson's Mother Earth Motherboard

Mirrored Drives

[ryanw]

Reminds me of how we went through all this trouble at [UNNAMED CORPORATION] of making sure to mirror the root disks for all 3000+ servers, but nobody setup alerts or notifications of a disk failure. So even though all the disks were mirrored if one drive failed, nobody knew. So we ended up running most our boxes off one drive until the other drive went out. So sure, mirroring delayed a major problem, but the major problem still existed.

We also had a similar problem with Fiber Storage. For all the servers they had run two seporate fiber runs to each box that needed to use the "SAN". Each server would have two fiber cards installed. This way if one network went out, it would just fall back to the other card. Well, of course, both cables were plugged into the same switch.. Smart. Yes, we did have a fiber switch go out once.

Re:The First Cable had more problems than that...

[drinkypoo]

Of course, there is a significant voltage drop across a line that long, so you have to increase the voltage to get a useful voltage out the other side. The other solution is to use AC power, which should be fine when your data is carried on fiber.

Re:No big loss.

[basingwerk]

Reminds me of the Brit newspaper headline - Storm Stops Ferries, Continent Cut Off!

Re:Oh well

[zulux]

Have you considered the irony of posting such a comment in a web based discussion forum,

Ahem.....some of us are using gopher.slashdot.org

And even a few old-timers use ftp.slashdot.org for their fix.

If you have a low account numbber under 1000 - you can still use slashdot's 1-800 dial-up number with your 300 baud modem. Besure to set your parity to 7 and not 8.

TTY service to Slashdot has been down for the last year though...

Re:Map of Trans-Atlantic Cable Network 14 (TAT-14C

[Xrc65kl]

Here are some aerial photos and maps of the US landing
sites for TAT-14 (and other cables) courtesy of Cryptome's Eyeball series.

Re:Map of Trans-Atlantic Cable Network 14 (TAT-14C

[jdh-22]

Interesting!

Play with numbers:

Sprints info on TAT-14

As stated, the TAT-14 is 16 pairs of STM-64 fiber. With a help from google, the average cost was $6000/km per cable.

6000 * 16 = $960,000/km For All 16 pairs.

The total length of the cable is around 15,000km long.

$960,000 * 15,000 = $1,440,000,000

The cost of a transatlantic link cost almost 1 and a half billion dollars that is capable of 640Gbits throughput!

Brunel's first ship

[Slashamatic]

Isambard Kingdom Brunel designed and supervised the construction of the Great Western Railway (including locomotives and designing the buildings). This is why the ship was called "The Great Eastern". The GWR was his first railway and the The Great Eastern, his first ship. His range as an engineer cannot be understated.

The thing is that he tended to run the projects himself, including getting funding. The strain was immense, solving technical issues, managing the projects as well as the finance.

Re:Brunel's first ship

[mikerich]

The Great Eastern, his first ship. His range as an engineer cannot be understated.

An amazing guy - he'd be so depressed if he came to Britain today, not only do we no longer have the fastest trains in the World, but our fastest trains are French and Italian! What he'd say about the West Coast Main Line rebuilding taking longer than the construction of the Great Western is anyone's guess. But he'd find the fact we'd built the fastest airliner then scrapped it for something slower incomprehensible.

One slight correction. Great Eastern was the third of Brunel's ships. The first, Great Western was the first ship to steam commercially across the Atlantic in April 1838 (she was beaten to being the first steamer fulls stop, by two days by Sirius although that ship was too small for commercial traffic).

His second was Great Britain launched in 1843 and was the first large iron ship and the first to be powered by a screw. She is now preserved in Bristol and is well worth a visit.

Great Eastern was the third and largest by far.

As you say, his range was phenomenal. To produce one revolutionary ship would have been achievement enough, but three? Had he just done ships that would have been a monumental achievement - but to build the World's fastest railway, the World's first large iron building (Paddington station), a series of monumental suspension bridges, flat-pack hospitals, stations, tunnels...

Incredible.

And his monument? If you travel across the Royal Albert Bridge between Devon and Cornwall - a bridge so revolutionary that its only recently been proven how it stands up - look up, its there in 6 foot letters.

I.K. BRUNEL
ENGINEER
1859

What a guy!

Best wishes,
Mike.

So, then you might say....

[switcha]

As TAT-14 is a dual, bi-directional ring of cable, a single serious fault should not be enough to break it,...

...customers may currently be experiencing problems with all Internet access, including Web browsing, email, ftp and newsgroups.

So....there was just one ring to screw them all?

I think we just

[JohnnyGTO]

/. ed the last of their cable capacity :-)