Slashdot Mirror
A Hole In the Net, Down Under
cjm_in_oz writes: "Since 4pm yesterday, Australia's leading ISP has lost 60% of its bandwidth due to either an earthquake, or as is more likely, a ship's anchor. Read more here ." Most of the entire continent's bandwidth, you see, courses through a particular manhole ... sheesh. This sure sounds like an argument for more and more fiber, along different courses.
from the article:
"...circumstantial evidence suggests this species is a member of the DSL community..."
Mmmmm, high-bandwidth sharks...
I use Macs for work, Linux for education, and Windows for cardplaying.
This is just a cover story for the AU government's new censorship system. As they get the bugs worked out, the speed will come back, and they will claim to have "fixed" the cable....
www.eFax.com are spammers
The captain of an American container ship off the coast of Indonesia is reported to have said "Whoops, my bad" and totally tried to play the whole thing off.
EOF
The vulnerability and high cost of cables is one reason the infosphere wants to be in orbit. Hardening satellites against nuclear electromagnetic pulse attacks has, however, been inadequately addressed outside of military satellites. The bulk of the hardening can occur with a relatively light-weight faraday cage enclosure so it shouldn't add too much to the mass budget of the satellite.
Seastead this.
Hell, it's a freaking 13/14h flight from Sydney to LA or ~12 from Aukland to LA (which is odd as it's ~3h from Sydney to Aukland and they're on a similar latitude).
Bill - aka taniwha
--
Bill - aka taniwha
--
Leave others their otherness. -- Aratak
Bill - aka taniwha
--
Bill - aka taniwha
--
Leave others their otherness. -- Aratak
A lot of people are saying that yes, Telstra has ONLY lost 60% of their international connectivity.
Let's put it into perspective.
Telstra currently have 980Mbps of international connectivity. They have lost their largest link, 622Mbps worth, so they need to route around the problem.
It's not that simple.
The moment the link died, it wrecked havoc with BGP routes everywhere. I couldn't get outside my ISPs own network (optus) to Telstra for around one and a half hours. Yes, true, other ISPs here also have int'l bandwidth, but it hardly compares to the near gigabit that Telstra have.
Any southern-cross delivered bandwidth has yet to be utilized since customers (i.e. the ISPs) are either waiting for their connection or are still playing around with the configuration (read: playing Quake III with sub-150msec latency before the cable is loaded). Optus/CW, MCI and NZ Telecom are *shareholders* basically. They still have to purchase their own bandwidth (and line their pockets with money after a while). Telstra has got Southern Cross cable capacity too but just like the smaller ISPs who might only have a few megabits, they don't *own* the cable.
Thankfully, I'm on the optus@home cable modem service so I don't need to touch Telstra int'l bandwidth but connectivity to local sites is still erratic.
Now, stop slashdotting our link -
11 FastEthernet0-0-0.pad18.Sydney.telstra.net (139.130.249.239) * 376/383/401 (6.78) ms 9/10 pkts (10% loss)
12 * * * * * 0/5 pkts (100% loss)
13 * * * * * 0/5 pkts (100% loss)
-tsg
And a couple of additional bits of information:
Bill - aka taniwha
--
Bill - aka taniwha
--
Leave others their otherness. -- Aratak
...but 60% of total bandwidth available to the ISP called Telstra. Their backup line was flooded and 1 in 3 requests where lost. "Other ISPs and networks such as Optus were uncongested." It is very intresting that this same line connects so many distant contries (Japan, Indonesia, Hong Kong, Germany, UK and the USA). I was under the impression that connections are made between contries, and then those contries in turn connect to others. I find it very interesting that so many contries are all on one pipe! ARPANET was originally supposed to be 'multi-node-failure-tolerent' so that many whole-cities could be destroyed (atomic strike or whatnot) with the system continiously delivering packets successfully (unless the destination is the one destroyed!) by re-routing around failures. Its amazing that such a fine example of fault-tolerent design can become so weak that a backhoe, anchor, or small localized earthquake can successfully disable so much capacity!
Disclaimer: sure, I lived in Australia for 7 years (in Queensland :/, bloody banana benders (my mom's a cockroach (dad and I are Canadian));), but that was 6 years ago (in NZ during that time) and now I'm in Canada, so my grasp of .au stats is way rusty.
Bill - aka taniwha
--
Bill - aka taniwha
--
Leave others their otherness. -- Aratak
I mean, I've experienced the MAE West packet loss and such, but at least my packets could move on to other ISPs.
Of course, the article was a little sketchey on details. Does anyone know?
Bleh!
SO that when one gets cut, the other takes over. The STM protocol happily can accomadate this.
Then again, building redundant transcontinental lines are not a trivial thing....
Mode (3) smart-aleck mode. Press * to return to main menu.
Ekkkk.. that's not good.
That's like there being a car wreck someplace in the U.S. and UUNet going down because of it. This reminds me though, of that article I read once, where the sharkes were eating the fiber optic cable, cause they liked the taste of the gel inside it.
(sorry don't know where a link to the story is)
Fuck Telstra and avoid them like the plague. Your local calls are cheaper elsewhere, your STD calls are cheaper elsewhere, your mobile calls are cheaper elsewhere, your International calls are cheaper elsewhere, and your Internet access is not only cheaper elsewherethen their overpriced Big Pond, but is now faster, more stable and more reliable though a company like Optus with the new link.
Xenex
- Who has been totally Telstra free for over 15 months, and had loved it.
When an ISP is operating under limited bandwidth, the LAST thing you want to do is POST A STORY ON SLASHDOT POINTING TO A WEB SERVER ON THE OTHER SIDE OF THE PIPE, restricting said bandwidth even further. *sigh*
Some drunk, old and likely insane freighter captain decides to drop anchor 100 miles out to sea and accidentally kicks half the world in the nuts. I guess even the little people still have some power... :D
----------
Even as you read this, your pants are strangling your loins! Aaa!
> You make Australia sound like some kind of backwater that's up there with the places they have more guns than food.
You mean, like the USofA?
Sheesh, evil *and* a jerk. -- Jade
Informative?! Dear God, did the idiot moderator even click the link?
--
Obfuscated e-mail addresses won't stop sadistic 12-year-old ACs.
Win dain a lotica, en vai tu ri silota
I work for a medium size telco doing backbone work.
I can just imagine what a pain in the ass this will be to fix!
For those who are unfamiliar with SONET fiber technology:
You have to understand that for most companies, (until just recently), SONET is set up in a dual ring configuration. This is somewhat analogous to FDDI in a Lan. There are generally 4 fibers between offices, 2 per cable, (transmit and recieve). If you get backhoe fade on one cable, the signal gets transferred to the other cable's transmit and recieve. To set up a star type topography has traditionally been cost prohibitive, and is not done too often.
Although the internet is highly redundant on the IP level, most of continental and international traffic is jammed onto a few huge capacity strands of fiber to save money.
One cable cut in a long haul situation can quite possibly be disastorous in many areas of the world. I imagine Australia, being an island in the middle of nowhere, (internet wise), can't have too many redundant links.
Just a thought
That reminds me of a situation here in MN a few years back...
:)
Internet service was cut off for several days after a homeless guy was sleeping under a bridge...I beleive his blankets were covering some exposed electrical power equipment and a fire started. All the fiber lines nearby were melted (this is from memory, the story is probably off a bit). ALL the lines into MN at that time were through this area (maybe they still are?) so there was no net for a few days. If someone from MN remembers the exact story, post a reply
I think the point is that a fair amount of the net is fragile in this manner. Remember, you only have to cut the line once over any distance to prevent it from working. Obvious yeah, but people seem to forget.
Has anoybody checked the possibility of a radioactive dinosaur tripping on the wire on its way to Darwin?
__
__
Men with no respect for life must never be allowed to control the ultimate instruments of death.
GW Bu
Check out the latest stats on Oz's bandwidth via the Internet Traffic Report at this address
I watched C-beams glitter in the dark near the Tannhauser gate.
Ocean conditions and prevailing currents prevent undersea cable from being layed in most areas of the ocean.
You see, while daily tides don't seem too powerful to the average person, the stress imposed on a cable that is exposed to miles upon miles of oceanic currents will break even the strongest of cables. A similar situation was previously discussed here
With respect to these conditions, Floreat Beach in Perth is the ideal area to lay cable. This coastline is sheltered from the impact of high energy swell by a series of submerged calcaranite barriers and offshore islands. As a result, low energy conditions prevail, especially in summer when incident wave heights are generally less than 0.5 m, giving the beaches a lake-like appearance.
In addition, Perth is home to many major Australian dotcoms and ISP's. It is considered by many to be Australia's version of Silicon Valley in the US.
-atrowe: Card-carrying Mensa member. I have no toleranse for stupidity.
To expand on what was mentioned earlier about redundacy a bit...
Redundancy in the telecom world takes on two basic forms - equipment redundancy and circuit redundancy.
Equipment redundancy is pretty much what it says - redundant equipment in place with the same end service in mind. Should a processor in a fiber multiplexer die, fry, go up in smoke, whatever, the equipment fails over to the backup processor with little or no interruption of service. This is more or less a minimum standard in the telecom world.
Then there is circuit redundancy, which takes two more basic forms, either over a single cable (with multiple fiber strands), or over multiple cables. Unfortunately, more often than not in more "rural" areas, this is usually multiple fiber redundancy within a single cable. Which covers almost all equipment failures involving the physical cable interface, but does nothing at all for when Joe-Backhoe-Operator digs without a locate and tears up 15 feet of cable without even noticing it. Which is the most common failure I've experienced.
Even multiple cables isn't always the answer, especially if the physical routes aren't diverse enough. I can think of two instances, both in the upper midwest, where multiple cables in proximity (read - one or more conduits in a very small space) were damaged or destroyed, once by fire, and another by a vandal who knew where to find them.
The ideal situation is for telephone companies to have two routes out to the "network" running in almost opposite directions. However, again, although this works well for switched voice calls, most of the time it doesn't work for "nailed up" point to point circuits, which still leaves those types of circuits, commonly used by ISP's where frame relay isn't available, in the cold when routes are destroyed. And, we all know how fragile frame relay can be, especially when Worldcom is at the helm - "Oh, let's just globally upgrade the software in our network without any phasing or large scale testing at all. Oh, yeah, and when it falls on it's nose, let's let it fester for a week or so before we get the network to re-converge."
So, there you have it in a nutshell. Nothing is foolproof, especially if you don't own or control all of the network from point to point. But, I think most everyone would agree, there are very few times I've ever picked up the phone and not had dialtone. I think most people in the states would agree with that. Remember, the telephone network is the most complicated, expensive, diverse, available electronic network in the world. And although the phone companies are far from perfect, especially the former US West and Worldcom, when you think about it, it's really amazing how stable it is, and that it works as well as it does!
Brad
I bet the cable-cut incident as occuring sometime around 19:14 PM Pacific time on Sunday, Nov 19.
Why do I say that? Because I run a web service that criss-crosses between California and Australia (and Tasmania, for what it's worth). We have an n-Tier web application that is hosted, in tiers, both in AU and in CA. The customer's sites are hosted in AU but link to web app pages in CA. Then the web server connects via secured named pipe to a legacy application server back in AU. (Yes, the transaction pings and pongs from the US to AU multiple times before completion). This is usually extremely transparent to the user (you can bet were not running Flash animation over these links!). But beginning Sunday at 19:14 PM my time, the usual became excruciatingly unusual. That's when I began logging timeout errors.
I don't know whether to be shocked at what effect one cut cable has caused, or to marvel that even with a severe blow to a continental backbone the Internet is routing around the damage, as designed.
One thing for sure -- I'm glad I'm not running a leased-line application that requires me or my company to troubleshoot the network! How nice to completely off-load responibility for intercontinental networking to completely unrelated third parties. There's antother point to be made about the distribution of labor in this story...and the economies of scale... but I digress...
-- @rjamestaylor on Ello
Australia=australian=aussie=ozzie=oz
I've never seen a guard at Floreat Beach guarding a man hole. But that said I've never seen anyone guarding any of Perth's telco infrastructure apart from telco facilities or exchanges.
Actually, about four years ago, I worked for SWBell. One of our projects was to lay several T3 lines bewteen California and Hawaii.
"The transatlantic route is the most populated with cables and has also seen the greatest drop in the price of capacity. Last year, $250,000 would buy a T-1line between London and the USA. Five years ago, that is what it cost to lease a line for a year. Prices have not fallen as far in the Pacific, but will continue to drop worldwide"
Found here
-atrowe: Card-carrying Mensa member. I have no toleranse for stupidity.
I wonder what the bandwidth of that pipe is....
"It is a greater offense to steal men's labor, than their clothes"
How about c|net's story off of the ap: Qwest ordered to pay AT&T $350M for repeatedly cutting a fiber-optic phone line.
[
building one link is expensive enough, but 2 when not needed is just not likely to happen, unless they need the bandwidth to sell it. because .au is probably a hard market to get into (upfront costs ++), its rather monopolistic.
therefore, as the only company around, they don't really need to be uber reliable, only decent. from their point of view, redundancy is probably only a cost which they can slash... capitalism strikes again
SSL Certificate
the available bandwidth will decrease even more as sites in Australia are Slashdotted by everyone checking this out.
The damaged cable does not usually handle "Most of the entire continent's bandwidth...." It handle 60% of a particular company's bandwidth, which is very different.
And this event does illustrate the need for more connections. The story also explains that there are more connections, including a new cable with 5x the bandwidth of the one damaged. Unfortunately, that new cable is owned by a different company which is not experience technical difficulties.
My mom is not a Karma whore!
Backhoes can be a problem, but we shouldn't always place the blame on the operator. While it is true that some heavy equipment operators are either drunk or on-something-else, many are excellent operators who take time and care with their work.
My GF's brother-in-law owns and operates a backhoe, and I rode "shotgun" with him on it many times (one of these days I hope he can show me how to operate the thing). I can tell you that when digging, even knowing there is a pipe, cable, conduit, something - it it hard to tell the difference between it and everything else that is "down there". Unless you are operating in "virgin" soil, many times there will be plant roots, old runs, rebar, branches - you name it, and many times it looks just like what you are trying to avoid. A good operator will either have a spotter, or be looking carefully, and stop immediately after seeing something that even looks like a cable or pipe, get off the rig, and inspect the area. More often than not, it is nothing - but every once in a while it is something, then you have to back off, and hand dig around it.
The biggest problem is when you are digging, and you don't spot the hazard - a boom on a backhoe can be moved quickly, or delicately - but even at it's most delicate, there is still a bit of inertia (hey, it's a big mass of steel, for cryin' out loud!) - which can still cause a break, if you don't catch it in time.
I know there are dangerous operators out there - but the majority are good workers, doing a job you or I might not ever think about doing...
I support the EFF - do you?
Reason is the Path to God - Anon
This made the newspapers (at least in a minor way) here, and believe me, there's a lot of unhappy people. I'd imagine that Telstra might well be leasing some space on that big, redundant connection in the not-too-distant future :)
Any sufficiently advanced technology is indistinguishable from a rigged demo
--Andy Finkel (J. Klass?)
Cable gets cut and power blows up. Obviously using the same computer technologies as Hollywood uses.
One seaworthy vessel; ten million dollars
A solid steel anchor; two thousand dollars
Whoring for karma with bad Mastercard ripoffs; priceless
There are some things having a life can't get you. 87 Karma is one of them.
In Oz, as a Telstra customer last night, data rates to the US were down to a few hundred bytes per second (assuming you could successfully connect at all), and ping times were up to ~5000 msec. Thankfully, I didn't have any big up/downloads that needed doing. Things have improved today - ping times are back to normal and bandwidth is up to ~5-6 kilobytes per second (I'm on a cable modem).
How is it for others in the region? All we've heard so far is Australian reports, but this outage will unfortunately be affecting people throughout Asia :(
Any sufficiently advanced technology is indistinguishable from a rigged demo
--Andy Finkel (J. Klass?)
- Most of the entire continent's bandwidth, you see, courses through a particular manhole
... sheesh
Read the article, and you'll see:- Other ISPs and networks such as Optus were uncongested.
Yes, Timothy, we do have more than one ISP out here. And I believe Telstra carries a minority of traffic (given they're over 100% times more expensive than other bandwidth providers). Anyone that has someone like Optus or uuNet (my ISP has redundant links to both) as their upstream would not have been any more affected by this than your typical American (some Aussie sites may have been down).Can't you realize what's happening?! Look at the moon tonight! LOOK AT IT! Can't you see that the light reflected off its dark side is an indicator that an asteroid has hit Australia, and is emitting a fireball brighter than the sun? THIS COULD MEAN THE END oh wait that's a streetlamp outside, never mind.
I thought I was having a flashback to when pr0n arraived at 2400 baud...
--
Only a week ago a competing company fired up a new link.. Now suddenly their only competition has their wire cut. Hmm. Sounds like a good way to acquire customers for that new expensive link, -- while permanently destroying your competition.
Sounds like what they were talking about in Cryptonomicon, cable cutting wars. Easy to start, but nobody dares start them.
If this fiber links Australia to two other continents, then why isn't it fully looped? I mean, you run one cable from australia to continent a, then another cable to continent b. Hopefully, you are smart enough to run a cable from continent a to continent b, and you have a loop. Say you have OC192 running in this loop, then you have sub 50ms switching capacity and you can switch all traffic from the broken loop to a working loop.
People cut stuff, they drive dump trucks through aerial fiber (hitting power lines as well), and other really stupid stuff. If you have an entire continent running on one link, then that link better be redundant and fully looped. These people have one to blame but themselves (people being the inet company).
Joseph W. Breu
Routing through the mideast is a little dicey given the political instability. The infrastructure costs make a fat pipe via siberia a real pain. The point is simply redundancy, as well as opening up the net to other areas of the world. a fat pipe going through that part of the world would help this out tremendously.
Now it seems that this point has been brought home in spades. (shudder)
"It is a greater offense to steal men's labor, than their clothes"
Well this has sort of flattened out now, and Australia is doing a lot better, but it's still interesting to note whats probably happened here.
1. Earthquake/plate shift (and it seems like an earthquake/plate shift) has disrupted the cable between Jakarta and Singapore. Because of the way the cable is arranged, this will mean Jakarta and Australia are effectively getting zero bandwidth from Singapore. Earthquake/plate shift is highly suspected due to the recent (not long after the cable break) earthquake in Papua New Guinea. Telstra didn't allow for any real redundancy (by running dual cables seperated by a few hundred feet or more) in their cable run. One cable handles all the traffic, so a single break causes complete data loss. (Southern Cross uses a two cable system to allow redundancy in the case of a cable break).
2. The night of the cable break, a power station in Paddington, a suburb just east of Sydney (NSW) burnt down, dropping 20,000+ homes and businesses into a blackout. Paddington is where Telstra routes most of it's eastern states traffic, and indeed all traffic destined to go through the previously existing pipes from Sydney goes through Paddington. A simple hiccup or loss of power caused by failing generators could have caused havoc. Such a scenario seems very likely given the problems that appeared on Telstra's network that night. Any sub-system that kicked in may have not been able to handle multiple failures dealing with traffic problems (eg: fibre to Singapore being down, which goes via Perth, Western Australia), causing more headaches.
3. Due to floods, much (approximately one-third) of the NSW outback is under water, making any problems with cables running across this area difficult. Cables that might be under 6 feet of dirt are under an additional 12 feet of water making any repair hard. Microwave links running across this area could also suffer, due to power outages, or possibly tower foundations shifting and throwing communications out of whack.
Anyway, things seem a lot better for Aussie internal traffic, and external traffic isn't TOO bad (apparently). Fortunately I'm with Optus Cable (who have their own link to the US) at home, and UEComm (One of the many companies who use Southern Cross) at work. *grin*
http://smw3.fcr.fr/SMW/SMWB2.htm
Didn't anyone teach these guys in elementary school that water is blue and land is !blue? I spent ten minutes studing this map trying to parse out recognizable shapes before I realized that blue was land. I thought it was an extreme close up of Indonesia, and I kept wondering, "Why is Perth on the East coast of this landmass??"
Silly me.
-Chris
...More Powerful than Otto Preminger...