Slashdot Mirror


Rats 'Cripple' NZ Web Access

An anonymous reader writes "Drat! It was the rat! Telephone, mobile and Internet access in New Zealand was disrupted over four hours after rats were found gnawing through cables. More than 100,000 customers were affected and even the country's stock exchange came to a standstill. Powerless to take action against the rats, Telecom New Zealand is seeking compensation from the electricity company it says is responsible for knocking out another pipeline which eliminated backup services. Nothing like a backup plan."

61 of 305 comments (clear)

  1. Quadruple independent redundancy. by CyricZ · · Score: 3, Insightful

    Binary independent redundancy is often not enough these days. Like this incident shows, a single backup system is not enough. You need at least four systems to have a somewhat reliable system.

    --
    Cyric Zndovzny at your service.
    1. Re:Quadruple independent redundancy. by Jeff+DeMaagd · · Score: 4, Insightful

      In some ways, I agree, but every bit of redundancy costs a lot, esp. if it involves laying more underwater cabling to Australia and Asia. Two lines are fine most of the time because the likelyhood of the second one going down before the first can be repaired is extremely small such that it is practically a fluke, like this.

      It happens occasionally, and there are excesses to try to fight against it, quad redundancy is quite a lot and probably overpaying by twice for something that almost never happens. Is it worth paying twice as much to add another "nine" to the uptime? I had some line problems myself, while frustrating, I just did other things instead.

    2. Re:Quadruple independent redundancy. by Bane1998 · · Score: 5, Funny

      I keep telling this to my fiance. I need at least four prostitutes to back her up. I don't know what bothers her more, the idea of being 'backed up' or being 'redundant'.

    3. Re:Quadruple independent redundancy. by CyricZ · · Score: 2, Funny

      I keep telling this to my fiance. I need at least four prostitutes to back her up. I don't know what bothers her more, the idea of being 'backed up' or being 'redundant'.

      Why would you need four male prostitutes? You really don't have to resort to paying male whores to pleasure your woman. There are drugs out there, called penis drugs, that will make your erections actually erect. They might just cost less than these four male prostitutes you have hired (unless you got really cheap ones).

      --
      Cyric Zndovzny at your service.
    4. Re:Quadruple independent redundancy. by GileadGreene · · Score: 2, Insightful
      That depends entirely on the reliability of the individual systems that make up your redundant system. A binary-backup system can be as or more reliable than a quad system, if the individual components of the binary system are more reliable than the quad system.

      Having a quad backup adds a huge amount of cost, both in terms of buying the extra components, and in terms of the additional complexity involved in switching from one backup to the next. In fact, the more complex switching system can actually *decrease* the overall reliability of the system.

      More is not always better. Blindly adding redundancy without doing a real reliability analysis is a poor approach.

    5. Re:Quadruple independent redundancy. by flithm · · Score: 2, Informative

      Either that or a rat free government policty.

      Leave it to a bunch of red-necks to come up with a poster that says "Kill Rats at Sight!"

      btw try the "rat quizes." My favorite question and answer is:

      9. Why do we control rats?
      a) because they are ugly
      b) because they spread disease
      c) because they taste bad

      Heh.

      btw you can mod me off-topic. But if new-zealanders were a bunch of oil happy gun toting beer drinking rat haters they'd never have lost their precious precious pornography.

      Also of note is the fact that the Alberta government estimates saving over 1 billion dollars since the institution of the rat control policy. 1 Billion!

      I'm pretty much a non-violent pacifist and all that, but seriously... kill the rats!!

    6. Re:Quadruple independent redundancy. by sn00ker · · Score: 2, Informative
      Is it worth paying twice as much to add another "nine" to the uptime?
      When it keeps your stock exchange running? Sure.
      The stock exchange can place the blame on nobody but themselves. They have NO redundancy. A single connection provider, through a single firewall. They're not even peering at one of the peering points, unlike the National Library who were able to get a connection through Telstra Clear (with whom they had no previous relationship) up and running long before the fault was resolved. See this NZNOG post for details.
      --
      "God, root, what is difference?" - Pitr, userfriendly
  2. Secret Plan by Adrilla · · Score: 4, Funny

    The real story is that the RIAA paid these rats to take down the lines, and if they have to take out some phone services to cut down on piracy, then so be it. Those sneaky sunsabitches, I'm watchin you RIAA/MPAA.

    --

    "Plans are for fools! Oglethorpe, the plutonian (Aqua Teen Hunger Force)
    1. Re:Secret Plan by timeOday · · Score: 3, Insightful

      I'm just glad terrorists didn't do this in the US. Who knows what would have happened.

    2. Re:Secret Plan by the+real+darkskye · · Score: 3, Funny

      We'd all have had a lot less spam today!

      --
      Music is everybody's possession.
      It's only publishers who think that people own it.
      Fuck Beta
      ~John Lenno
  3. Nothing like a backup plan... by Anonymous Coward · · Score: 3, Funny

    And that's nothing like a backup plan...

    Plans? We don't need no stinking plans!

  4. Rats by Al+Mutasim · · Score: 5, Informative

    This must hit a special nerve with New Zealanders, who are trying to eradicate rats wherever possible. There are three types of wild rats in New Zealand, and none of them really belong there: black rats were introduced in the 1860's, brown rats were introduced on Captain Cook's ship in 1760's, and Kiore rats were introduced by Maori settlers in the 960's (plus or minus).

    1. Re:Rats by Anonymous Coward · · Score: 5, Funny

      dude you are some weird son of a bitch if you memorize the history of rats in New Zealand.

    2. Re:Rats by myowntrueself · · Score: 4, Interesting

      "and Kiore rats were introduced by Maori settlers"

      They are also fat and delicious.

      Mainly fruiti-vores, they are less interested in eating eggs etc than the black or brown rat.

      Very mild mannered creatures. Typically enormously fat and slow moving.

      They should be farmed not eradicated :)

      --
      In the free world the media isn't government run; the government is media run.
    3. Re:Rats by Eradicator2k3 · · Score: 5, Funny

      "There are three types of wild rats in New Zealand"

      Because the domesticated rats are much more considerate....plus they're easier to milk.

      --
      Mr. T pitied this fool on 27 July 1992.
    4. Re:Rats by pipingguy · · Score: 4, Funny


      Hey, I'm a piper - need any help?

      Paul
      Piping Design Central

    5. Re:Rats by mantidae · · Score: 4, Informative

      Actually, its not unusual for kiwis who know their history, to know their rat history too.

      You see, there have been a few waves of colonisation , including the waves of the first polynesian voyagers.

      Rat remains have helped historians date those waves of settlement.

      NZ history book occaisonaly hit the bestseller lists here in Aotearoa (Moari name for NZ), and any discussion of dating early polynesian settlement is incomplete without discussion about rats.

  5. Suing the power company by katana · · Score: 5, Funny

    Of course they're suing the power company. Lawyers won't sue the rats because of professional courtesy.

  6. What I want to know is... by Anonymous Coward · · Score: 3, Funny

    who ratted the out?

  7. It's Always The Stinking Rats... by __aaclcg7560 · · Score: 4, Funny

    Yeah, blame the rats and the electric company for everyting. While you're at it, find the two-legged rats who couldn't bother to put in a redundant backup plan.

    1. Re:It's Always The Stinking Rats... by mpe · · Score: 2, Insightful

      Are you kidding? Telco's idea of a "redundant circuit" is two wires in the same conduit.

      Or two conduits in the same trench. Even a conduit either side of the same (low) bridge.
      Too often what gets overlooked is that a system is only as redundent as its least redundent part.

  8. Liability by debilo · · Score: 2, Insightful

    Telecom New Zealand is seeking compensation from the electricity company it says is responsible for knocking out another pipeline which eliminated backup services.

    Surely the electricity company put clauses in the contract excluding them from liability for failures and damages caused by things outside of their control? I take it for granted that every big company offering services of any kind have clauses for those cases.

    1. Re:Liability by RedWizzard · · Score: 4, Informative
      Surely the electricity company put clauses in the contract excluding them from liability for failures and damages caused by things outside of their control?
      The power company cut a cable while digging a hole. How is that outside their control?
    2. Re:Liability by Bob+The+Lizard · · Score: 3, Informative

      According to the news monday night (this was 5 days ago) they put a post holer through a trunk cable...

      (yes it sucked no highspeed for 4 hrs in the middle of a work day...)

      G/

  9. Outage by Anonymous Coward · · Score: 2, Informative

    This happened on Monday. Today is Friday.

    It was a quiet day at work though :)

    1. Re:Outage by nomadic · · Score: 4, Funny

      This happened on Monday. Today is Friday.

      With the lines down there was no way to communicate the outage to the outside world.

  10. Monopoly by fgl · · Score: 4, Insightful

    A good example why a monopoly is vulnerable, I was @ work during the outage, & so didnt really notice so much, but I hear it was very disruptive.
    What are the odds of rats chewing through a main trunk cable on the same day a local power company cutting a second main trunk?

    --
    Go Away! Not for Sale
    1. Re:Monopoly by bmgoau · · Score: 2, Funny

      And all the while we Australians sat and watched as the new zealand civilisation crumbled in the darkness

      Phase one of our takeover is complete....

      Now it is time for the invasion!

      Go rats, destroy the new zealanders, and there will be limitless piles of cheese for all!

  11. They Were Lucky This Time by Anonymous Coward · · Score: 4, Funny

    Thank god the sheep didn't rise up against their internet access or they'd have been in real trouble.

    1. Re:They Were Lucky This Time by Anonymous Coward · · Score: 5, Funny

      Flamebait? C'mon! Sheep outnumber humans in New Zealand 11:1! If they ever rose up, they'd have a real problem on their hands.

      I've been to New Zealand. It's a beautiful country. The Southern Alps, the hot springs, the Maori culture. Seriously, I recommend everyone visits. Just keep an eye on the sheep, though, that's all I'm saying.

    2. Re:They Were Lucky This Time by Anonymous Coward · · Score: 2, Insightful

      Actually, the number of sheep in NZ has dropped, there a more sheep per person in Australia - so they have dingo's and sheep to play with now.

  12. Unpunished? by venicebeach · · Score: 4, Funny

    But the rodents will go unpunished

    What?? How can they let them go unpunished? They need to at least go out and punish a few in a highly public manner to send the rest a message, kind of like when there is a shark attack and they go hunting for "the" shark that did it.... We can't let the rats get away with this!!

  13. Seeking compensation? by BiggerIsBetter · · Score: 5, Insightful

    So Telecom is seeking compenstaion for fixing this. Fair enough you might say? The irony here is that Telecom has publicly stated they will not be paying *other* businesses for their losses due to the network going down.

    To put this in perspective, much of the countrys EFTPOS system went down. Much of the countrys mobile network went down. Much of the countrys DSL network went down. That means, NO electronic sales transactions, NO websites, NO email, NO mobile calls. LOTS of lost productivity and sales here.

    Even the friggin stock exchange went down - the 2nd time in a few weeks due to a Telecom fault! No wonder they don't want to compensate people - even Telecom doesn't have enough money for this.

    --
    Forget thrust, drag, lift and weight. Airplanes fly because of money.
    1. Re:Seeking compensation? by sn00ker · · Score: 2, Informative
      Just to explain this for people not familiar with NZ's financial system, we're heavily electronic. The vast majority of retail purchases, even if they're only a few dollars, are conducted by electronic transaction not with cash - Electronic Funds Transfer at Point Of Sale.
      All the EFTPOS transaction processing is done in Auckland, so everyone south of the cut - about half the population - was isolated from the engine room of retail sales.

      To compound the matter, bank ATM networks are all run from Auckland, so people couldn't get cash out either. And similarly they couldn't use electronic credit card transactions because that relies on the EFTPOS network.

      The financial implications are pretty severe, though it was only for four hours and on a week day so they're well below what could've been.

      --
      "God, root, what is difference?" - Pitr, userfriendly
  14. I hear over in Australia by Anonymous Coward · · Score: 2, Funny

    A dingo ate their cable.

  15. Rats? Puerto Rico by ArmorFiend · · Score: 2, Funny

    Any other Puerto Rico players in the house thinking "man, and I thought having my 3 tobacco eaten by rats was bad, but internet, oh yeah".

  16. Nothing but sympathy by Gleenie · · Score: 2, Interesting

    I work for their competitor. Most people in operations here had nothing but sympathy - that kind of coincidence is nothing short of one in a million.

    And it just goes to show: Murphy rules, stuff breaks. At least in this country most people are willing to accept that...

    --
    -- Your mother uses Emacs.
    1. Re:Nothing but sympathy by taniwha · · Score: 4, Informative

      I call bullshit, I'm weeping croc tears here - it's now happened twice now in the past 6 months for me (in Dunedin), previous one was a backhoe someone put thru the fiber north of here, you may have redundancy where you live but Telecom here is always one accident away from me not being able work at any time. Given the way they are making money hand over fist from their monopoly you'd think they'd spend some money on infrastructure (I can call Auckland from my US Vonage account for 1/3 what it costs to use my telecom phone - the town is plastered with fliers for 1c/min calls to China, why not Auckland?)

  17. That's where the free market steps in. by CyricZ · · Score: 2, Insightful

    Indeed, there are economic repercussions from extra redundancy. That's where the free market steps it. It will result in redundancy equilibrium: the amount of redundancy the marketplace is willing to financially bare. In this case, there is a possibility that their two lines were not enough, and the redundancy equilibrium may now shift to three lines as a result of this systemic failure.

    --
    Cyric Zndovzny at your service.
    1. Re:That's where the free market steps in. by Nutria · · Score: 2, Insightful

      So you're saying that 2 didn't work; but 3 will definately be worth the costs?

      Read more carefully next time. Grandparent wrote redundancy equilibrium may now shift to three lines.

      --
      "I don't know, therefore Aliens" Wafflebox1
  18. State of NZ broadband by Anonymous Coward · · Score: 5, Informative

    http://computerworld.co.nz/news.nsf/NL/FC31E734EFD D0739CC2570290016D8F1

    Telecom is an American owned company.

    The local loop they use to fleece NZ residents who use their sub-standard "broadband" (Telecom once tried to market 128k plans as broadband..) is in fact publically owned. As the NZ Commerce Commision has no balls Telecom remains in control of this and thus continue to be a greedy monopoly.

    The above article should remove any doubt of this.

  19. That's being unkind by lheal · · Score: 4, Funny

    to the rats.

    Actually, I love lawyers. They taste like chicken.

    --
    Raise your children as if you were teaching them to raise your grandchildren, because you are.
  20. This just proves that old addage about backhoes by Almost-Retired · · Score: 3, Funny

    being the absolute best at finding cables, beating any other method by at least 2 orders of magnitude.

    Like the most recent joke says "when lost, bury a short piece of cat5, then ask the backhoe operator the way home".

    He will come and find the cat5, it would be a violation of Murphy's Law to do otherwise.

    --
    Cheers, Gene
    "There are four boxes to be used in defense of liberty:
    soap, ballot, jury, and ammo. Please use in that order."
    -Ed Howdershelt (Author)

    1. Re:This just proves that old addage about backhoes by Merlisk · · Score: 2, Insightful

      In Telecom, we always say to bury the fiber near the power lines. The reason is that when backhoe operators are digging, the saying goes, "If fiber's under there, dig carefully. If power's under there, call before you dig."

      I can attest that it works!

      --
      Failure is not an option. It comes bundled with your Microsoft product. -- Ferenc Mantfeld
  21. Not my problem! by Bananatree3 · · Score: 3, Funny

    I run all of my data over cat5 cables, and I can rely on them ALWAYS to keep the rats away.

  22. Had a bad feeling about this by quokkapox · · Score: 2, Funny

    I *knew* we shouldn't have used Room 101 for the server room!

    --
    it's a blue bright blue Saturday hey hey
  23. Re:Would compensation really help? by taniwha · · Score: 5, Insightful

    I lost a day's work, but they still charged me for my DSL even though I couldn't use it - you figure out who's being ripped off here!

  24. NIMH by sremick · · Score: 3, Funny

    Damn... tapping into the farmer's electricity wasn't enough. Now those smart bastards want high-speed internet access.

  25. Ben by Anonymous Coward · · Score: 3, Funny

    Ben, the two of us need look no more
    We both found what we were looking for
    With a friend to call my own
    I'll never be alone
    And you my friend will see
    You've got a friend in me
    (You've got a friend in me)

    Ben, you're always running here and there
    (Here and there)
    You feel you're not wanted anywhere
    (Anywhere)
    If you ever look behind
    And don't like what you find
    There's something you should know
    You've got a place to go
    (You've got a place to go)

    I used to say, "I" and "me"
    Now it's "us", now it's "we"
    (I used to say, "I" and "me")
    (Now it's "us", now it's "we")

    Ben, most people would turn you away
    I don't listen to a word they say
    They don't see you as I do
    I wish they would try to
    I'm sure they'd think again
    If they had a friend like Ben
    (A friend)
    Like Ben
    (Like Ben)
    Like Ben

    1. Re:Ben by Zontar+The+Mindless · · Score: 4, Insightful

      Note for moderators: "I don't get it" == "Off-Topic" evaluates as false.

      Parent is quoting the lyrics to the theme song from the movie of the same name. The movie's title character is... wait for it... a rat.

      The reference is made even more timely given that the song was performed by Michael Jackson, who's been in the news quite a bit lately.

      Parent might or might not be be indulging in copyright infringement rather than "fair use", and might or might not be sufficiently entertaining to deserve a +1 Funny mod, but does not by any reasonable stretch of the imagination merit -1 Off-Topic.

      --
      Il n'y a pas de Planet B.
  26. The 'Internet' no longer exists in New Zealand. by ikekrull · · Score: 4, Insightful

    The Internet was designed to survive a nuclear attack, but as implemented by the greedy idiots in control of Telecom and TelstraClear, the internet in New Zealand can't even withstand an assault by frickin rat and some clown with a power tool.

    Its really embarrassing, and sad, that these corporations' policies effectively deny the public in New Zealand a robust network infrastructure - Telecom and TelstraClear purposely depeered from various internet exchanges through which they could have easily redirected traffic during this outage, rendering it a non-event, because they can't stand the idea that people might run VOIP systems, cutting into their monopoly profits.

    Its just really pathetic that their action has resulted in the biggest network outage
    that I can remember, and instead of realising they probably should work with other network providers to ensure the internet in New Zealand is resistant to this type of 'attack' they just want to blame the power company.

    Telecom, you suck.

    --
    I gots ta ding a ding dang my dang a long ling long
    1. Re:The 'Internet' no longer exists in New Zealand. by KiwiSurfer · · Score: 2, Interesting

      What a troll. Please mod the parent post a troll.

      ...Telecom and TelstraClear purposely depeered from various internet exchanges through which they could have easily redirected traffic during this outage,...

      The ISPs Telecom and TelstraClear previously peered with all had their Auckland-Wellington routes on either Telecom or TelstraClear. No other ISPs, as far as I know, have their own cable linking Wellington and Auckland. So Even if Telecom was peered with other ISPs they would still have overloaded so much traffic onto TelstraClear's network that it would probally slow to a crawl. Therefore peering wouldn't be a great idea in this situation. Although I do agree that they should kept their peering agreement with the Auckland and Wellington peering exchanges.

      -James

    2. Re:The 'Internet' no longer exists in New Zealand. by jonbrewer · · Score: 2, Informative

      No other ISPs, as far as I know, have their own cable linking Wellington and Auckland.

      You've neglected to consider BCL, who have 4x 155mbps DMR up and down the two islands. That's easily enough capacity to handle all of NZ's Intraweb south of Auckland at this point in time. Since they're a State Owned Enterprise, they can't exactly tell Telecom to fuck off if Telecom comes to them asking for redundant capacity.

      Even if Telecom was peered with other ISPs they would still have overloaded so much traffic onto TelstraClear's network that it would probally slow to a crawl.

      You've also failed to consider the capacity of Telstra's network. Fibre is fibre, and when you have a pair of bundles of fibre running up and down the country, it's not terribly difficult to come up with more gigabits per second than an Advanced Network would know what to do with, let alone the pittance of commercial web traffic this country sees.

  27. availability and compensation by Rogue+Animal · · Score: 2, Insightful

    Fibre optic cables are relatively expensive, say $60k/km rural & rising to up to >$1m/km downtown, so beyond two diverse cables is usually justifiable only where the traffic is very large (hint- not NZ), or where geography favours it (think a mesh of cities).

    The availability with two cables can be 5 9s. Overall availability depends on how risk there is (most likely risk - construction near the cable e.g. new housing estates), and how fast the operations is fixing faults. Fixing a fault may take 3 - 12 hours, and it is when Field Ops gets a bit slack in fixing things that the risk rises dramatically. Theoretically this is very rare, but these double faults do happen (think Optus Melbourne - Sydney about 5 years ago - and periodically others).

    As for compensation. In general those who constructed other facilities have a responsibility to check for the presence of others external plant (gas, elec, fibre optic) - In OZ: Dial before you dig. If you don't do this, then you are liable - and be sued for at a minimum costs. If you have, and the records are off, then you are probably OK. So, in this case it may be that the electricity company or their contractor did not check for other services (or most likely checked, but stuffed up anyway!).

  28. Eerr... by EvilCabbage · · Score: 4, Funny

    "They are also fat and delicious." The Maori people, or the rats?

  29. Re:One in a million? Hardly by rimu+guy · · Score: 2, Interesting

    One in a million? So telecom says. But you're talking about a loop going around the whole North Island. That's a lot of cable. At any point in the loop it is subject to being dug up, eaten by rats, or affected by an equipment malfunction. The odds of any of those are, what?, once every 30 days. You probably never notice any single outage because of the redundancy. But the odds of a dual failure, hence are 1/30days*1/30=1/900days=1/2.4 years. Hardly one in a million. For it to be that there would be a fault only once every thousand days. And that is not the telecom I know and, well, have to use.

    --
    Working to provide reliable VPS hosting

  30. Happened at my parents house too by Anonymous Coward · · Score: 2, Interesting

    Years ago, my parents called me because the power to half of their house went dead.

    I checked the breaker and it was fine, and power was also leaving it.

    So I started tracing the wiring from the Garage back the the Main circuit Panel on the other side of the house.

    Up into the attic I went, still finding no power in the 08 gauge feeder cable, until I got to the outside wall just above the mains panel.

    At this point I was truly surprised, because somewhere inside that 36 inches of wall, there just had to be a break in the wire!

    The next part was really FUN , since I had to use a sledge hammer to break open the stucco wall, outside and above the mains panel to see what was going wrong.

    When I discovered the cause, I was totally shocked, a rat had chewed through the hot 08 gauge wire of the 3 wire cable!

    I could see the teeth marks on the wire, and tiny copper chips laying below it on the fire block in the wall.

    The rat didn't get electrocuted because it never touched the other conductors, which would have killed it instantly.

    It's almost too weird for words, but I saved a 12 inch piece of cable where the break was to show to my clients (electrical contractor).

  31. yeah its tough being a tight arsed monopoly by webname · · Score: 2, Funny

    bloody cabling is so crappy here i cant get dsl
    serves em right
    go rats- umm thats the furry ones i mean not the 2 legged ones :)

  32. The French by Aussie · · Score: 3, Funny

    It was French rats, with scuba gear.

  33. This shows that 'competative commerce' is a ... by chris_sawtell · · Score: 2, Informative
    ... total and absolute write off, particularly for small countries.

    What all the published articles fail to mention is that there is a third fibre running from one end of the country to the other. It is owned by Telstra, the competition. It would not be too much of an exaggeration to say that Telstra and Telecom have such a level of psychotic hatred for one another, that they cannot talk to each other except in a Court room. Thus the very idea of setting up the routers so that all three of the fibres are shared is such an anathema that it just won't happen without Government regulation and intervention. Needless to say the Government is essentially a bunch of ignorent wimps who can no more understand the technicalities of the situation than fly. So it won't happen and we will have to suffer the consequencies of serious telecom infrastructure failure from time to time.

    It's time for the little peoples of the world to take back ownership of their infrastructures by whatever means necessary. Fighting talk may be, but many of us in the rest of the world are sick of being fleeced by the avaricious in the powerful countries. Oh shit! - I forgot - big countries make up excuses to invade little ones so they can steal their natural resources.

  34. NZ lost power to largest city for 5 weeks in 1998 by evilandi · · Score: 2, Informative
    This sounds very familiar to when in 1998, all four major mains electricity power trunk cables failed to their major city, Auckland, for five weeks.

    There seems to be something about the New Zealand psyche that just doesn't understand the concept of separate routing and protection of cables.

    --
    Andrew Oakley - www.aoakley.com