Slashdot Mirror


Extreme Heat Knocks Out Internet In Australia

An anonymous reader writes with news that bad weather caused internet connectivity problems for users in Perth, Western Australia on Monday. But it wasn't raging storms or lightning that caused this outage — it was extreme heat. Monday was the 6th hottest day on record for Perth, peaking around 44.4 C (111.9 F). Thousands of iiNet customers across Australia found themselves offline for about six and a half hours after the company shut down some of its systems at its Perth data center at about 4.30pm AEDST because of record breaking-temperatures. ... "[W]e shut down our servers as a precautionary measure," an iiNet spokesman said late Monday night. "Although redundancy plans ensured over 98 per cent of customers remained unaffected, some customers experienced issues reconnecting to the internet." ... Users in Western Australia, NSW, Victoria and South Australia took to Twitter, Facebook and broadband forum Whirlpool to post their frustrations to the country's second largest DSL internet service provider.

17 of 103 comments (clear)

  1. One ISP is not 'the internet' by pipedwho · · Score: 4, Informative

    Yes, many people were affected, but iiNet is not 'the internet'. All the other big providers were still running just fine.

    You could post the same headline every time someone's modem cable gets knocked out or their router crashes.

    1. Re:One ISP is not 'the internet' by TapeCutter · · Score: 2

      The recent heatwave will have knocked plenty of people offline at the pole, the first hot day in a Melbourne summer is always chaos on the trains for similar reasons (hot metal expands). However this incident just appears to be iiNet's server room air-con that fell over, nothing to do with the weather.

      --
      And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
  2. Re:(in)Tolerance by talis9 · · Score: 5, Informative

    they had multiple failures. The primary air con failed, and so did the backups

    http://blog.iinet.net.au/statement-chief-technology-officer-mark-dioguardi/

  3. Statement from CTO of iiNet by Anonymous Coward · · Score: 4, Informative

    http://blog.iinet.net.au/statement-chief-technology-officer-mark-dioguardi/

    Basically both main and backup aircon went down.

    1. Re:Statement from CTO of iiNet by turbidostato · · Score: 2

      "Basically both main and backup aircon went down."

      Which is only, oh, so unsual.

      There're two (basic) kinds of high avaliability: load balancing and redundancy with their typical failure modes:
      1) load balancing: the surviving part can't cope with the aggregated load and it also goes down.
      2) redundancy: once the main fails, either the migration protocol fails or the reserve doesn't work.

      This is HA 101 knowledge but as long as CTOs can go with an "oh! who could expect two failures in a row!" instead of being fired on the spot we'll see this kind of things happening once and again.

      The URL above should have been his resignation letter.

    2. Re:Statement from CTO of iiNet by aaarrrgggh · · Score: 2

      0.4% design temperature for Perth is 36.2C. A DX system with condensers on the roof would be designed for a temporary stature of 41-43C typically. Once you get much above that, there isn't much you can do with DX; you will overload your compressors quickly. A cooling tower should be more robust, but your envelope load could have exceeded the primary system capacity.

      Typically in extreme temperatures a Tier 3 data center will need to eat into its redundancy for cooling. Tier 4 facilities should be more robust, but you would not expect 2N redundancy when you have record temperatures.

      As for load balancing and other edge conditions, it really depends on how heavily loaded a facility/portfolio is. What I expect happened was one part of their facility went down with a "hot spot" that didn't have adequate redundancy in the first place. It is an edge condition that likely required more capital than it was worth to resolve.

  4. Coincidence? by Anonymous Coward · · Score: 2, Funny

    Someone call Al Gore--as he's an expert on both the Internet and Global Warming--he'll know what to do.

  5. last summer by slashmydots · · Score: 2

    Last summer in Wisconsin, believe it or not, it got around 100F for several days and it knocked out our internet. It wasn't some morons with inadequate server cooling though. Apparently Time Warner equipment runs on 90V lines and our energy company's equipment that drops to 90V was overheating. Unbelievable! Our digital phones were down too.

  6. Re:Slow news day? by ChunderDownunder · · Score: 2

    You're a galah - the summary clearly says it wasn't a storm.

  7. It's not the heat! by Vinegar+Joe · · Score: 2

    It's the humidity.

    --
    "The average reporter we talk to is 27 years old......They literally know nothing." - Ben Rhodes
    1. Re:It's not the heat! by mjwx · · Score: 2

      It's the humidity.

      And if that doesn't get you we have sharks, snakes, spiders, jelly fish, drop bears and backpacker murderers.

      --
      Calling someone a "hater" only means you can not rationally rebut their argument.
  8. Re:(in)Tolerance by itzly · · Score: 4, Insightful

    Yes, it's unbelievable that something, somewhere goes wrong.

  9. Click bait...Nothing to see...Move along people by Anonymous Coward · · Score: 5, Informative

    Sigh... first world problems. Few services were shutdown for precaution as result of A/C failure (primary/backup).

    Hi All,

    Due to heat in Perth we have lost a number of services and precautionary shut others down.

    Customer will notice impact to the following services:

    - iiNet Toolbox and Westnet MyAccount [RESTORED 7:00PM WST]
    - Westnet Email [RESTORED 7:45PM WST]
    - Westnet Hosted Websites [RESTORED 9:00PM WST]
    - iiNet hosted email [RESTORED 7:45PM WST]
    - iiNet/Westnet/Adam/Netspace Webmail [RESTORED 7:00PM WST]
    - Customers may be unable to re-authenticate after disconnecting from the internet [RESTORED 8:00PM WST]

    A number of internal tools are also affected, which will impact our ability to respond to certain customer enquiries.

    Update 6pm WST: Due to issues with staff access, some contact centre queues have been closed. Affected queues will be reopened once the incident has been resolved.

    Update 8pm WST: Most services have been restored, Engineers are continuing to review all services impacted by the incident. Customers that were off-line are recommended to perform a modem power-cycle to get back on-line.

    Thanks,

    Basically few gen Y's screaming that they can't post their sweaty selfies for a few hours.
     

  10. At 44'C ambient a lot of air conditioners fail by ZombieEngineer · · Score: 2, Informative

    Depending on the refrigerant used it is possible that the condenser temperature (the bit exposed to the outside air) exceeded the critical point of the gas at which point it is impossible to tell the difference between liquid or gas. The trouble is phase change cooling works best (most efficient) the closer to the critical point you can go but not past it.

    The second problem is the condenser pressure would increase with increasing ambient air temperature. In the past this was enough to stall the compressor motors on a hot day.

    My guess is they went for a system with a high efficiency that should work for 99.9% of the time, that last 0.1% is the 8 hours of the year when the temperature is above 42'C (normally for Perth it is normally only an hours before the sea breeze kicks in and drops the temperature by at least 5'C). This time the temperature went up and stayed up for a period of time.

  11. Re:Internet in Oz down by mjwx · · Score: 2

    How do they get on in the outback? It must get near 50C there

    The hottest place in Australia is Marble Bar.
    http://en.wikipedia.org/wiki/M...

    It average (maximum) summer temperatures in excess of 41 C. Average yearly temps are around 35 C so it doesn't get much cooler in the winter.

    I'm certain that they would see the odd day above 50 C there.

    --
    Calling someone a "hater" only means you can not rationally rebut their argument.
  12. Re:Slow news day? by natd · · Score: 2

    It was more widespread than just a three states. I'm in the most populous state (NSW, where Sydney is) and many people at work today commended that they were down, as I was. iiNet is the #2 DSL provider and I suspect it was a lot more than this 2% they spout. Bottom line, a significant amount os Australian residential customers and business had no Internet. Someone is saying "inept isn't the Internet" but if the pipe from someones computer to the Internet isn't working, "the Internet is down" to that person.

    --
    Only big ligs use sigs.
  13. Re: Slow news day? by serviscope_minor · · Score: 2, Informative

    Perth is very remote. sure it's the state capital, but once you're out of the metropolitan area it's a thousand kilometers to anywhere else.

    --
    SJW n. One who posts facts.