Slashdot Mirror

← Back to Stories (view on slashdot.org)

Microsoft Azure's Southern US Data Center Goes Down For Hours, Impacting Office365 and Active Directory Customers (geekwire.com)

Posted by msmash on from the outage-report dept.

New submitter courcoul alerted us to Azure outage, which is affecting several customers in many parts of the world: Some Microsoft Azure customers with workloads running in its South Central US data center are having big problems coming back from the holiday weekend Tuesday, after shutdown procedures were initiated following a spike in temperature inside one of its facilities. Around 230am Pacific Time, Microsoft identified problems with the cooling systems in one part of its Texas data center complex, which caused a spike in temperature and forced it to shut down equipment in order to prevent a more catastrophic failure, according to the Azure status page. These issues have also caused cascading effects for some Microsoft Office 365 users as well as those who rely on Microsoft Active Directory to log into their accounts. The cooling system is the most critical part of a modern data center, given the intense heat produced by thousands of servers cranking away in an enclosed area. More resources: The official status page of Azure; and third-party web tracking tool DownDetector's assessment. Further reading: Microsoft Azure suffers outage after cooling issue.

57 of 86 comments (clear)

  1. Dad told you not to touch the thermostat! by Anonymous Coward · · Score: 1

    D'oh!

  2. This is why by Anonymous Coward · · Score: 5, Insightful

    I do not like software that requires you to phone home to the mothership. The second something go wrong outside of your control it borks all your work. Office 365 is a bad joke if I have ever seen one.

    Aside: Yes I know video games do this a lot but games are games and work is work.

  3. Comment by WallyL · · Score: 4, Interesting

    My employer was affected. Many employees could not authenticate to our third-party webapps because we use whatever the cloud-provided Active Directory SSO solution is. Ah, well. I wonder if this violated SLAs and we get some money back... My company is always concerned about not violating our SLAs to our customers (Saas), so hopefully we extract the same pound of flesh from our vendors.

    1. Re:Comment by Darinbob · · Score: 2

      Being dependent upon "the cloud" is not a good thing, and yet so many companies are throwing out their brains and signing up in the hope to reduce costs. The company that recently purchased my previous employer is in whole hog for Microsoft, Microsoft 360, Microsoft cloud, and anything with the word Microsoft attached, most of it all online only. To read some corporate announcements I have to log into a third party site which just seems absurd to me. When the cloud servers eventually get their inevitable downtime, I predict a lot of hand wringing.

      I haven't seen this level of slavish devotion to a single vendor since the IBM administration.

    2. Re:Comment by hawguy · · Score: 3, Interesting

      Being dependent upon "the cloud" is not a good thing, and yet so many companies are throwing out their brains and signing up in the hope to reduce costs. The company that recently purchased my previous employer is in whole hog for Microsoft, Microsoft 360, Microsoft cloud, and anything with the word Microsoft attached, most of it all online only. To read some corporate announcements I have to log into a third party site which just seems absurd to me. When the cloud servers eventually get their inevitable downtime, I predict a lot of hand wringing.

      I haven't seen this level of slavish devotion to a single vendor since the IBM administration.

      For most small to mid-sized businesses, "the cloud" is more reliable than any solution they'd be willing to pay for. I don't know Microsoft's redundancy model, but AWS's multi-AZ model gives much more redundancy than most businesses would build themselves -- even more so for multi-region redundancy since most companies aren't going to spend the money to duplicate their production environment in another region on the other side of the country (or world).

      Though the side effect of using a cloud provider is that when a major cloud provider goes down, so do a *lot* of businesses -- but that doesn't mean they would have been better off building their own datacenter.

    3. Re:Comment by Darinbob · · Score: 1

      "Datacenter:" means a lot of things. When I hear it I think of something giant to support thousands of customers, as opposed to the servers supporting just internal email and documents and backups.

    4. Re:Comment by guruevi · · Score: 1

      You should re-read your SLA. Microsoft only gives a 95% uptime guarantee for a 100% service credit and only if Microsoft agrees to the credit. They only 'promise' a 99.9% uptime for a 25% credit, that means you can be down for ~45 minutes/month before you get ANY credit and up to 1.5 days before you get a full refund.

      --
      Custom electronics and digital signage for your business: www.evcircuits.com
    5. Re:Comment by buchanmilne · · Score: 1

      The equivalent to multi-AZ is pick the region - "Southern USA" is one of 54 different regions - when deploying a resource, one simply choose which region it goes to (different regions have different costs, different packet latency profiles, different data sovereignty and sometimes other restricted use cases.

      1)If multi-AZ is "pick multiple regions", why is Azure starting (https://azure.microsoft.com/en-us/updates/azure-availability-zones-ga/ ) with multi-AZ regions? Maybe they realise AWS is onto something?

      It looks like MS would bill you for inter-region traffic, where AWS doesn't bill you for intra-region (inter-AZ) traffic. Going multi-region on Azure looks much more expensive than multi-AZ on AWS ...

      2)54 different regions? You believe Microsoft's big number on the page, when it seems to be counting:
      - Regions not accessible to you (e.g. DoD) (2)
      - Regions that you aren't allow to use (US Gov) (4)
      - Regions that aren't built yet (10)

      On their map, I count 35 public "regions", but only 2 of those are multi-AZ.

      AWS has 18 public launched regions most of which have 3 public AZs (four regions are still 2-AZ and will be addressed in future like the others that were back-filled to 3 AZs in the past 2 years). That doesn't include the 3 public regions and one GovCloud region that are currently being built.

      3)While multi-AZ should result in very few issues on AWS, if you need more availability, AWS has been building lots of multi-region tools (e.g. inter-region replication for S3, Dynamo DB, Aurora etc.). Does Azure have any real tooling for multi-region (on top of multi-AZ)?

    6. Re:Comment by thegarbz · · Score: 1

      Being dependent upon "the cloud" is not a good thing, and yet so many companies are throwing out their brains and signing up in the hope to reduce costs.

      Hardly. Being dependent upon the cloud is infinitely better than what most companies have proven themselves as being capable of.

      Remember the cloud is someone else's computer, and that someone else is quite often better at managing it.

  4. 4.5 hours downtime! by Anonymous Coward · · Score: 1

    4.5 hours downtime for us. Thanks Microsoft.

    1. Re:4.5 hours downtime! by mu51c10rd · · Score: 2

      On the positive note, as least you can blame the outage on Microsoft and not take the heat yourself for Exchange crashing and being down for 4.5 hours.

    2. Re:4.5 hours downtime! by Chris+Mattern · · Score: 1

      On the positive note, as least you can blame the outage on Microsoft and not take the heat yourself for Exchange crashing and being down for 4.5 hours.

      Ah, but your boss will tell you it was your decision to depend on a vendor who turned out to be undependable, making it your fault. And if it was in fact your decision and not something you argued against and was overridden, he has a point.

    3. Re:4.5 hours downtime! by mu51c10rd · · Score: 1

      Touché

    4. Re:4.5 hours downtime! by sjames · · Score: 1

      For that matter, even if you did argue against it and HE is the one that did it anyway, it's your fault for not being persuasive enough.

    5. Re:4.5 hours downtime! by bjwest · · Score: 1

      On the positive note, as least you can blame the outage on Microsoft and not take the heat yourself for Exchange crashing and being down for 4.5 hours.

      Ah, but your boss will tell you it was your decision to depend on a vendor who turned out to be undependable, making it your fault. And if it was in fact your decision and not something you argued against and was overridden, he has a point.

      And he has a valid point, even if HE was the one who decided to make the move. If you're not willing to argue against bad decisions of your boss, you're nothing more than a 'yes man', and useless to the organization.

      --

      --- Keep the choice with the user..
  5. Cloud - I don't think that word... by RelaxedTension · · Score: 4, Insightful

    I don't think that word means what they think it means. They need to rethink their distributed model if one data center takes down customers. Isn't the pitch for those services that they basically bulletproof for businesses?

  6. Same old same old by Tough+Love · · Score: 2

    London Stock Exchange outage blamed on Microsoft

    --
    When all you have is a hammer, every problem starts to look like a thumb.
  7. Irony by gregarican · · Score: 3, Funny

    The most ironic part is that the Azure Support Twitter account keeps pointing customers to the Azure status page. Which also happens to be down with 503 errors. Guess they could e-mail for support, unless they are using Office 365. Or request help via the Management Portal, but guess that's down too. lol.

    1. Re:Irony by PingSpike · · Score: 1

      Maybe Microsoft can move their Azure services onto Twitter's servers to improve uptime and reduce maintenance costs.

    2. Re:Irony by gregarican · · Score: 1

      And the POTUS will avail himself to perform thorough regression testing at all hours for any new releases.

    3. Re:Irony by jaa101 · · Score: 1

      Having an estimate of when a service will be back up is valuable information. Just because contacting support won't immediately resolve an issue doesn't make doing so useless.

    4. Re:Irony by fjordboy · · Score: 1

      ETAs are so helpful. When this first started yesterday morning, I assumed maybe a couple of hours of downtime - AT MOST. Now we're nearly 30 hours in, with no ETA, and my builds for alternative systems started hours behind. Learning experience for me I guess.

      --
      The anti-salmon
  8. Stay calm and keep paying the subscription fees by Anonymous Coward · · Score: 1

    Who wants the reliability of locally installed software when you can have Office 365 running on 'the cloud'. With Office 365 your staff can enjoy a nice break at your expense until the service comes back online. Think of the moral benefits this offers, and it's only available with 'the cloud'. As for Azure, what's a few hour outage if it's just some critical business systems you're running.

    You don't want to have control over your IT infrastructure. It's far better to pay for subscription based services and let us fuck up on your behalf. And think of the cost saving. While an Office licence was around $200, now you can pay $1200 over ten years for something that's less reliable.

    The cloud is your friend. Subscriptions are your friend. Keep paying the subscriptions.

  9. Minor correction by sjames · · Score: 5, Funny

    It is now "Office 364"

    1. Re:Minor correction by Anne+Thwacks · · Score: 2

      Not so much "cloud" as "smog".

      --
      Sent from my ASR33 using ASCII
    2. Re:Minor correction by The+Original+CDR · · Score: 1

      Revised version: "Office 365 +/- a day as the new intern plays around with the power switch."

    3. Re:Minor correction by wildfish · · Score: 1

      thanks for the laugh

    4. Re:Minor correction by guruevi · · Score: 1

      Microsoft's SLA is 95%. So it's more like Office 346.

      --
      Custom electronics and digital signage for your business: www.evcircuits.com
    5. Re:Minor correction by thegarbz · · Score: 1

      Right. Now we can add 4th September to the list that already includes 29th February.

  10. Texas? by Anne+Thwacks · · Score: 3, Funny

    You have something that needs to be cool, and you put it in Texas?

    --
    Sent from my ASR33 using ASCII
    1. Re:Texas? by jbengt · · Score: 1

      My first thought as well.

    2. Re:Texas? by jittles · · Score: 1

      You have something that needs to be cool, and you put it in Texas?

      Faithful slashdot reader here for 60 years. Sorry, couldn’t be bothered to read the summary. What’s this got to do with Texas? I’ve seen clouds there before! Does Microsoft own one of them now?

    3. Re:Texas? by Fencepost · · Score: 1

      If you have something that needs a ton of electricity, you put it where the rates are low and likely to remain low. Based on May 2018 commercial rates I found, Texas has the third lowest commercial electric rates in the country, behind only Arkansas and Nevada. If Microsoft wants to get on the renewable energy bandwagon, Texas already has massive wind farms (probably 30,000 megawatts and close to 20% of the state's electrical production) and could also add solar in a lot of the state.

      It's also a major enough business and residential player to be someplace where it makes sense to put a data center; Arkansas not so much.

      --
      fencepost
      just a little off
    4. Re:Texas? by buchanmilne · · Score: 1

      If you have something that needs a ton of electricity, you put it where the rates are low and likely to remain low.

      How easily can electricity be transported, compared to heat?

      How much does the extra cooling contribute to electricity consumption, and does the lower electricity price compensate?

  11. No wonder they're a distant second by ilsaloving · · Score: 1

    Microsoft wants to be a cloud provider but they can't keep their infrastructure up worth a damned.

    I honestly don't understand how they can be in second place considering how often they have major news-worthy failures.

  12. I think this affected some XBoxLive stuff too by the_skywise · · Score: 1

    I was unable to access some of my game saves for Fallout Shelter this morning saying the cloud sync was failing. (and it stupidly deleted the local save)

  13. And this... by Anonymous Coward · · Score: 1

    And this is why you don't keep critical files and documents only in the cloud.

    1. Re:And this... by sjames · · Score: 2

      Or the software you need to view them or the services you need to keep machines on your LAN running.

  14. Well crap... by laosland · · Score: 1

    I'm almost done setting up a hybrid configuration now :-(

  15. Infrastructure management by klubar · · Score: 2

    Isn't this what backup generators and N+1 infrastructure is for? I can understand Joe's hosting and bait shop emporium going down, but power and HVAC are pretty well solved sciences. The weather in Texas is hot -- this is not a surprise. There are lightning storms in Texas, this is also not a surprise.

    It seems like if you a positioning a data center in Texas (which there as some reasons for), you prepare for both heat and lightning. I could understand if there was an incredibly unusual weather event (asteroid landing on data center, or death rays from the moon) but hot is not unusual in Texas.

    However, when major cloud service providers it does provide an excuse for everyone else who manages a data center -- even the biggest cloud provider occasionally has an outage, so when our data center has an issue it's no worse. We say thank you Microsoft/AWS/Joe's!

    1. Re:Infrastructure management by klubar · · Score: 1

      They should have redundant chillers, air handlers and environmental controls. Especially as cooling is critical to data center functionality.

      A well designed data center has (at a minimum) redundant power feeds from two separate power networks, redundant network connections (from three or more providers) and cooling capacity. Losing a chiller or air handler should not take out the data center.

    2. Re:Infrastructure management by bobbied · · Score: 1

      Isn't this what backup generators and N+1 infrastructure is for?

      Yea, that's the theory.. However, in practice, maintaining no single point of failure fault tolerance is harder than it sounds. I've seen (and implemented) many N+1 system designs. Building them isn't too much of a problem if you have a careful plan and follow it. BUT.. Keeping it N+1 as maintenance and improvements get done is *really* hard.

      Remember that you can break N+1 redundancy by simply plugging in a device to the wrong outlet, or moving a network cable from one switch to another during some late night debugging session. Even if you get the wires in the right place, you can break the network redundancy by a simple misconfiguration of a network device. There are just too many ways N+1 can be broken over time and many of these ways don't affect anything until something goes wrong.

      It takes a lot of effort to maintain N+1, so I'm not surprised to hear that even large expert providers have issues doing it reliably.

      --
      "File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
    3. Re:Infrastructure management by Anonymous Coward · · Score: 2, Interesting

      I maintain HVAC for cell sites. EVERYONE I've worked on had TWO independent HVAC systems.

      They toggle back and forth to equalize wear and tear, but when one fails the other system takes control and sends me an email asking for attention.

      There is no reason in the world for a data center NOT to have multiple HVAC systems in place. The equipment is pocket change compared to the electronics it protects.

      It could be M$ should put more thought in the design of their data centers than was put into Win95 or Vista.

      Just saying....

    4. Re:Infrastructure management by sjames · · Score: 2

      N+1 is common. So you take the total cooling need and divide by (for example) 4, then install 5 systems of that size so you can lose one and be at full capacity. That's necessary anyway since you may need to shut one down for routine maintenance from time to time.

      Ideally you don't let everything be inter-dependant so if you lose 2, you can still get by with shutting down 1/4 of the hardware.

    5. Re:Infrastructure management by dgatwood · · Score: 1

      BUT.. Keeping it N+1 as maintenance and improvements get done is *really* hard.

      Not really. You just periodically do disaster tests. Shut down one of the 'N' periodically, and if infra that isn't supposed to go down does, turn it back on quickly, then do a post-mortem to figure out what screw-up needs to be fixed, and fix it.

      --

      Check out my sci-fi/humor trilogy at PatriotsBooks.

    6. Re:Infrastructure management by bobbied · · Score: 1

      BUT.. Keeping it N+1 as maintenance and improvements get done is *really* hard.

      Not really. You just periodically do disaster tests. Shut down one of the 'N' periodically, and if infra that isn't supposed to go down does, turn it back on quickly, then do a post-mortem to figure out what screw-up needs to be fixed, and fix it.

      Sounds great, again in theory... Taking down 1 is to temporally do away with your redundancy (if it exists) and risks causing an actual outage if it doesn't. If you have SLA's to keep, you are not going to willingly perform such tests except in rather unique situations. You never willingly give up your +1.

      The way you maintain N+1 is by first intensive design reviews of installation plans that include comprehensive wire labeling and inspection of same. Strict engineering controls for existing modifications and maintenance activities including software versions and configurations and extensive testing on representative hardware, software and configurations before fielding. Also, if you truly have strict SLA's with your customers, you have N+1 sites, not just systems, and a failure recovery plan that moves customer's services to another location.

      If you are going to do any kind of testing for N+1, it's site to site. Or you will first set up N+2, move your customers off the system under test and THEN test in ways which cannot break things for your customers. In short, you NEVER test N+1 systems which are in production use.

      --
      "File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
    7. Re:Infrastructure management by dgatwood · · Score: 1

      In short, you NEVER test N+1 systems which are in production use.

      Unless you're a fairly small operation, you need N+1 redundancy per data center, and you need multiple extra data centers to handle fluctuations in your capacity needs. If, during your low-usage periods (typically the wee hours of the morning in the country where your service is most popular), you can't lose one entire data center and still keep running, you're already screwed, because it's just a matter of time before your entire operation comes crashing down.

      In fact, this is really true even if you're small enough that your redundancy plan involves only hot failover to a backup data center. And if you don't even have that, then you don't really have N+1 redundancy, no matter how your main data center is set up.

      --

      Check out my sci-fi/humor trilogy at PatriotsBooks.

    8. Re:Infrastructure management by bobbied · · Score: 1

      You still do not physically test N+1 solutions which are in operation. Even if it's fictional N+1 redundancy because of some configuration error, it's better to stay running than risk breaking your SLA's.

      Now, that doesn't mean you don't fully test and vet your N+1 solution prior to putting it into service and then regularly by physical inspection and on paper. You can even test portions of your system which are not in service or part of the +1 redundancy, but you don't risk disrupting operational systems with silly redundancy tests, not when you have SLA's to keep.

      There will be enough natural failures to make keeping your SLA's hard enough, you don't need to add risk by doing redundancy testing. You need to regularly fail over the whole system, verify your backups actually work and stuff like that, but you don't go into the server room and flip random breakers off just to make sure everything keeps running..

      --
      "File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
  16. There is no cloud.... by klubar · · Score: 2

    There is no cloud, just other people's computers.

    Back in my day, we called this time sharing. Now you kids get off my lawn.

  17. Russian Hackers!!! by Vinegar+Joe · · Score: 1

    Turned up the thermostat!!!

    --
    "The average reporter we talk to is 27 years old......They literally know nothing." - Ben Rhodes
  18. This has to be affecting a lot of shops by your_mother_sews_soc · · Score: 1

    Where I work all our source control is hosted by TFS on Azure. So all of our checkouts/checkins and code reviews are in limbo. Surely we aren't the only ones who have bought into the whole cloud idea. I may be too old fashioned, but I have a hard time putting all my eggs in someone else's basket. Besides, isn't the cloud supposed to prevent this from happening? I'm curious to know how many shops are affected by this.

    --
    My user name was a mistake. Input wasn't restricted, my bad.
  19. So it wasn't Patch Tuesday then? by bobbied · · Score: 1

    It was the cooling equipment eh?

    I think they are making excuses for patch Tuesday.... :)

    --
    "File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
  20. How shortsighted by filesiteguy · · Score: 1

    My company has 77,000 O365 licenses and use it for our email hosting as well as storing documents on OneDrive for Business. I kept seeing warnings about the web page for o365 not working.

    I would expect that a cloud vendor such as Microsoft or AWS would have multiple redundancies in place so that any one data center going down would not affect usage.

    I just am still not a cloud fan. It works great but when it doesn't - there's nothing we can do.

    --
    The Kai's Semi-Updated Website Thingy
    1. Re:How shortsighted by DigiShaman · · Score: 1

      Admin: Say it with me - "I am a mushroom"
      User: I AM A MUSHROOM..
      User: Wait, why am I mushroom?
      Admin: Because, you're to be kept in the dark and fed shit. Now repeat!
      Admin and User together: I AM A MUSHROOM!
      Admin: See? Doesn't that just clear your mind of all the care in the world?

      --
      Life is not for the lazy.
    2. Re:How shortsighted by buchanmilne · · Score: 1

      I would expect that a cloud vendor such as Microsoft or AWS would have multiple redundancies in place so that any one data center going down would not affect usage.

      Why are you lumping AWS in with Microsoft here?

      AWS has this redundancy multiple availability zones in every region, so that especially the data storage services offered can provide the kind of redundancy that would have been useful in this scenario, but which, according to the Azure status updates, Azure doesn't have:

      "Engineers have restored access to storage resources for the majority of services, and most customers should be seeing signs of recovery."

      This is the update almost 2 days into this incident.

      When AWS had a power failure in a data centre this year (https://www.theregister.co.uk/2018/06/01/aws_outage/) there was minimal impact for a short time.

      Maybe Azure shouldn't build one "massive data centre, and instead build a few large ones.

    3. Re:How shortsighted by filesiteguy · · Score: 1

      I'm mentioning them because of the overall cloud vendor solution. I'm expecting both Microsoft and Amazon to have multiple redundant centers. We have AWS projects, Azure projects, and use hosted Office 365 for Exchange.

      The company has gone all-in with cloud.

      --
      The Kai's Semi-Updated Website Thingy
  21. Executives by Anonymous Coward · · Score: 1

    My favorite part of today was the executive screaming on the outage line about getting it back up. Sorry guys, you all put it in the cloud. Now you’ve had a longer outage than the last 5 years combined for the same price you paid for the people, process, and tech. At the same time they lost fileshares (which moved to OneDrive), workflow apps (which are on sharepoint), and every app you moved from the internal sso solution to azure ad. Super fun!