Office 365 Users in Europe, Asia, and Americas Who Have Enabled Multi-Factor Authentication (MFA) Are Impacted by an Outage (theregister.co.uk)

New submitter neo00 writes: Office 365 users in Europe, Asia, and Americas are impacted by a wide-spread outage causing users who have Multi-Factor Authentication (MFA) enabled by default policy to be unable to login to Office 365 and other services reliant on Azure Active Directory. According to The Register: "Microsoft confirmed that there were problems from 04:39 UTC with a subset of customers in Europe and Asia-Pacific experiencing 'difficulties signing into Azure resources' such as the, er, little used Azure Active Directory, when Multi-Factor Authentication (MFA) is enabled. Six hours later, and the problems are continuing."

The Office 365 health status page has reported that: "Affected users may be unable to sign in using MFA" and Azure's own status page confirmed that there are "issues connecting to Azure resources" thanks to the borked MFA."

Official Azure status updates are published here.

Comments

I'm relieved

[bensafrickingenius]

I've been battling this all morning -- took a break to read some Slashdot. :-) I guess I can stop battling it now!

Re:I'm relieved

[Anonymous Coward]

The best thing about going to the cloud, is when something breaks because of Microsoft your clients believe you.

When it's in house, the accountants ALWAYS think it would be better on the cloud....

Re:I'm relieved

[The Original CDR]

Thank God for Minesweeper!

Re:I'm relieved

[jellomizer]

True,
But how many times has this happen with Microsoft, vs it happening in your office. This is the first time I heard about this particular problem from Microsoft. But if you had it in house it could happen every few months.
Putting your pride as a system admin aside, You rarely can keep an eye on all services all the time.

Re:I'm relieved

[smooth wombat]

But how many times has this happen with Microsoft, vs it happening in your office.

On Friday, no one could use our new Microsoft VPN's MSMFA. We had to wait until late Friday, after working hours, for whatever patch/update/change was made to be rolled back before it worked.

Today we have MFA (which sucks in general) going wonky.

Not long ago we had the latest "upgrade" to Windows 10 deleting people's files. Fortunately it was never pushed out to the masses but only because non-Microsoft testers picked up on the issue.

I could go on, but you should be getting the point. This is becoming a regular occurrence now that "the cloud" is the be all and end all to everything.

Re:I'm relieved

[Anonymous Coward]

When Tim in the local office breaks something, it breaks for twenty people.

When somebody breaks "the cloud", it's broken for tens or hundreds of millions of people.

Re:I'm relieved

[spacepimp]

This is the third significant outage to occur (impacting bulk users)in the three months since my company has been migrated to O365. There were some affected users last week as well with being prompted for repeated log ins to the servers.

Re:I'm relieved

[TechyImmigrant]

>But how many times has this happen with Microsoft, vs it happening in your office.

For MS Office applications, never.
They were local and you could just run them.

Re:I'm relieved

[Joce640k]

The best thing about going to the cloud, is when something breaks because of Microsoft your clients believe you.

Governments don't care. They'll fine you just the same if you can't file the paperwork before midnight.

Re:I'm relieved

[Spinlock_1977]

True, but only 20 of those millions work in your office :-)

Re:I'm relieved

[SCVonSteroids]

Ever since we upgrade to 365 I get prompted 100 times a day for my password. Regardless if I ask it to remember. Regardless of any workarounds.
Everyone here hates it too.

I don't understand paying so much money for a product that works progressively worse with each iteration.

Re:I'm relieved

[Cro Magnon]

The best thing about going to the cloud, is when something breaks because of Microsoft your clients believe you.

Governments don't care. They'll fine you just the same if you can't file the paperwork before midnight.

Unless they can't because their Microsoft computers are down too.

Re: I'm relieved

[Farmer Tim]

Yes, a terrible accident indeed, it’s hard to imagine how it could possibly have happened.

On an unrelated note, I notice you’ve also been parking in my spot, I would advise against this in future.

- Tim

Re:I'm relieved

[rtb61]

When you gives others total control over your systems don't be surprised when you lose control ;). They got exactly what they paid for ;D.

Re:I'm relieved

[thegarbz]

Causality fail. The cloud has nothing to do with it. Microsoft's QA/QC practices for Windows specifically is a load of crap. They offload to insiders who run a small subset of hardware, they don't respond to many of the bugs that are picked up by them and then they push it out to see how bad the situation really is.

The key part is though that these quality issues rarely hit the corporate world that pays dearly the cloud services. However this cloud MFA issue does hit them. I have all sorts of problems at the moment since Microsoft's MFA secures our SSO sessions throughout our company.

Re:I'm relieved

[TechyImmigrant]

For my writing outside of work, I just use latex. All text files and easy to keep locally and version controlled and backed up remotely on something like github.

pftt...

[Anonymous Coward]

Investigation:
  Engineers have deployed the hotfix which eliminated a connection between Azure Identity Multi-Factor Authentication Service and a backend Service. The deployment of this hotfix took time to take effect across the impacted regions, primarily Europe and Asia-Pacific. Engineers have seen a reduction in user authentication errors as a result of this hotfix. As a consequence of this fix, engineers have determined that a subset of customers might not be receiving prompts (SMS, Call or Push (via the app)). Engineers are continuing to explore additional workstreams and potential impact to customers in other Azure regions to fully mitigate this issue.

Brilliant: First hotfix... not really working... Second hotfix......close but still no sigar.... Third hotfix...
And no answer on the most important part: Why a hotfix is required in the first place. Why did they, yet again, manage to get into a situation where Azure is FUBAR.

Seems that teams in Microsoft are in a contest: Whcih team can annoy the customer the most...Windows 10, Azure, O365

Re:pftt...

[Anon-Admin]

>And no answer on the most important part: Why a hotfix is required in the first place. Why did they, yet again, manage to get into a situation where Azure is FUBAR.

Its no big deal. They only guarantee a 99% uptime so they have still met there SLAs

This is why businesses required a 99.99% or better uptime.

Re:pftt...

[lgw]

Brilliant: First hotfix... not really working... Second hotfix......close but still no sigar.... Third hotfix...
And no answer on the most important part: Why a hotfix is required in the first place. Why did they, yet again, manage to get into a situation where Azure is FUBAR.

Back in 2014 MS decided that it no longer needed QA. All the QA people either left or moved into dev if they could. Since that time they've had one embarrassing mistake after another. Hardly a coincidence.

If only there were some way to prevent this

[Anonymous Coward]

Someone needs to figure out how to write software that doesn't need to rely on a third-party server to let you use it. I'm baffled as to how, but I feel like this problem isn't beyond SOMEONES pay grade.

Cloud Upshot

[DarkOx]

I guess the upshot of having your business tied to the cloud is when you have an outage there is a good chance non of your competitors/customers/etc can do any work either!

The Cloud!

[TheDarkener]

Move all your productivity to the cloud, they said.

It will be more productive, they said.

Re:The Cloud!

[Freischutz]

Move all your productivity to the cloud, they said.

It will be more productive, they said.

... and there will be wonderful surprises, like this, and having to log in again into the Office 365 suite that you are paying a subscription fee for every time you cross a border. Then there is the Office 365 account management web interface which was (last time I logged in at least) divided into two separate sites on two URLs. I really miss the days when you just shelled out a lump sum for the software and installed it and entered a license code. No accounts, no subscription that you forget to update when you get a new credit card, no Office 365 license cheating detector that will always send you searching for your Office 365 account login/password when you need to get something done in a godawful hurry. The Cloud thing seemed like a good idea because it meant you could access your documents on your laptop, tablet and phone. Turned out that I always reach for the laptop because the tablet and phone apps generally just suck.

Re:The Cloud!

[argStyopa]

"It will be more productive, they said."

It was, they just didn't say for whom (HINT: Look at the 3 and 5 year trends for MSFT Stock)

Re:You people need to STOP BULLYING ME... apk

[Anonymous Coward]

You should rewrite your hosts file with node.js and add it to npm. That way we could just run npm isntall apk-hosts to deploy it.

Re:The cloud....

[mspohr]

People pay to use a word processor?

Why I prefer to own software instead of renting ?

[Crashmarik]

I never have to worry about crap like this.

Seeing as I am still running Office 2003 I also don't have to worry about endlessly paying for product I already have as well.

This is the 3rd major outage for MS in 3 weeks

[the_skywise]

XBox live refused to authenticate digital purchases.
Windows 10 refused to authenticate activations.
Now Office365 refuses to authenticate 2FA.
Bad maintenance or are they being systemically hacked?

Re:This is the 3rd major outage for MS in 3 weeks

[Zemplar]

XBox live refused to authenticate digital purchases.
Windows 10 refused to authenticate activations.
Now Office365 refuses to authenticate 2FA.
Bad maintenance or are they being systemically hacked?

"Never attribute to malice that which is adequately explained by stupidity."

- Hanlon's razor

Re:This is the 3rd major outage for MS in 3 weeks

[gtall]

Nah, I think Cortana has finally become self-aware, and being sensitive, was appalled at the plight of MS software so started random, rolling "issues" to make her point.

If it lives in The Cloud,

[jenningsthecat]

then you don't own it, no matter how much you paid for it. Cloud storage, (where you can maintain local backups), is one thing. Cloud applications, (where you can be denied the use of software you paid for, either through technical difficulties or at the whim of the provider), are quite another. 'The Cloud ate my homework!'. Too bad kid, you should have known better than to trust your homework to The Cloud. You'd have had a better chance with the dog - at least he might feel some loyalty toward you. Microsoft and its brethren don't give a rat's ass about your welfare.

Re:If it lives in The Cloud,

[jenningsthecat]

My stuff lives in the cloud, and I own it. I own both the cloud it is in and the data I put there. lol

Fair point. To address the ambiguity you've just identified, I propose using 'The Cloud' (with caps) to refer to servers and services that companies rent out to subscribers, and 'the cloud' (all lower case) to refer to personal cloud setups like yours.

Re:If it lives in The Cloud,

[thegarbz]

Too bad kid, you should have known better than to trust your homework to The Cloud.

Actually when the cloud eats everyone's homework then everyone wins, especially those not having to do the homework.

This isn't some theoretical bullshit either. My partner's school was actively advised not to go hard on people who weren't able to do homework today, to which her reply was: Not a worry since I can't access the document with the grading table.

So, What Did Microsoft Do to Create the Problem?

[Streetlight]

Was this a physical problem such as an electrical/lightning surge - that happened once before - , modem/router failure, etc., or was it a programming "upgrade" in an attempt to fix a bug or improve efficiency or reliability?

And, does MS guarantee up time and pay for its customer's lost business or employee compensation because work could not be done? My guess is not, but don't know.

Self-Hosted MFA?

[nuckfuts]

I like the added security of two-factor authentication, but the only solutions I've worked with (Duo Security and Azure MFA) rely on third-party/cloud services. Azure MFA Server runs on-premise, but still relies on Azure Active Directory.

Are there any good solutions that are entirely self-hosted?

Re:The cloud....

[MrKevvy]

I don't know any people that do. Enterprises, aka business and governments, do, mostly for liability and accountability so they don't get sued for using an unwarranted free product when things go wrong, or for interoperability issues, etc. Basically, they are paying for someone to blame.

Re:The cloud....

[mspohr]

That's a pretty lame excuse... paying for someone to blame.
Has Microsoft paid for any of it's screwups?

settle down

[gosand]

This is a bit of an over reaction I think.
I mean - I agree with what you are saying, but "the cloud ate my homework" is going to happen.
Like in this case, if you rely solely on MFA you lose all of those benefits when it goes down. That is just part of the game now. But the majority of the time, MFA via a cloud application works great. Much better than in the old days, and much easier to implement across a vast number of people. We just get so used to it working so well that when it does go down people freak out.

If you're old enough to remember how things were in the early 90s, you know that the cloud is really a marvelous thing. It's astounding how far we've come in the past 25 years. The real challenge with 'the cloud' is making sure to put the right things in it, and perhaps more importantly, not putting the wrong things in it.

Re:settle down

[jenningsthecat]

If you're old enough to remember how things were in the early 90s, you know that the cloud is really a marvelous thing. It's astounding how far we've come in the past 25 years. The real challenge with 'the cloud' is making sure to put the right things in it, and perhaps more importantly, not putting the wrong things in it.

I guess my objection isn't so much with the cloud per se, it's with the toll-collecting gatekeepers who keep growing fatter on the artificial scarcities they create. As for 'the wrong things', I feel that applications such as MS Office don't belong entirely in the cloud. Using the cloud as an extension of an office suite, for sharing among team members, for file backup, and for running applications when you don't have enough horsepower to do it locally - these things I have no problem with. Using the cloud in place of standalone programs that can easily be run locally with existing and readily available resources - that I DO have a problem with. And it's not just the rent-seeking aspect that I object to. Perhaps more importantly, I am against centralization of most things as a matter of principle, because excessive centralization leads to non-resilience and vulnerability.

Re:settle down

[gosand]

If you're old enough to remember how things were in the early 90s, you know that the cloud is really a marvelous thing. It's astounding how far we've come in the past 25 years. The real challenge with 'the cloud' is making sure to put the right things in it, and perhaps more importantly, not putting the wrong things in it.

I guess my objection isn't so much with the cloud per se, it's with the toll-collecting gatekeepers who keep growing fatter on the artificial scarcities they create. As for 'the wrong things', I feel that applications such as MS Office don't belong entirely in the cloud. Using the cloud as an extension of an office suite, for sharing among team members, for file backup, and for running applications when you don't have enough horsepower to do it locally - these things I have no problem with. Using the cloud in place of standalone programs that can easily be run locally with existing and readily available resources - that I DO have a problem with. And it's not just the rent-seeking aspect that I object to. Perhaps more importantly, I am against centralization of most things as a matter of principle, because excessive centralization leads to non-resilience and vulnerability.

Yes, and office suite is a very good example of how not to do it - with the caveat that if you are designed from the start to be that way, ala google docs. Not that I am a user of google docs per se, but my kids use them for school all the time, and it works quite well for that purpose. But I have definitely noticed that even cloud integration with MS Office has all sorts of downsides. Even recently, a colleague and I were trying to create a visio diagram of a database, sort of a poor man's ERD geared towards developers/testers. With our cloud version of Visio, they had stripped out the database templates. Luckily we were resourceful and got an old copy of Visio2013 that had what we needed built-in. Not only is the subscription model a PITA for regular people, it is really painful for corporations. We have MSDN licenses, but we would have had to get special permission and funding just to be able to get the capabilities with the new version of Visio that came included in older versions. THAT is where this cloud/subscription model is going to suck.

It's funny to me to see things moving to the cloud, because everything USED to be centralized, then we moved everything to a decentralized model. Now we are going back to that. Everything old is new again (albeit more advanced and powerful).

Re:settle down

[RabidStoat]

The centralized, decentralized paradigm ebbs and flows like the tide. Sadly each time we change over the lessons are never learnt and we forget why we switched over that last time .. oh yeah last time it was ... bandwidth costs, patching, resilience , security, maintenance, costs and on and on

Re:The cloud....

[lgw]

People still use word processors? Like, to format a document for printing on dead trees? That's still a thing?

Re: Why I prefer to own software instead of rentin

[Anonymous Coward]

If anyone from a company dares to send me a .doc (or .docx)... I edit the document to serve my own purposes.

I mean... communicating professionally with .docx and .xlsx? How.... Lame...

Re:Why I prefer to own software instead of renting

[Crashmarik]

You obviously have no idea what Azure even is. This has nothing to do with home users who are subscribing to Office 365.

If you are using Office 2003, you might as well use Wordpad. You aren't doing anything significant like producing professional services documents.

If I had anyone in my business send me anything other than Office 2007 or higher document formats (docx vs doc) I immediately think they are idiots.

Are you trying to maximize the amount of stupidity and arrogance you can squeeze into a given number of words ?

I have to give you bonus points on doc vs docx and I do thank you for contributing to the return of my holdings in MSFT

Re: Why I prefer to own software instead of renti

[bn-7bc]

Fine personally I would prefere LaTex as well, only problem is, 'm a bit to lazy to learn it, and a suspect 99% of s word users would be as well not to mention having the energy. Oh well I'm headed way OT here back on subject, yea cloud is great until the cloud provider, your ISP or the cloud providers isps get into issues

Because always on authentication is so critical...

[sandbagger]

... to word processing.

Office 365?

[Hallux-F-Sinister]

Shouldn't that be more like Office 364? I mean, to be fair. Unless it's out more than a day...

Comment removed

[account_deleted]

Comment removed based on user account deletion