Slashdot Mirror


Server Snafu Makes Microsoft Beg For CA Audit Data From Its Partners (softpedia.com)

An anonymous reader writes: Microsoft, just like Google, Apple, and Mozilla, is part of the CA/BForum, an organization of web browser vendors and certification authorities (CAs). As a browser vendor, Microsoft maintains a list of authorized CAs and their respective root certificates. According to a message on the CA/BForum, there was an error on the server that was running a CRM application that managed this list of trusted certificates and the adjacent details regarding each certificate and CA. The data is lost forever and Microsoft is now asking CAs to resend their most recent audits. Currently a lot of certs are broken in Edge and IE. Microsoft says that it lost audit data for 147 root certificates, which resulted in many SSL/TLS certificates showing errors inside the company's products.

50 of 115 comments (clear)

  1. wtf by lastman71 · · Score: 5, Interesting

    Seriously. No backup?

    1. Re:wtf by Forever+Wondering · · Score: 4, Insightful

      Seriously. No backup?

      Maybe they used Azure for their backup ...

      --
      Like a good neighbor, fsck is there ...
    2. Re:wtf by Forever+Wondering · · Score: 2, Informative

      Actually, what seems to have happened is that they _did_ have a backup. But, they had to roll back to an old one.

      --
      Like a good neighbor, fsck is there ...
    3. Re:wtf by Anonymous Coward · · Score: 5, Insightful

      This might be the correct explanation. I have seen the technology management to actually trust on their "the cloud is the backup" fairytale. And then we lost data multiple times thanks to software or administration errors which deleted the data from all replicates. After fourth data loss the dumb ass management started to plan a real write-only backup system. Thankfully I don''t work on that company anymore as the management is still there planning for their next failures.

    4. Re:wtf by Anonymous Coward · · Score: 1

      from the actual request:

      "Our CRM system suffered a data loss, and it looks like it rolled back to an old backup. As a result, we lost audit data for about 147 roots."

      see: https://cabforum.org/pipermail...

    5. Re:wtf by unrtst · · Score: 3, Insightful

      ... rolled back to an old backup. As a result, we lost audit data for about 147 roots.

      How the fuck are there that many changes for root CA's withing the period of one backup?

    6. Re:wtf by fustakrakich · · Score: 2

      It should be on their OneDrive... you know, in the cloud

      Really, just how brittle is this "Internet"? And how will Microsoft verify these certificates? Hmmm?

      --
      “He’s not deformed, he’s just drunk!”
    7. Re:wtf by Anonymous Coward · · Score: 1

      ... rolled back to an old backup. As a result, we lost audit data for about 147 roots.

      How the fuck are there that many changes for root CA's withing the period of one backup?

      They rolled back to an old backup. Not necessarily the most recent. Perhaps all of the more recent backups were borked, and that was the most recent unborked backup...

    8. Re:wtf by zopper · · Score: 2

      So instead of people, into whom you invested a lot right now (fixing the shit), and who will be much more careful next time, you hire a bunch of new people who will do a similar mistake in few years... Everyone can make a mistake. Good employees will learn from it.

    9. Re:wtf by Anonymous Coward · · Score: 1

      The files may have been open so they weren't able to back them up so it wasn't within the period of one backup. DOS/Windows isn't like UNIX. You can't work with open files. That's why Windows has to crash completely for even minor updates.

    10. Re:wtf by Anonymous Coward · · Score: 1

      These are audit records for public CA roots. Every one of these 147 is a public CA _root_ certificate. Not an intermediary, not a site certificate, not some bozo's SMIME cert, a public _root_ that every Windows user trusts to sign any non-EV certificate.

      Some Certificate Authorities manage several CA roots, particularly the oldest ones because they issued their initial certs when nobody knew how any of this would work, there wasn't a CA/B to decide any rules, it was the Wild West. But even today a new CA might well choose to operate say three CA roots, with one EV root, one for an OV and DV business (these are valid but don't show a company name in the address bar on your browser) and one "Intranet" root that isn't a public CA root, and so doesn't have to obey CA/B rules. **

      There are probably less than a thousand public roots _in total_ in existence, and somehow whoops, Microsoft lost audit records for 147 of them.

      ** Intranet certs is a real money maker. These certs are worthless garbage, with no CA/B rules everybody can get a cert on a flimsy excuse if they have cash. Stuff that CAs got told to stop doing 5-10 years ago in their public roots is still routine in the Intranet business, things like wildcards (why have *.example.com when the CA will sell you the much more useful *.com or even *) or RFC 1918 IPv4 addresses in the cert. All with no security value whatsoever. But it's usually for a corporate client, they don't know what security is, they've been told "get an SSL cert" and they pay whatever they're asked. License to print money.

    11. Re:wtf by gmack · · Score: 3, Interesting

      It's Microsoft. Data loss from lack of backups is has happened to them before. Unfortunately they didn't learn from past mistakes.

    12. Re:wtf by Trax3001BBS · · Score: 1

      ... rolled back to an old backup. As a result, we lost audit data for about 147 roots.

      How the fuck are there that many changes for root CA's withing the period of one backup?

      Edge is involved, Win10 is a different beast, if one has the proper certs they can bypass the windows firewall. Every since Windows supplied a firewall with their OS that's been the way it has worked.

    13. Re:wtf by JustAnotherOldGuy · · Score: 1

      Seriously. No backup?

      "We're Microsoft, trust us with your data, hurr durr."

      --
      Just cruising through this digital world at 33 1/3 rpm...
    14. Re:wtf by JustAnotherOldGuy · · Score: 1

      "Our CRM system suffered a data loss, and it looks like we were too fucking stupid to have a recent backup."

      --
      Just cruising through this digital world at 33 1/3 rpm...
    15. Re:wtf by Sarten-X · · Score: 3, Insightful

      There are fallbacks, backups, and disaster recovery mechanisms. They are three different things, with three different purposes, and managers love to confuse them.

      --
      You do not have a moral or legal right to do absolutely anything you want.
    16. Re:wtf by Sarten-X · · Score: 1

      Or the first and second-level managers were the ones who laid out an effective plan, and their subordinates (whom you'd promote) didn't bother to implement it correctly.

      Perhaps it'd be better to investigate the whole situation first, rather than jump to any knee-jerk response.

      --
      You do not have a moral or legal right to do absolutely anything you want.
    17. Re: wtf by WarJolt · · Score: 2

      Manually.

      Backups should never be read by the server to ensure it has no dependency on the data.

      Backup should never be overwritten by the server to protect the backup.

      Backups should be independent verified for completeness because servers and engineers do unexpected things.

      I just made that up, but it sounds about right.

    18. Re: wtf by xlsior · · Score: 1

      Windows built-in volume shadow system let's you back up open/locked files just fine, and has for many years

    19. Re:wtf by macs4all · · Score: 1

      Seriously. No backup?

      I know. And this is the company that has been one of the most aggressive about pushing their products into "the Cloud".

    20. Re:wtf by macs4all · · Score: 2

      Actually, what seems to have happened is that they _did_ have a backup. But, they had to roll back to an old one.

      Sounds like the excuse I'd give if I was worrying about keeping my job.

    21. Re:wtf by macs4all · · Score: 1

      ... rolled back to an old backup. As a result, we lost audit data for about 147 roots.

      How the fuck are there that many changes for root CA's withing the period of one backup?

      Because they only backed up the system once, and then never actually started the backups running on their regular schedule, I'll bet.

    22. Re:wtf by macs4all · · Score: 1

      The files may have been open so they weren't able to back them up so it wasn't within the period of one backup. DOS/Windows isn't like UNIX. You can't work with open files. That's why Windows has to crash completely for even minor updates.

      I am not a real Windows Admin, but that just isn't true. Modern backups of Windows servers take advantage of a snapshot-ting capability (I think it's called VSS) so that all files can be backed-up. I have no idea how it actually works, but I know that it does.

    23. Re: wtf by lucm · · Score: 1, Insightful

      Microsoft like many other tech companies has lots of problems with middle management. Good managers get promoted quickly to more senior roles because there's constant growth and new projects; this means that what's left in middle management ris mediocre lifers or total noobs who haven't shown their potential yet. It's a dead layer with zero potential for improvement unless the company goes stale like IBM. Promoting insiders to middle management doesn't fill the void, it accelerates the spiral.

      This is one of the drivers of the flat structure that some startups are embracing with varying degrees of success. Not sure if it could help Microsoft; we'll have to wait and see how it worked at Zappos.

      --
      lucm, indeed.
    24. Re:wtf by davester666 · · Score: 1

      All these technical terms confused Microsoft management, and it all cost more money, so they checked the 'no' box.

      --
      Sleep your way to a whiter smile...date a dentist!
    25. Re:wtf by johncandale · · Score: 1

      Nah, the manager should always be the one accountable. Upper bosses have no time for "well I told them to do it and they didn't" No, it's your department, your problem. The point of a manger is to have one person accountable, otherwise you are just a team leader or a supervisor.

    26. Re: wtf by arglebargle_xiv · · Score: 2

      This bit doesn't sound right:

      Backups should never be read by the server to ensure it has no dependency on the data.

      If you never read your backups, how do you verify that the data was successfully backed up? I've seen dual-backup systems fail because, after several years of apparent backups, when the data was needed it turned out that nothing (copy #1) and the wrong data (copy #2) had been backed up.

  2. What a joke by Anonymous Coward · · Score: 1

    I wonder if these are the same people making gui design decisions for windows 10.. I bet the same department head signs both teams' checks.

    1. Re:What a joke by Etherwalk · · Score: 2

      I wonder if these are the same people making gui design decisions for windows 10.. I bet the same department head signs both teams' checks.

      They have 118,000 employees. Blaming them all is like blaming the army when you don't get your social security check.

  3. Looking Back by SuperKendall · · Score: 1

    I'd hate to be in the Retrospective meeting for THAT iteration.

    You're supposed to deliver a releasable product, not release all your products (obscure Objective-C reference counting joke).

    --
    "There is more worth loving than we have strength to love." - Brian Jay Stanley
  4. chrome by bugs2squash · · Score: 2

    can't they just download chrome or firefox and get the equivalent list.

    --
    Nullius in verba
    1. Re:chrome by Anonymous Coward · · Score: 2, Informative

      They aren't missing the certificate data, but rather the audit data associated with those certificates which is NOT stored with the certificates that are on computers everywhere. The summary is a bit misleading on that point.

  5. Time to double check my own backups by Mostly+a+lurker · · Score: 1

    If Microsoft can perpetrate something like this, I think I had better set aside some time to verify that I do not have omissions in my own backup and disaster recovery procedures.I cannot imagine having to report something like this to top management.

  6. If you don't mind my asking... by westlake · · Score: 1

    How many root certificates does Microsoft hold and how long did it take to recover the 147 that were lost? Tech news posted to Slashdot tends to be a little skeletal and runs on the principle of "better late than never."

    Microsoft says that it lost audit data for 147 root certificates, which resulted in many SSL/TLS certificates showing errors inside the company's products.

    I am curious as well about how often these certificates change. How old a backup is too old?

    1. Re:If you don't mind my asking... by Anonymous Coward · · Score: 1

      Due to a weird design decision, Windows management tools only show currently cached root certificates, not the full list (currently 343).
      More info at

        http://hexatomium.github.io/2015/08/29/why-is-windows/
        http://trax.x10.mx/apps.html

    2. Re:If you don't mind my asking... by subk · · Score: 1

      How many root certificates does Microsoft hold and how long did it take to recover the 147 that were lost? Tech news posted to Slashdot tends to be a little skeletal and runs on the principle of "better late than never."

      343 total, and they're required to be audited annually. It doesn't take a mathematician to see how old thier tarball was!

      --
      Now, if you'll excuse me, I have backups to corrupt.
  7. How long.. by subk · · Score: 1

    ..Before we find out they were running SSLv2 and got DROWN'ed?

    --
    Now, if you'll excuse me, I have backups to corrupt.
  8. Passive voice to the rescue by DNS-and-BIND · · Score: 3, Funny

    "there was an error on the server" "Our CRM system suffered a data loss" way to state the fact that a major company like Microsoft can't even run their own systems correctly. Well where are the fucking backups? Whoopsy-doodle! Looks like Microsoft is about as competent as a 15-man company at backing up critical data.

    --
    Shutting down free speech with violence isn't fighting fascism. It IS fascism!
  9. Re:Exaggerated? by Gerv · · Score: 2, Informative

    It a load of rubbish from the original author. There's no reason whatsoever that loss of this data would cause problems in IE or Edge. Removing roots from MS's program doesn't happen without human input.

  10. The realy SNAFU ist another one. by aix+tom · · Score: 1

    A system crashing and having to restore from an "older" backup is something that could happen to almost anybody.

    The one thing that got me in the article:

    "As many of you may have just noticed, our system just generated a bunch of emails informing many of you that you are subject to removal because Microsoft does not have evidence of a qualifying audit on file,"

    And that they then asked them to re-send the data....

    1) If I restore from an older backup, and know I may have (for example) lost payment data, I don't activate batch-jobs that generate demand notes to customers that possibly have already paid, and I just lost the data.

    2) Any "important" incoming data, (like for example payment data or SSL Audit data) should be backed once right when it enters the company, so that in the event of your system crashing (or your import-jobs wreaking havoc and losing it) you can re-populate it from that incoming data without having to ask your customers to supply the data again.

    So the problem is not really the crashed system, it is the general data flow.

    1. Re:The realy SNAFU ist another one. by aix+tom · · Score: 1

      No, not if the system handles something really important (and/or highly visible like this). A system will occationally break, so you use sufficient redundancy. RAID avoids loss from disk breakage. Backups avoid loss from destruction of complete systems (fire) or griveous admin mistakes. (delete wrong database...) Logging transactions on another server makes sure you don't loose what happened between the last backup and the disaster.

      I do all that. But in the event that a plane crashes right between our two server rooms which are ~500 metres apart (thus loosing all the RAID and Online-replication backups) I might still have to go back to an off-site backup, where the transaction log replication happens only every 10 minutes, so the backup might be "10 minutes old" in that case.

      Which would prompt me to start up the system (that is, after I somehow got hold of new hardware, and if me and my co-workers didn't go up in the same ball of fire that the server rooms did, which would make it "someone else's problem") , and "have a look what the state of the system is" before activating any sort of batch-jobs.

  11. Re:funniest thing by greenfruitsalad · · Score: 2

    stories such as this make me smirk but also check if my backups are working properly. they are. back to smirking.

    but seriously, how often do people normally back up? my /home directory is on a NAS with ZFS and keeps 24 hourly snapshots, 7 daily snapshots, 4 weekly snapshots and 6 monthly ones. this gets automatically synced to my secondary (backup) NAS and once a week i manually sync it to a nas at my parents' house. i lost all my data in the late 90s and never want to go through that experience again.

  12. Comedy of errors by QuietLagoon · · Score: 1

    Redmond appears to be morphing into a comedy of errors in the tech world.

  13. Re:funniest thing by Ol+Olsoc · · Score: 1

    but seriously, how often do people normally back up? my /home directory is on a NAS with ZFS and keeps 24 hourly snapshots, 7 daily snapshots, 4 weekly snapshots and 6 monthly ones. this gets automatically synced to my secondary (backup) NAS and once a week i manually sync it to a nas at my parents' house. i lost all my data in the late 90s and never want to go through that experience again.

    Mine is very similar. I can roll back quite a way, and it has come in very helpful.

    But the answer to your question is: Most regular people simply don't back up at all. And professional setups aren't always a whole lot better.

    --
    The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
  14. Re:ACRONYMS... ATTENTION WHIPLASH, BEAUHD by Ol+Olsoc · · Score: 1

    Please define the acronyms in the summaries so those of us who aren't experts in a particular topic can follow along.

    This should be at a +5. My directors always stopped presenters at dry runs every time they made a Alphabet soup statement. All it takes is giving the letters, then what they stand for, and after that people follow it just fine. And in a multi- skillset place like /., its pretty helpful.

    And be careful calling anything an acronym around here, the pedants will jump on you like crocodiles on a wildebeest. Then we'll have 50 posts on what an acronym is or isn't.

    --
    The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
  15. Re:ACRONYMS... ATTENTION WHIPLASH, BEAUHD by KGIII · · Score: 1

    Double click on the word - this will highlight it. Right click on the word and select search, this will open search in a new browser tab. You can even set up a variety of search engines as you go, they'll make it much easier for you.

    --
    "So long and thanks for all the fish."
  16. Re:Melinda gates... by HiThere · · Score: 1

    Everybody seems to know what you're talking about, but I've got no idea. Was is spam e-mail or what? (Or was it actually a Bellevue exercise studio? The first page of a Google search didn't list that, and I'd think it would.)

    --

    I think we've pushed this "anyone can grow up to be president" thing too far.
  17. exaggerated FUD by art123 · · Score: 1

    Where is the evidence of any SSL/TLS certificates showing errors? Seems like total conjecture based on poor reading of this audit data request made by Microsoft.

    This is AUDIT data, not the actual cert info. Read the details of the audit requirements here: http://social.technet.microsoft.com/wiki/contents/articles/31635.microsoft-trusted-root-certificate-program-audit-requirements.aspx

    This just means that Microsoft lost the documentation showing that the Certificate Authorities had performed their annual audit. Under normal circumstances, this might mean that those certs would be invalidated but seeing as how this was just a bookkeeping problem on Microsoft's end, they obviously won't invalidate anything.

    This is an embarrassment for Microsoft but nothing else.

  18. Re:So... by daniel23 · · Score: 1

    automated since win10

    --
    605413? Yes, it's a prime.
  19. Re:ACRONYMS... ATTENTION WHIPLASH, BEAUHD by Ol+Olsoc · · Score: 1

    IT'S AN ACRONYM IF YOU PRONOUNCE IT AS A WORD, LIKE NASA

    IT'S AN INITIALISM IF YOU READ THE LETTERS, LIKE CIA OR FBI

    BLARGARGLARGLARGLARGLARGLARGLARGLARGLARGL!!!!!

    Filter error: Don't use so many caps. It's like YELLING. Filter error: Don't use so many caps. It's like YELLING. Filter error: Don't use so many caps. It's like YELLING.

    Ya gotta stop after the third espresso!

    --
    The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.