Slashdot Mirror


Server Snafu Makes Microsoft Beg For CA Audit Data From Its Partners (softpedia.com)

An anonymous reader writes: Microsoft, just like Google, Apple, and Mozilla, is part of the CA/BForum, an organization of web browser vendors and certification authorities (CAs). As a browser vendor, Microsoft maintains a list of authorized CAs and their respective root certificates. According to a message on the CA/BForum, there was an error on the server that was running a CRM application that managed this list of trusted certificates and the adjacent details regarding each certificate and CA. The data is lost forever and Microsoft is now asking CAs to resend their most recent audits. Currently a lot of certs are broken in Edge and IE. Microsoft says that it lost audit data for 147 root certificates, which resulted in many SSL/TLS certificates showing errors inside the company's products.

115 comments

  1. wtf by lastman71 · · Score: 5, Interesting

    Seriously. No backup?

    1. Re:wtf by Forever+Wondering · · Score: 4, Insightful

      Seriously. No backup?

      Maybe they used Azure for their backup ...

      --
      Like a good neighbor, fsck is there ...
    2. Re:wtf by Forever+Wondering · · Score: 2, Informative

      Actually, what seems to have happened is that they _did_ have a backup. But, they had to roll back to an old one.

      --
      Like a good neighbor, fsck is there ...
    3. Re:wtf by Anonymous Coward · · Score: 5, Insightful

      This might be the correct explanation. I have seen the technology management to actually trust on their "the cloud is the backup" fairytale. And then we lost data multiple times thanks to software or administration errors which deleted the data from all replicates. After fourth data loss the dumb ass management started to plan a real write-only backup system. Thankfully I don''t work on that company anymore as the management is still there planning for their next failures.

    4. Re:wtf by Anonymous Coward · · Score: 1

      from the actual request:

      "Our CRM system suffered a data loss, and it looks like it rolled back to an old backup. As a result, we lost audit data for about 147 roots."

      see: https://cabforum.org/pipermail...

    5. Re:wtf by unrtst · · Score: 3, Insightful

      ... rolled back to an old backup. As a result, we lost audit data for about 147 roots.

      How the fuck are there that many changes for root CA's withing the period of one backup?

    6. Re:wtf by fustakrakich · · Score: 2

      It should be on their OneDrive... you know, in the cloud

      Really, just how brittle is this "Internet"? And how will Microsoft verify these certificates? Hmmm?

      --
      “He’s not deformed, he’s just drunk!”
    7. Re:wtf by Anonymous Coward · · Score: 0

      Audit data? Perhaps there are some changelogs etc that are passed around?

    8. Re:wtf by Anonymous Coward · · Score: 0

      They should have used Windows 7. I just installed it ten minutes ago, and it already starts complaining that I have no backup software installed.

    9. Re:wtf by Anonymous Coward · · Score: 0

      ... rolled back to an old backup. As a result, we lost audit data for about 147 roots.

      How the fuck are there that many changes for root CA's withing the period of one backup?

      Read between the lines. Their backup system - like most of yours - was fucked up.

      Don't just "test" backups, use them often like a universal, poor man's git/svn.

    10. Re:wtf by Anonymous Coward · · Score: 0

      Do you have a domain with its own CA? That is one record. 147 domains requiring audit records isn't much for a browser vendor.

    11. Re:wtf by Anonymous Coward · · Score: 1

      ... rolled back to an old backup. As a result, we lost audit data for about 147 roots.

      How the fuck are there that many changes for root CA's withing the period of one backup?

      They rolled back to an old backup. Not necessarily the most recent. Perhaps all of the more recent backups were borked, and that was the most recent unborked backup...

    12. Re:wtf by zopper · · Score: 2

      So instead of people, into whom you invested a lot right now (fixing the shit), and who will be much more careful next time, you hire a bunch of new people who will do a similar mistake in few years... Everyone can make a mistake. Good employees will learn from it.

    13. Re:wtf by Anonymous Coward · · Score: 0

      Professional software. This is serious, not toys like Free software.

      You get what you pay for and everybody knows there's no such thing as a free lunch.

      BTW, can't they just copy certificates from Firefox or Chromium as a stopgap? It's just data, I think, their precious super optimal software won't be tainted by the GPL cancer...

    14. Re:wtf by Anonymous Coward · · Score: 0

      Professional software. This is serious, not toys like Free software.

      Professional software is always bad. They make changes for the sake of change - to justify a new version. And they don't concentrate on bugfixing or even useability - just the "looks" that drive buying decisions. Put a minimum of work in, to maximize profit. That is "professional" software. "Professional" is a derogatory word - the software business has so much room for conmen.

      Free sw has to be good - or it gets tossed (no monetary loss in dumping it). Source code availability also means you can get bad reviews by people who understands code. Perform or die - which is why free sw is so much better. Games are the only place where sw compete properly - so games are the only good pro sw. The rest mostly stinks.

      You get what you pay for and everybody knows there's no such thing as a free lunch.

      A lie repeated is still a lie. Paid sw gets you some polish & support - that is all. These days you can buy support for the superior free sw too though.

      BTW, can't they just copy certificates from Firefox or Chromium as a stopgap?

      If it was that simple, they'd do it. The legalities (gpl etc.) can usually be handled with money. But if it was that simple, they could simply copy from their own product. But it is not that simple - necessary information is lost forever. They can issue new certificates though - and wait for updates to trickle through the world.

    15. Re:wtf by Anonymous Coward · · Score: 1

      The files may have been open so they weren't able to back them up so it wasn't within the period of one backup. DOS/Windows isn't like UNIX. You can't work with open files. That's why Windows has to crash completely for even minor updates.

    16. Re:wtf by Anonymous Coward · · Score: 0

      Yeah, something else must be going on. Because they could easily just look at any computer on their network to get the latest patch with all of the certificates.

      The fact that they didn't do this hints to me that Microsoft's signing key must have been compromised (or worse: that they don't sign the patches that update the CAs).

    17. Re: wtf by Anonymous Coward · · Score: 0

      If you're not rebooting Windows servers several times a week then you have a much greater chance of crashes. Also, you need to reboot so you can run backups.

    18. Re:wtf by Anonymous Coward · · Score: 1

      These are audit records for public CA roots. Every one of these 147 is a public CA _root_ certificate. Not an intermediary, not a site certificate, not some bozo's SMIME cert, a public _root_ that every Windows user trusts to sign any non-EV certificate.

      Some Certificate Authorities manage several CA roots, particularly the oldest ones because they issued their initial certs when nobody knew how any of this would work, there wasn't a CA/B to decide any rules, it was the Wild West. But even today a new CA might well choose to operate say three CA roots, with one EV root, one for an OV and DV business (these are valid but don't show a company name in the address bar on your browser) and one "Intranet" root that isn't a public CA root, and so doesn't have to obey CA/B rules. **

      There are probably less than a thousand public roots _in total_ in existence, and somehow whoops, Microsoft lost audit records for 147 of them.

      ** Intranet certs is a real money maker. These certs are worthless garbage, with no CA/B rules everybody can get a cert on a flimsy excuse if they have cash. Stuff that CAs got told to stop doing 5-10 years ago in their public roots is still routine in the Intranet business, things like wildcards (why have *.example.com when the CA will sell you the much more useful *.com or even *) or RFC 1918 IPv4 addresses in the cert. All with no security value whatsoever. But it's usually for a corporate client, they don't know what security is, they've been told "get an SSL cert" and they pay whatever they're asked. License to print money.

    19. Re:wtf by Anonymous Coward · · Score: 0

      To be fair, the audit data is something CAs have to present periodically. Suppose there are only 240 roots (I'd guess there's at least twice that many in fact), and they're required to send audit data at least once every year, that's already 20 new audits every month.

      The _idea_ of the audit is that it verifies the CAs are doing their jobs properly. However over the years there has been every reason to think audits are not effective at achieving this. The auditors are paid by the company they're auditing. It is in their interests to find maybe a small problem once in a while to prove they do something, but never to say "Oh, this is a huge can of worms, turn off this CA cert right now!" because that would cut off their revenue source.

      In any case, the audits are basically a bunch of paperwork. This is, in that sense, about as exciting as if you lost the paperwork the IRS sent you in 2013. Legally you're probably supposed to keep that. But, you can probably manage not to freak out if you lose it.

      Because it has become clear that audits aren't effective, Google invented Certificate Transparency. CT says that the act of signing a certificate, at least one trusted by a public CA root, should be a public act, one that can be logged and monitored by third parties. CT participants must prove (via maths) that they're publishing every certificate they've signed with a particular CA root, if anyone tries to use an unpublished cert in a CT-implementing browser, it just doesn't work and flags up as an error. With CT monitoring, e.g. Slashdot can watch for anybody registering a "Slashdot" cert, and if they didn't authorise one, they know that's an attack in progress or a monumental screw up, without waiting until end users have problems.

      Today on Google Chrome checks for CT, and only for EV certificates. But hopefully some day more browsers will do it (Mozilla said Firefox would, but didn't specify when they'd get there) and for all types of certificate.

    20. Re:wtf by Anonymous Coward · · Score: 0

      They should have been using CrashPlan to back up their systems. My company does and it just works.

    21. Re:wtf by gmack · · Score: 3, Interesting

      It's Microsoft. Data loss from lack of backups is has happened to them before. Unfortunately they didn't learn from past mistakes.

    22. Re:wtf by Anonymous Coward · · Score: 0

      >Our CRM system suffered a data loss, and it looks like it rolled back to an old backup.

      What part of that did you not understand?

    23. Re:wtf by Anonymous Coward · · Score: 0

      First time someone missed my irony and sarcasm (whoosh!, BTW) and it actually resulted in a good thing, because your post is educational for the general public.

      Or maybe you got it (then, sorry!) and decided to supply the explanations the lazy OP (me!) didn't provide (if so, congrats & thanks).

    24. Re:wtf by Anonymous Coward · · Score: 0

      A write only backup system? Sounds pretty dumb. How the hell do you restore anything.

    25. Re: wtf by Anonymous Coward · · Score: 0

      I don't know much about the CA process, but you'd think "catch-up" synchronization a la blockchains would be baked in. Even if it's not, this is a trust based system; it requires cooperation between participants to be successful, and in cases like this, the answer from other participants should be "no problem," after which they can work on root cause.

    26. Re:wtf by Trax3001BBS · · Score: 1

      ... rolled back to an old backup. As a result, we lost audit data for about 147 roots.

      How the fuck are there that many changes for root CA's withing the period of one backup?

      Edge is involved, Win10 is a different beast, if one has the proper certs they can bypass the windows firewall. Every since Windows supplied a firewall with their OS that's been the way it has worked.

    27. Re:wtf by Anonymous Coward · · Score: 0

      Not to get to 'paranoid' , but something as damning and damaging as this, I find a little too suspicious. This gives several Government agencies cart blanche access to introduce rather glaring holes into the trusted website infrastructure.

      And agreed. No backups, or historical archive retention? This raises an eyebrow...

    28. Re:wtf by JustAnotherOldGuy · · Score: 1

      Seriously. No backup?

      "We're Microsoft, trust us with your data, hurr durr."

      --
      Just cruising through this digital world at 33 1/3 rpm...
    29. Re:wtf by JustAnotherOldGuy · · Score: 1

      "Our CRM system suffered a data loss, and it looks like we were too fucking stupid to have a recent backup."

      --
      Just cruising through this digital world at 33 1/3 rpm...
    30. Re:wtf by Anonymous Coward · · Score: 0

      No, you hire new first and second level managers by promoting from within - people who saw the fuck up happen, and saw what happens when serious fuck ups are allowed to happen. They will not repeat the mistake.

    31. Re:wtf by Sarten-X · · Score: 3, Insightful

      There are fallbacks, backups, and disaster recovery mechanisms. They are three different things, with three different purposes, and managers love to confuse them.

      --
      You do not have a moral or legal right to do absolutely anything you want.
    32. Re:wtf by Sarten-X · · Score: 1

      Or the first and second-level managers were the ones who laid out an effective plan, and their subordinates (whom you'd promote) didn't bother to implement it correctly.

      Perhaps it'd be better to investigate the whole situation first, rather than jump to any knee-jerk response.

      --
      You do not have a moral or legal right to do absolutely anything you want.
    33. Re:wtf by Anonymous Coward · · Score: 0

      You don't backup a database engine by opening its data files from outside, you backup by connecting to the database engine and requesting a live dump of the complete database structure and content. This way respects locks and transactions and doesn't race against simultaneous changes, nor fail entirely to access the data because of a sharing violation.

    34. Re:wtf by Anonymous Coward · · Score: 0

      You really need to freshen up on how PKI works. Anyone anywhere can create a root CA just like anyone anywhere can create a domain. The only thing differentiating public versus private is semantics. For them to work they must be publicly accessible. Hell, it is easy to spin up a VM and make a dozen root CAs. The hard work part of it is integrating into the web of trust, verifying your identity to other entities so people that do not know you may trust the intermediary's claim that you are who you say you are.

      Really, the words you have chosen to use like "bozo" and "probably" indicate you are the sort of person that thinks everything is a hierarchy of power, that a government controls the internet, and nothing can be done without the leave of your betters. Spin up a server and make a root CA. Do it. Give it a public IP. So long as you don't need to generate new keys or have a fast system the whole thing can be done in a lunch break. Tinker, learn how things actually work instead of relying on gut feelings of "probably" and realize any "bozo" can be an authority over their own domain (in this case a certificate authority domain, not an internet domain).

    35. Re: wtf by WarJolt · · Score: 2

      Manually.

      Backups should never be read by the server to ensure it has no dependency on the data.

      Backup should never be overwritten by the server to protect the backup.

      Backups should be independent verified for completeness because servers and engineers do unexpected things.

      I just made that up, but it sounds about right.

    36. Re: wtf by xlsior · · Score: 1

      Windows built-in volume shadow system let's you back up open/locked files just fine, and has for many years

    37. Re:wtf by macs4all · · Score: 1

      Seriously. No backup?

      I know. And this is the company that has been one of the most aggressive about pushing their products into "the Cloud".

    38. Re:wtf by macs4all · · Score: 2

      Actually, what seems to have happened is that they _did_ have a backup. But, they had to roll back to an old one.

      Sounds like the excuse I'd give if I was worrying about keeping my job.

    39. Re:wtf by macs4all · · Score: 1

      ... rolled back to an old backup. As a result, we lost audit data for about 147 roots.

      How the fuck are there that many changes for root CA's withing the period of one backup?

      Because they only backed up the system once, and then never actually started the backups running on their regular schedule, I'll bet.

    40. Re:wtf by macs4all · · Score: 1

      The files may have been open so they weren't able to back them up so it wasn't within the period of one backup. DOS/Windows isn't like UNIX. You can't work with open files. That's why Windows has to crash completely for even minor updates.

      I am not a real Windows Admin, but that just isn't true. Modern backups of Windows servers take advantage of a snapshot-ting capability (I think it's called VSS) so that all files can be backed-up. I have no idea how it actually works, but I know that it does.

    41. Re: wtf by lucm · · Score: 1, Insightful

      Microsoft like many other tech companies has lots of problems with middle management. Good managers get promoted quickly to more senior roles because there's constant growth and new projects; this means that what's left in middle management ris mediocre lifers or total noobs who haven't shown their potential yet. It's a dead layer with zero potential for improvement unless the company goes stale like IBM. Promoting insiders to middle management doesn't fill the void, it accelerates the spiral.

      This is one of the drivers of the flat structure that some startups are embracing with varying degrees of success. Not sure if it could help Microsoft; we'll have to wait and see how it worked at Zappos.

      --
      lucm, indeed.
    42. Re:wtf by davester666 · · Score: 1

      All these technical terms confused Microsoft management, and it all cost more money, so they checked the 'no' box.

      --
      Sleep your way to a whiter smile...date a dentist!
    43. Re:wtf by johncandale · · Score: 1

      Nah, the manager should always be the one accountable. Upper bosses have no time for "well I told them to do it and they didn't" No, it's your department, your problem. The point of a manger is to have one person accountable, otherwise you are just a team leader or a supervisor.

    44. Re: wtf by arglebargle_xiv · · Score: 2

      This bit doesn't sound right:

      Backups should never be read by the server to ensure it has no dependency on the data.

      If you never read your backups, how do you verify that the data was successfully backed up? I've seen dual-backup systems fail because, after several years of apparent backups, when the data was needed it turned out that nothing (copy #1) and the wrong data (copy #2) had been backed up.

    45. Re:wtf by Anonymous Coward · · Score: 0

      The autism is strong in this one.

  2. What a joke by Anonymous Coward · · Score: 1

    I wonder if these are the same people making gui design decisions for windows 10.. I bet the same department head signs both teams' checks.

    1. Re:What a joke by Etherwalk · · Score: 2

      I wonder if these are the same people making gui design decisions for windows 10.. I bet the same department head signs both teams' checks.

      They have 118,000 employees. Blaming them all is like blaming the army when you don't get your social security check.

    2. Re:What a joke by Anonymous Coward · · Score: 0

      I didn't blame all of them.. I linked two separate groups in a half serious way to draw a half serious conclusion.

    3. Re:What a joke by Anonymous Coward · · Score: 0

      You did, only let us rephrase the quote you found offense with:

      I wonder if these are the same people behind a snafu in the army... I bet they are the same group behind me not getting my social security check.

      Is that better?

    4. Re:What a joke by Anonymous Coward · · Score: 0

      And you're being taken half seriously, so STFU. Stop playing like a goddamned child and contribute something. This isn't fucking reddit.

    5. Re:What a joke by Anonymous Coward · · Score: 0

      But how does this work then with 118,000 employees? When something goes wrong is this some kind of 'the chain is as strong as it weakest link' or is it more '118,000 fingers pointing in another direction while saying it wasn't me'?

      I never understood these big companies under one banner producing only a limited amount of products (in the case of Microsoft only software and some hardware).

  3. Looking Back by SuperKendall · · Score: 1

    I'd hate to be in the Retrospective meeting for THAT iteration.

    You're supposed to deliver a releasable product, not release all your products (obscure Objective-C reference counting joke).

    --
    "There is more worth loving than we have strength to love." - Brian Jay Stanley
  4. chrome by bugs2squash · · Score: 2

    can't they just download chrome or firefox and get the equivalent list.

    --
    Nullius in verba
    1. Re:chrome by Anonymous Coward · · Score: 0

      Almost every computer in the world has this list of trusted certificates. (including every computer at Microsoft) It could be the most backed up piece of information ever.

    2. Re:chrome by Anonymous Coward · · Score: 2, Informative

      They aren't missing the certificate data, but rather the audit data associated with those certificates which is NOT stored with the certificates that are on computers everywhere. The summary is a bit misleading on that point.

    3. Re:chrome by Anonymous Coward · · Score: 0

      If the audit data is not stored inside the certificates, how are we (i.e. the users) supposed to be able to verify them?

    4. Re:chrome by Anonymous Coward · · Score: 0

      By the browser contacting the designated certificate revocation server...
      Unfortunately, the server has had a bad restore...

    5. Re:chrome by Anonymous Coward · · Score: 0

      You're not supposed to verify audit data. It's probably worthless anyway.

      Audit data says things like:

      "We are VERY RELIABLE AUDITING COMPANY. We went to CERT AUTHORITY's building at SOME ADDRESS and we saw they totally had an HSM [Hardware Security Module, a physical device which uses private keys to sign things but without revealing the private key] and they demonstrated that it can sign a certificate. And then they showed us a 400 page manual that says how their employees are supposed to issue certificates. The manual was very, very, very boring. Then we had a really nice meal in a Michelin star restaurant and I got completely hammered on their dime. Nice. So therefore this audit is passed."

  5. funniest thing by Anonymous Coward · · Score: 0

    funniest thing i've read all day

    1. Re:funniest thing by greenfruitsalad · · Score: 2

      stories such as this make me smirk but also check if my backups are working properly. they are. back to smirking.

      but seriously, how often do people normally back up? my /home directory is on a NAS with ZFS and keeps 24 hourly snapshots, 7 daily snapshots, 4 weekly snapshots and 6 monthly ones. this gets automatically synced to my secondary (backup) NAS and once a week i manually sync it to a nas at my parents' house. i lost all my data in the late 90s and never want to go through that experience again.

    2. Re:funniest thing by Ol+Olsoc · · Score: 1

      but seriously, how often do people normally back up? my /home directory is on a NAS with ZFS and keeps 24 hourly snapshots, 7 daily snapshots, 4 weekly snapshots and 6 monthly ones. this gets automatically synced to my secondary (backup) NAS and once a week i manually sync it to a nas at my parents' house. i lost all my data in the late 90s and never want to go through that experience again.

      Mine is very similar. I can roll back quite a way, and it has come in very helpful.

      But the answer to your question is: Most regular people simply don't back up at all. And professional setups aren't always a whole lot better.

      --
      The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
    3. Re:funniest thing by Anonymous Coward · · Score: 0

      For most end-users / home-users, daily backups (Duplicity is a good choice here due to how it stores files) are fine.

      I have a few servers running hourly backups using borg-backup (does variable sized chunk deduplication, plus compression, plus it doesn't create hundreds of thousands of files on the target). Because of how efficient borg-backup is, I can easily have hundreds of days worth of hourly backups (or I can prune them off as needed) in a single repository.

      Tried rdiff-backup in the past, but it's one of those backup systems that creates hundreds of thousands of files in the backup directory, which makes it really slow (about 30x-60x slower then borg-backup) when you ask it to backup lots and lots of files.

      No matter software what I use, a copy of the backup directory gets rsync'd off to external USB drives daily and taken offsite about once per week. It's a small shop, so this is enough. All servers write to two different backup servers, each located in different parts of the country. Those USB drives are automatically mounted on demand by autofs, and dismount after 5-15 minutes of inactivity. They're fully encrypted using a LUKS keyfile, to prevent against data leakage due to someone losing the backup drive on the way to/from the offsite backup storage location.

      It's all pretty foolproof, with the automatic unlocking and mounting of the USB file system as needed. Nobody needs to remember to change tapes daily (they just have to swap drives weekly), we have hundreds of days worth of backup history on the drive making it easy to go back to any point in time, and if one of the drives fails we can restore using one of the other 7 drives (there are 8 USB drives).

  6. Oh by Anonymous Coward · · Score: 0

    That Excel 2.0 sheet on that dusty beige 486 in the corner got saved over with a blank one, huh?

  7. NSA link? by Anonymous Coward · · Score: 0

    Now that some new NSA infrastructure is in place they need to re-capture all this audit data.

  8. Re: Melinda gates... by Anonymous Coward · · Score: 0

    u don't understand. I don't hate u, just ur childrenz. I'd be great if they died

  9. Exaggerated? by Anonymous Coward · · Score: 0

    Currently a lot of certs are broken in Edge and IE.

    I didn't see that mentioned in the mailing list, is that just something Softpedia (the author) assumed? If so, I guess it's not that bad ("just" archived audit logs gone missing from their CRM).

    1. Re:Exaggerated? by Gerv · · Score: 2, Informative

      It a load of rubbish from the original author. There's no reason whatsoever that loss of this data would cause problems in IE or Edge. Removing roots from MS's program doesn't happen without human input.

  10. Re: Melinda gates... by Anonymous Coward · · Score: 0

    Her husband screamed at my girls.

  11. Re:Melinda gates... by Anonymous Coward · · Score: 0

    There was a government-granted amnesty before they ordered you do destroy your business and family, so you shouldn't follow your fascist government's demands.

  12. Re: Melinda gates... by Anonymous Coward · · Score: 0

    Screaming iz de wayz of her conservative kindz.

  13. Re: Melinda gates... by Anonymous Coward · · Score: 0

    And how is this Microsoft's problem? We don't give a fuck about your whore of a daughter.

  14. Time to double check my own backups by Mostly+a+lurker · · Score: 1

    If Microsoft can perpetrate something like this, I think I had better set aside some time to verify that I do not have omissions in my own backup and disaster recovery procedures.I cannot imagine having to report something like this to top management.

  15. ACRONYMS... ATTENTION WHIPLASH, BEAUHD by Anonymous Coward · · Score: 0

    I understand that people reading this site probably know what SSL is. However, it's not a good assumption that people know what CRM means. Please define the acronyms in the summaries so those of us who aren't experts in a particular topic can follow along. The summary does a good job of defining what CA means. But they should give the full version of the acronyms or explain what they are for CRM and SSL/TLS.

    1. Re:ACRONYMS... ATTENTION WHIPLASH, BEAUHD by Ol+Olsoc · · Score: 1

      Please define the acronyms in the summaries so those of us who aren't experts in a particular topic can follow along.

      This should be at a +5. My directors always stopped presenters at dry runs every time they made a Alphabet soup statement. All it takes is giving the letters, then what they stand for, and after that people follow it just fine. And in a multi- skillset place like /., its pretty helpful.

      And be careful calling anything an acronym around here, the pedants will jump on you like crocodiles on a wildebeest. Then we'll have 50 posts on what an acronym is or isn't.

      --
      The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
    2. Re:ACRONYMS... ATTENTION WHIPLASH, BEAUHD by KGIII · · Score: 1

      Double click on the word - this will highlight it. Right click on the word and select search, this will open search in a new browser tab. You can even set up a variety of search engines as you go, they'll make it much easier for you.

      --
      "So long and thanks for all the fish."
    3. Re:ACRONYMS... ATTENTION WHIPLASH, BEAUHD by Anonymous Coward · · Score: 0

      IT'S AN ACRONYM IF YOU PRONOUNCE IT AS A WORD, LIKE NASA

      IT'S AN INITIALISM IF YOU READ THE LETTERS, LIKE CIA OR FBI

      BLARGARGLARGLARGLARGLARGLARGLARGLARGLARGL!!!!!

      Filter error: Don't use so many caps. It's like YELLING.
      Filter error: Don't use so many caps. It's like YELLING.
      Filter error: Don't use so many caps. It's like YELLING.

    4. Re:ACRONYMS... ATTENTION WHIPLASH, BEAUHD by Ol+Olsoc · · Score: 1

      IT'S AN ACRONYM IF YOU PRONOUNCE IT AS A WORD, LIKE NASA

      IT'S AN INITIALISM IF YOU READ THE LETTERS, LIKE CIA OR FBI

      BLARGARGLARGLARGLARGLARGLARGLARGLARGLARGL!!!!!

      Filter error: Don't use so many caps. It's like YELLING. Filter error: Don't use so many caps. It's like YELLING. Filter error: Don't use so many caps. It's like YELLING.

      Ya gotta stop after the third espresso!

      --
      The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
  16. ATTENTION WHIPLASH by Anonymous Coward · · Score: 0

    Oh, look, it's the "Republicans hate us and want us to die" jackass, now posting the same stupid shit about Melinda Gates. It's precisely the same posting style, unmistakable by its single line of fucking inane, barely coherent, poorly spelled comments pretending to have a discussion with itself about how someone (e.g., Republicans, Melinda Gates, Facebook) supposedly wants them to die. Please ban this poster, who has crapflooded many articles with this bullshit over the past several weeks.

    And to the AC posting this shit, I echo another AC who told you this is irrelevant and that nobody gives a fuck about your whore of a daughter. Fuck off.

  17. If you don't mind my asking... by westlake · · Score: 1

    How many root certificates does Microsoft hold and how long did it take to recover the 147 that were lost? Tech news posted to Slashdot tends to be a little skeletal and runs on the principle of "better late than never."

    Microsoft says that it lost audit data for 147 root certificates, which resulted in many SSL/TLS certificates showing errors inside the company's products.

    I am curious as well about how often these certificates change. How old a backup is too old?

    1. Re:If you don't mind my asking... by Anonymous Coward · · Score: 0

      These are public root CAs. So you can look in your web browser or in the OS Management tools and see, but it'll be hundreds but not thousands.

      They haven't lost the CAs, nor the CA certificates. Those are in every copy of Windows, and (mostly the same ones) Chrome, Firefox, Android, OS X, etc.

      What they've lost is _audit data_. The certificate authority is required to be audited by a third party. Then they send the audit reports to browser vendors like Microsoft and Mozilla. The reports are very tedious, and largely worthless to you as an end user. In theory the auditors might uncover something bad, and then Microsoft would remove the CA cert from Windows. But since auditors are paid to do this work by the company they're auditing, and aren't idiots, they are very unlikely to find anything really bad.

    2. Re:If you don't mind my asking... by Anonymous Coward · · Score: 1

      Due to a weird design decision, Windows management tools only show currently cached root certificates, not the full list (currently 343).
      More info at

        http://hexatomium.github.io/2015/08/29/why-is-windows/
        http://trax.x10.mx/apps.html

    3. Re:If you don't mind my asking... by subk · · Score: 1

      How many root certificates does Microsoft hold and how long did it take to recover the 147 that were lost? Tech news posted to Slashdot tends to be a little skeletal and runs on the principle of "better late than never."

      343 total, and they're required to be audited annually. It doesn't take a mathematician to see how old thier tarball was!

      --
      Now, if you'll excuse me, I have backups to corrupt.
  18. Double WTF by Anonymous Coward · · Score: 0

    Also SPOF for everyone dependent on this particular CA certificate store.

  19. Let the 1-member-short SCROTUM take this case by Anonymous Coward · · Score: 0

    and settle it once and for all. Unless it is a tie. One member short, after all.

    That's what she said. Literally.

  20. How long.. by subk · · Score: 1

    ..Before we find out they were running SSLv2 and got DROWN'ed?

    --
    Now, if you'll excuse me, I have backups to corrupt.
  21. Passive voice to the rescue by DNS-and-BIND · · Score: 3, Funny

    "there was an error on the server" "Our CRM system suffered a data loss" way to state the fact that a major company like Microsoft can't even run their own systems correctly. Well where are the fucking backups? Whoopsy-doodle! Looks like Microsoft is about as competent as a 15-man company at backing up critical data.

    --
    Shutting down free speech with violence isn't fighting fascism. It IS fascism!
    1. Re:Passive voice to the rescue by Anonymous Coward · · Score: 0

      From the request:

      Our CRM system suffered a data loss, and it looks like it rolled back to an old backup. As a result, we lost audit data for about 147 roots.

      147 have changed since the last backup. At the top of the hierarchy the RPO is greater than zero.

      As much enjoyment about bashing Microsoft some people get, this is a near non-event to professionals.

    2. Re:Passive voice to the rescue by Anonymous Coward · · Score: 0

      RTFA, They restored from backup successfully, 147 changes had occurred since that backup though.

    3. Re:Passive voice to the rescue by Anonymous Coward · · Score: 0

      That and the fact that Microsoft doesn't seem to understand how to use a backup server...

    4. Re:Passive voice to the rescue by Anonymous Coward · · Score: 0

      A backup isn't how you handle this. You obviously have no professional IT skills or background. Business based changes are recorded at the most fundamental level. It's only been this way since, ohh, the 1960s where mainframes used trace-files, since renamed to journalling; where pre and post changes are logged and can be replayed at will over a given backup. Fucking hell, do you not even know that?!

      Microsoft have been shown to be cowboys once again, ignoring half a century of the most basic of system designs. 147 change per atto-second is still handled by the most basic of logging methods. They have no excuse. Whoever designed this system needs firing instantly, as do their management.

    5. Re:Passive voice to the rescue by Anonymous Coward · · Score: 0

      Just wondering... when has it been shown that Microsoft _aren't_ cowboys? Still, its all down to the people. Its the same in my company... there are many competent people, but there is 1 bloke who just screws everything it. Whatever he touches goes to hell. He'll remove and disable things on a whim, and make up his on procedures on the spot instead of following those laid out before him. Crazed!

    6. Re:Passive voice to the rescue by Anonymous Coward · · Score: 0

      The article mentions a much older backup.

      Which is funny, because I have to say it's easy to setup daily backups and feed those into deduplication + compression on Windows Server.

    7. Re:Passive voice to the rescue by Anonymous Coward · · Score: 0

      I'm the makeshift IT support for my ~15-person company. It's only ~5% of my job; the other 95% is software development. I know I'm not the most knowledgeable at IT, but we at least have multiple rotating offsite backups of all our user data, email, etc. I'm offended that you compared people like me to those jokers over at Microsoft!

    8. Re:Passive voice to the rescue by Anonymous Coward · · Score: 0

      You literally only copied the part of the sentence that doesn't mention the backup. Are you retarded or was being a misleading asshole your intention?

    9. Re:Passive voice to the rescue by Anonymous Coward · · Score: 0

      Do you also regularly validate those backups?

  22. So... by Anonymous Coward · · Score: 0

    ...all Microsoft customers have to tell Microsoft what licenses they have ?

    1. Re:So... by daniel23 · · Score: 1

      automated since win10

      --
      605413? Yes, it's a prime.
  23. Re: Melinda gates... by Anonymous Coward · · Score: 0

    It was her, and not her husband, that got asked to leave LA Fitness in Bellevue this week.

  24. The realy SNAFU ist another one. by aix+tom · · Score: 1

    A system crashing and having to restore from an "older" backup is something that could happen to almost anybody.

    The one thing that got me in the article:

    "As many of you may have just noticed, our system just generated a bunch of emails informing many of you that you are subject to removal because Microsoft does not have evidence of a qualifying audit on file,"

    And that they then asked them to re-send the data....

    1) If I restore from an older backup, and know I may have (for example) lost payment data, I don't activate batch-jobs that generate demand notes to customers that possibly have already paid, and I just lost the data.

    2) Any "important" incoming data, (like for example payment data or SSL Audit data) should be backed once right when it enters the company, so that in the event of your system crashing (or your import-jobs wreaking havoc and losing it) you can re-populate it from that incoming data without having to ask your customers to supply the data again.

    So the problem is not really the crashed system, it is the general data flow.

    1. Re:The realy SNAFU ist another one. by Anonymous Coward · · Score: 0

      A system crashing and having to restore from an "older" backup is something that could happen to almost anybody.

      No, not if the system handles something really important (and/or highly visible like this). A system will occationally break, so you use sufficient redundancy. RAID avoids loss from disk breakage. Backups avoid loss from destruction of complete systems (fire) or griveous admin mistakes. (delete wrong database...) Logging transactions on another server makes sure you don't loose what happened between the last backup and the disaster.

      If you are "big" and also "sane", you take precautions. On the other hand, a company who cranks out an os that still needs third-party antivirus bandaids instead of being invulnerable . . .

    2. Re:The realy SNAFU ist another one. by aix+tom · · Score: 1

      No, not if the system handles something really important (and/or highly visible like this). A system will occationally break, so you use sufficient redundancy. RAID avoids loss from disk breakage. Backups avoid loss from destruction of complete systems (fire) or griveous admin mistakes. (delete wrong database...) Logging transactions on another server makes sure you don't loose what happened between the last backup and the disaster.

      I do all that. But in the event that a plane crashes right between our two server rooms which are ~500 metres apart (thus loosing all the RAID and Online-replication backups) I might still have to go back to an off-site backup, where the transaction log replication happens only every 10 minutes, so the backup might be "10 minutes old" in that case.

      Which would prompt me to start up the system (that is, after I somehow got hold of new hardware, and if me and my co-workers didn't go up in the same ball of fire that the server rooms did, which would make it "someone else's problem") , and "have a look what the state of the system is" before activating any sort of batch-jobs.

  25. Stop using Microsoft software by Anonymous Coward · · Score: 0

    They should stop using Microsoft software. I know, they're Microsoft, but that means they should know best to avoid it :)

  26. Distributed to end users? by Anonymous Coward · · Score: 0

    "Currently a lot of certs are broken in Edge and IE. Microsoft says that it lost audit data for 147 root certificates, which resulted in many SSL/TLS certificates showing errors inside the company's products."

    WTF does this mean? Did they push an incomplete CA list as an update? And Chrome uses the system root CA database, is that broken too?

  27. Comedy of errors by QuietLagoon · · Score: 1

    Redmond appears to be morphing into a comedy of errors in the tech world.

    1. Re:Comedy of errors by Anonymous Coward · · Score: 0

      Definitely a comedy of errors from the folks the cram^h^h^h^h brought you W10.

      The good news is you no longer have to wonder about their competency.

    2. Re:Comedy of errors by Anonymous Coward · · Score: 0

      Microsoft has always been a joke in the tech world.

      Nobody else has ever had so many failures of design, implementation, or delivery.

  28. Can't even run their own secure severs. by Anonymous Coward · · Score: 0

    You trust them that they're not really spying on you with windows 10.

    1. Re:Can't even run their own secure severs. by Anonymous Coward · · Score: 0

      Oh well, even if they are "spying", they will probably lose all those data, too. Don't worry; be happy!

  29. Credit where it is due by Anonymous Coward · · Score: 0

    You have to grant it to Microsoft - they do know how to look stupid and ridiculous.

  30. Re:Melinda gates... by HiThere · · Score: 1

    Everybody seems to know what you're talking about, but I've got no idea. Was is spam e-mail or what? (Or was it actually a Bellevue exercise studio? The first page of a Google search didn't list that, and I'd think it would.)

    --

    I think we've pushed this "anyone can grow up to be president" thing too far.
  31. exaggerated FUD by art123 · · Score: 1

    Where is the evidence of any SSL/TLS certificates showing errors? Seems like total conjecture based on poor reading of this audit data request made by Microsoft.

    This is AUDIT data, not the actual cert info. Read the details of the audit requirements here: http://social.technet.microsoft.com/wiki/contents/articles/31635.microsoft-trusted-root-certificate-program-audit-requirements.aspx

    This just means that Microsoft lost the documentation showing that the Certificate Authorities had performed their annual audit. Under normal circumstances, this might mean that those certs would be invalidated but seeing as how this was just a bookkeeping problem on Microsoft's end, they obviously won't invalidate anything.

    This is an embarrassment for Microsoft but nothing else.