Slashdot Mirror


German Health Insurance Card CA Loses Secret Key

Christiane writes "The SSL Root CA responsible for issuing the German digital health insurance card lost its secret private key during a test enrollment. After their Hardware Security Module (HSM) dutifully deleted its crypto keys during a power outage, it was all 'Oops, why is there no backup?' All issued cards must be replaced: 'Gematik spokesman Daniel Poeschkens poured scorn on the statement that Gematik had insisted on the service provider carrying out a test without backing up the root CA private keys. "We did not decide against a back-up service. The fact of the matter is that the service provider took over the running of the test system, so it also has to warrant its continuous operation. How it fulfills this obligation is its own responsibility."'"

40 of 174 comments (clear)

  1. Oh c'mon, be fair! by Opportunist · · Score: 4, Funny

    Not even a month ago you chided them because there were too many copies (some of them even offsite, they just didn't know who had them now), now you chew them out for having too few. Make up your effing mind!

    --
    We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
    1. Re:Oh c'mon, be fair! by MindKata · · Score: 3, Funny

      "too many copies" ... "having too few"

      This kind of organisation usually has a backup somewhere, they just have to find it. Its usually backed up on a post-it note somewhere. Maybe they should ask all of us to look for it, on the sides of our monitors.

      --
      There are 10 kinds of people in the world... those who understand binary and those who don't.
    2. Re:Oh c'mon, be fair! by Vu1turEMaN · · Score: 5, Funny

      My Day 1:

      I actually found the administrator password on a post-it note on the back of the server's CRT monitor while cleaning the server room.

      "Fucking amazing" I said out loud, and as I pulled it off, on the back was the AmEx credit card number, expiration date, and 3digit pin for our organization to order IT stuff.

      Then I noticed on the left underside of the CRT there was another post-it that said Ctrl Shift Alt Num+....so I pressed that and up came a hidden menu of hidden apps running (SysTrayX + a sketchy prog to hide services in TaskManager), 90% of them illegal. Also uTorrent was running, seeding about 50 anime series buried deep within the network and using about half of the T3 connection's throughput.

      And to top it all off, I deduced that the server had never had a fresh install of Windows. It used to have NT Server, then they used software to upgrade it to 2000 Server, and software again to upgrade it to Server 2003. ......

      Day 7:

      I get a call from the old IT guy asking me whats wrong with the connection, and I told him I reinstalled Server 2003, deleted his anime cache, changed the WPA-PSK keys from 1111111111 to something way more secure, reported the AmEx card as stolen to get a new one, changed the admin password and set password age limits on all accounts, and replaced the rootkit infected SCSI drives with new ones that would last longer. Also, I told the managers that his 5000$ quote for network-wide unlimited antivirus was utter bullshit and that he only got a cracked key for Norton 2003 and installed it only on the server, and prolly pocketed the money.

      Damn dude was like "BUT I DIDNT BACK UP THE ANIME TO DVD YET!!!". Now I love anime as much as the next person, but I think he has other stuff to worry about at this point.

      But you know what got me the most mad and prompted all of this? The server was named Odie, and the computers were all garfield characters.

      CALVIN AND HOBBES FTW!!!!

    3. Re:Oh c'mon, be fair! by Vu1turEMaN · · Score: 3, Funny

      Oh, and his DAT72 backups had been failing for the last 2 years and he had never checked the logs.

      Good thing he left to start his own business! /shudder

    4. Re:Oh c'mon, be fair! by Hurricane78 · · Score: 2, Funny

      Day 8:
      You got fired, and the system got "restored" because your "fixes" halted the whole "business".
      It was a sad day.

      --
      Any sufficiently advanced intelligence is indistinguishable from stupidity.
    5. Re:Oh c'mon, be fair! by Vu1turEMaN · · Score: 4, Funny

      Heh...I'm actually just doing a paid internship at a non-profit after their full-time guy left. It was supposed to end on May 1st, but hey I guess they love what I've done.

      Got them a cheap dedicated backup system, updated all the systems and reinstalled an NLite-ed XP on every computer, and moved them from Exchange to Google. Oh, and the lab computers run Ubuntu.

      They also loved it when I found the IT guy's secret paypal business account with 3000$ sitting in it that was supposed to be used for something else (battery backup replacement batteries). Putting passwords in a file on the administrator's desktop called "passwords for everything.txt" is sooooo helpful for when you're trying to be sneaky.

      Seriously, this shit is a soap opera of IT-isms.

    6. Re:Oh c'mon, be fair! by v1 · · Score: 2, Insightful

      Its usually backed up on a post-it note somewhere.

      For a root CA private key, better be a big post-it note

      (or written in really small letters)

      --
      I work for the Department of Redundancy Department.
  2. Wrong Title, Wrong summary by freedom_india · · Score: 3, Informative

    Once again, misleading title to a different summary.
    For fuck's sake, the Germans didn't lose the key.
    The SSL Root CA lost that.
    Get the facts right.
    For a second i was wondering how Germans could that stupid. That is unlike the Germany i know. And exactly as i suspected, the German insurer had been insisting the root CA for backup while the CA thought it was unnecessary.
    Is it the German company's fault?

     

    --
    "Doing what i can, with what i have." ~ Burt Gummer
    1. Re:Wrong Title, Wrong summary by Opportunist · · Score: 5, Funny

      After all, we all know Germans are exact and punctual, Poles are thieves, Russians are drunk and Fins are even more so. Oh, and Mexicans are lazy and US people are simple minded. Any stereotype missing?

      --
      We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
    2. Re:Wrong Title, Wrong summary by Sockatume · · Score: 2, Insightful

      The summary even states that Gematik insisted on a back-up less operation, and then provides a quote explicitly stating that they did no such thing! Slashdot: doing for editorial accuracy what Fox does for editorial neutrality.

      --
      No kidding!!! What do you say at this point?
    3. Re:Wrong Title, Wrong summary by MancunianMaskMan · · Score: 4, Informative

      Any stereotype missing?

      yes.

      we British are all of the above.

    4. Re:Wrong Title, Wrong summary by Opportunist · · Score: 5, Funny

      Not only that, they have really weird tastes, too. In food and bed. Sometimes at the same time.

      --
      We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
    5. Re:Wrong Title, Wrong summary by multisync · · Score: 2, Informative

      The summary even states that Gematik insisted on a back-up less operation, and then provides a quote explicitly stating that they did no such thing!

      Gematik commissioned D-Trust to provide the root CA as a service. D-Trust managing director Matthias Merx stated "Gematik decided to 'do without a back-up'. As a service provider, we have to accept that."

      From the article and summary:

      "Gematik spokesman Daniel Poeschkens poured scorn on the statement that Gematik had insisted on the service provider carrying out a test without backing up the root CA private keys. "We did not decide against a back-up service ..."

      Slashdot: doing for editorial accuracy what Fox does for editorial neutrality.

      Indeed. Two sides claiming different things. Must be Slashdot's fault.

      --
      I don't care why you're posting AC
    6. Re:Wrong Title, Wrong summary by Bemopolis · · Score: 2, Insightful

      After all, we all know Germans are exact and punctual

      Well, we DO know that they are awfully good at writing numbers down. Sometimes even up the arm.

      --
      "I guess the moral of the story is, don't paint your airship with rocket fuel." -- Addison Bain
    7. Re:Wrong Title, Wrong summary by Bigjeff5 · · Score: 2, Informative

      The title/summary are not necessarilly incorrect, just ambiguous. English can do that, and if you aren't paying attention your meaning can be taken in a way other than you intended.

      In this case, there are a few ways to read "German Health Insurance Card CA":

      1.) The Health Insurance Card CA of German origin
      2.) The CA for the German Health Insurance Card
      3.) The Card CA for German Health Insurance
      4.) The Insurance Card CA for German Health

      Obviously they aren't saying 3 or 4, those work gramatically but don't make a lot of sense in the context of health insurance and certificate authorities. 1 and 2 though, work pretty well either way. They should have used the unambiguous form, obviously with a small amount of research we can see that 2 is the correct meaning, but a number of people will read the sentance to mean 1 instead, as you did.

      It's poor writing, not an attack or attempt to slight Germans. Remember the old saying: Never ascribe to malice what can be explained by incompetance.

      Lastly, while it was the CA's responsibility to ensure they have backups and the like, it is the client company's responsibility to ensure they can maintain their business. If the health insurance company never asked for or verified a disaster recovery plan, it's their ass that is in hot water if they cannot provide service.

      Make no mistake, they WILL lose business over this, even if the failure isn't directly their fault.

      --
      Security is mostly a superstition... Avoiding danger is no safer in the long run than outright exposure. - Helen Keller
    8. Re:Wrong Title, Wrong summary by JaredOfEuropa · · Score: 4, Insightful

      Even so, this line struck me as all too familiar: "The fact of the matter is that the service provider took over the running of the test system, so it also has to warrant its continuous operation. How it fulfills this obligation is its own responsibility."

      This is why managers (especially the MBA types) love outsourcing of everything. It is also in part because numbers and KPIs are so much more easy to manage than actual people. But mainly, by outsourcing a function you also get to outsource the responsibility for that particular function. If things go tits up, the worst you'll be blamed for is picking the wrong service provider, or perhaps not monitoring a particular KPI properly. Minor stuff.

      I've seen plenty of managers like that, and I have heard a variation of that one line all too often.

      --
      If construction was anything like programming, an incorrectly fitted lock would bring down the entire building...
    9. Re:Wrong Title, Wrong summary by WarlockD · · Score: 3, Interesting
      I don't know..

      "We did not decide against a back-up service ..."

      That double negative sounds awful like "At the time, we didn't know what they were asking":P I guess its just with personal experence. Evey time I hear a manager use double negatives to defend a decision, its because they didn't really know what they were deciding in the first place. Atleast in IT.

  3. Re:An HSM That Requires Continuous Power? by Opportunist · · Score: 3, Informative

    Don't blame the cards for the stupidity of their administrators.

    --
    We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
  4. NSA/CIA by Anonymous Coward · · Score: 4, Funny

    Maybe they should check with the NSA or CIA? They've got a backdoor into EVERY system, and may still have the key saved on a laptop lying around somewhere.

  5. Could be worse by bradgoodman · · Score: 4, Insightful

    I'd rather the key be lost, than stolen, hacked, made-public, etc. At least it didn't breach security in the typical manner.

    1. Re:Could be worse by Opportunist · · Score: 3, Interesting

      What's worst about it is that this is probably presumed to be worse. Had the key be stolen, they'd probably not even report it because business could continue as usual, maybe nobody finds out...

      --
      We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
    2. Re:Could be worse by Anonymous Coward · · Score: 2, Interesting

      It could be worse, but this incident exposes a design flaw: The loss of a private key should not stop them from issuing new cards which are compatible with the existing cards.

      If a CA key is lost, then there should be a layer above it which can create a new CA key. Cards are checked against the top CA public key, so the old and the new cards can both be verified. Because the top CA is only used to create intermediate CAs, its private key can be kept safer than the key of a CA which is regularly used for signing certificates. Should it get lost anyway, at least the intermediate CA still exists and can continue signing new cards.

  6. This would never happen in Britain by Curmudgeonlyoldbloke · · Score: 2, Funny

    It would easily be found be searching the nearest pub car park for USB keys, or checking the train that the relevant civil servant travelled home on.

  7. You can fall off the road on either side by starfishsystems · · Score: 3, Interesting

    There are two fundamental ways to fail as a CA. There must be exactly one party in effective possession of the private key of the root cert. If the number of parties becomes less than or more than one, fail.

    Mistakes happen, of course, and certificate infrastructures can be enormously complex. But if you're going to do any kind of risk mitigation, the absolutely most basic place to start would be with these two scenarios.

    --
    Parity: What to do when the weekend comes.
    1. Re:You can fall off the road on either side by jvkjvk · · Score: 2, Informative

      As an optimist, I'd say that least they didn't fail in the worst possible way.

      The pessimist in me thinks I should get a bit more than "not failing in the worst possible way" when I pay somebody a barrel of cash to hash a couple numbers for me.

      No, that's also the optimist in you.

      Cheers. :)

  8. Rootkeylosin! by 192939495969798999 · · Score: 2, Funny

    Q: How do you learn every German swear word in about 20 seconds?
    A: Tell the German admin that you lost the root key.

    --
    stuff |
    1. Re:Rootkeylosin! by BlackCobra43 · · Score: 2, Insightful

      The best part is it will all be contained in a single, monstrously large word. Ah, german efficience.

      --
      I never spellcheck and I freely admit it. Save your karma for more worthwhile "lol erorrs" replies
  9. Re:Public Key Infrastructure by FlyingBishop · · Score: 2, Insightful

    That's just silly. They obviously take security seriously enough that they found re-issuing all of their certs preferable to adding a second storage place for the private key, thus doubling the possibility of the system being compromised.

    If the key had been compromised, that would be a breach of trust. This is more an example of the fact that as security increases, usability decreases.

  10. I'm confused by Candid88 · · Score: 4, Insightful

    card lost its secret private key during a test enrollment

    I'm confused, isn't this sort of problem exactly why you carry out system tests?

    Sending out new cards to card testers during a systems test is hardly extraordinary.

    1. Re:I'm confused by WarlockD · · Score: 3, Informative
      See I read that part differently.

      Matthias Merx, the firm's managing director, told heise online that following a voltage drop, something happened in D-Trust's "Trustcenter" that does occasionally occur. "The HSM independently deleted the data because it suspected an attack."

      Translation? "Someone unplugged the backup power supply before setting the proper mode in the card because we didn't fully understand how sensitive the card is for root CA certs"

      Merx explained that "Gematik decided to 'do without a back-up'. As a service provider, we have to accept that,"

      Translation? "We asked Gematik that it might be a good idea to back it up and they said its fine its just for testing." or "We recommended to Gematik to back up the card before shipping it to us. They shipped it to us and we just shrugged our shoulders." Bonus points if you guessed they asked a low level manager at Gematik who thinks CA is the first two letters of a cat.

      Gematik spokesman Daniel Poeschkens poured scorn on the statement that Gematik had insisted on the service provider carrying out a test without backing up the root CA private keys. "We did not decide against a back-up service. The fact of the matter is that the service provider took over the running of the test system, so it also has to warrant its continuous operation. How it fulfils this obligation is its own responsibility."

      Traslation? "Gematik is taking NO RESPONSABLITY WHATSOEVER for doing any safty checks before giving our root ca to an outside vendor."

      All in all its not a big deal though. It looks like they just lost the issuing CA and not the revoke keys. It looks like they can still authenticate too. Now if this was the MAIN system germany with 80+ million plus medical cards? I think people are going to be shot:P

  11. Place blame by ubrgeek · · Score: 4, Funny

    Poeschkens claimed, "I know nothing! noth-thing!" and proceeded to blame the problem on a man he would only identify as "Hogan."

    --
    Bark less. Wag more.
  12. What is "CA"? by T+Murphy · · Score: 5, Funny

    For those of you who are wondering what CA is, it stands for Certificate Authority. You see, the Germans have a hard time functioning without a constant stream of praise, so they have this authority in place that prints and sends certificates to people. Every day thousands of Germans get congratualted for crossing the street, for finding their car keys or for eating their 1000th potato of the month. You know you've walked into a German household when you see the wallpaper of framed certificates.

    The problem here is that the company deleted the certificate-printing program since they thought someone was trying to hack in and print more certificates for themselves- no one is THAT special so they had to stop him. They forgot to have another program ready to print more certificates, so now Germany is under threat of entering a depression since they no longer get certificates telling them how special they are.

    On a serious note: I don't follow this article very well with all the acronyms being spelled out but not explained, and no background knowledge of anything going on here. If someone would care to explain what is going on here to someone that has never heard the term CA, you should get a +5 informative easily.

    1. Re:What is "CA"? by Ritorix · · Score: 5, Informative

      I will simplify, but basically a CA (Certificate Authority, that much of the parent wasnt a joke) is a server that creates encryption certificates. In this case, SSL certificates. For example, when you goto https://mail.google.com/ that SSL certificate was created by the Thawte SGC CA. Thawte is one of many companies that you can pay to create you an SSL cert, so your users can communicate with your server via https.

      The CA itself also has an encryption key, which is stored on hardware. In some cases its a PCIe board, others its a removable PCMCIA card, etc. This particular CA used an add-on board which lost power during operation, wiping out its only key. The board seems to have been working as intended, preventing attack (removal of board, which would cause power loss) by wiping itself.

      Without that key, the CA can no longer create revocation lists (CRLs, lists of certs a CA has created that have since been revoked or expired) or any new certs. They are dead in the water, also causing every cert they have ever made to become invalid as they can no longer be checked against a recent CRL. They have to start from scratch, recreating every_single_cert.

      This was only a test system, but if this happened in reality 80 million Germans would have invalid health cards. At least they discovered the value of a backup during testing.

    2. Re:What is "CA"? by CrashandDie · · Score: 2, Informative

      Disclaimer: I work for a company that specialises in these kind of deployments, however, they do not endorse anything I am about to say, and I am doing so as an individual. All the information in here is public knowledge.

      A few points first:
      - A CA doesn't only create encryption certificate. It can create a variety of certificates, including Windows Logon, Signing certificates, etc. It all depends on the Certificate Policies that are configured on the CA.

      - We have no information the CA was indeed issuing SSL certificates. More likely, they were encryption certificates used to decrypt the patient's medical files. According to this link the card also contains a signing certificate.

      - There are a lot of different types of HSMs. Some will encrypt their data (Security World, for nCipher) on a remote filesystem (RFS, especially in the case of netHSMs) that can be on any machine. As long as you have the Administrator Card Set (or a quorum of those, m of n cards required to perform specific tasks), you can reload the Security World, reload the keys, and are good to go. Other HSMs provide in-hardware "protection" of the keys, such as SafeNet HSMs, and the data in these can be backed up through hardware tokens (which are just very secure PCMCIA cards). HSMs usually have built-in hardware protections so you can't break it open or something, without destruction of the data.

      - It is stupid for both the service provider and the customer to have gone without backups of the Root CA. What is the point of replicating your production environment in a reference/test environment, if you're not going to do a full replication?

      - As each company looks like an idiot, they will try to blame each other, and they already do. Quite typical. The Service Provider is saying "we did what the customer wanted", and the customer says "The service provider was taking care of the tests". They are both stupid and wrong.

      - Smartcards are protected by master keys. When the smartcard manufacturer creates the cards, he initialises them, usually with a "Manufacturer Key". This key is known by anyone who ever worked in the industry. In a normal setup, when the customer (the company issuing the cards to their users) gets the cards, during the card personalisation, they swap the Master Keys using their own keys. This is probably the most important part. Without those Master Keys, nobody can access the card's applets, or perform administrative actions on the card. Not even the owner with the PIN. It is very likely that for a solution of this size, the card manufacturer (according to this link, GnD and Gemalto are part of the project. GnD will supply their 80k card) were using the customer's Master Keys initially, so that the key swapping wasn't needed (or simply, the cards wouldn't be usable anywhere else).

      If in the same accident, the Master Keys for the smartcards were lost, then they can effectively throw away all the cards that were created in that batch, as nobody will be able to access the applets on the cards, thus, nobody will be able to update the certificates, or even erase the cards. This doesn't mean the certificates are dead, the certificates can still be used on a daily basis without any issue, but considering the CA will not be able to publish its CRL (which needs to be published every x hours/days, and has an expiry threshold), the certificate chain would become untrusted after some time (probably a few years, considering the Root CA should NOT be connected, but rather locked in a safe, and never need to publish its CRL for the length of the certificate's lifetime), and only then will problems start to arise.

      I do hope for the sake of the companies involved in this project that they didn't ask the manufacturers to produce the test batches with the customer's Master Keys and that the Master Key was lost,

  13. Best practices by Shulai · · Score: 2, Informative

    Best practices about CA management says you should have your secret key in a (physical) safe. Better yet, divide it in two pieces and put it along the passphrase in three different safes (part1+pass,part2+pass,part1+part2), so you can't lose key access even if you lose one safe, and nobody can take the key by opening a single safe.

  14. Re:Public Key Infrastructure by mlts · · Score: 2, Insightful

    PGP Desktop has this option. You can share a key and split it among people, where x amount of y pieces are needed to recover the original key, where both x and y are user selectable values.

    However, if a key is a top root CA key, you would not be using it on a general purpose computer. You would have the key generated in a HSM and stored there, where someone can perhaps use the key to sign and decrypt stuff, but would have to go to a lot of trouble to get past all the hardware tamper evident stuff in the HSM to access the raw private key material.

    Most newer HSM devices I've seen have a way to back up keys generated on the device (usually to USB flash drives), provided at key generation time you set a flag allowing the key to leave the device. If this "allow private key material to leave the HSM" flag isn't explicitly set, you are screwed when it comes to backups, and your best workaround is to create another key with the flag set, then do some cross signing. Depending on task, you might be able to get away with revoking the old key, but sometimes (especially if the old key signed a lot of code certificates), this may be almost impossible.

    This lost key should be a lesson to people. Making sure the keys that are in the armored box are backed up can be just as important to security as keeping them in the armored box in the first place. Ideally, consider multiple HSM hardware at multiple locations, including an offline HSM stored in padded packaging that goes in the Iron Mountain tub, as well as the means to access the key inside the box.

  15. Re:Public Key Infrastructure by Sloppy · · Score: 2, Insightful

    Actually, I can think of a reason, after all. Since this CA no longer has the ability to revoke prior signatures made with that key, then that key can no longer be trusted as a signer. You can check to see if a CA has certified something, but there's no way to check to see if the CA changed their mind, because the CA no longer has a way to say that.

    --
    As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
  16. My advice in the past by meerling · · Score: 2, Informative

    In talking with people (or company representatives) about their security regarding passwords and keys, I always told them two things.

    First, all security experts will tell you that you should not keep copies of that stuff around.

    Second, that's not a realistic expectation, stuff happens. The IT guy goes on vacation, has an accident, or dies. (Seen all 3 numerous times.) You fire the Admin for some reason. This building burns down. Etc.

    A reasonable thing to do, is keep a password/key log with that critical information that is kept up to date at all times. You have two copies of it. Both are kept secure in good quality safes (not a $200 lockbox).
    Both safes are in different physical locations, at least separate buildings, preferably miles apart.
    The reason for this is pretty easy. Once again, things happen. I've seen buildings burnt down, flooded, inaccessible due to chemical hazards from a truck wreck, etc. You don't know what will happen, but if you have them stored at separate physical locations, you at least know you will be able to get to one of them if you need to, assuming nobody uses a nuke.

    It all falls under that old techie saying, "So, when did your data become important to you? Before or after you lost it...".

  17. Spoonerism by Curate · · Score: 2, Funny
    Gematik spokesman Daniel Poeschkens poured scorn

    I literally read that as scoured porn...

  18. Does German work like English? by russotto · · Score: 2, Funny

    "We did not decide against a back-up service. The fact of the matter is that the service provider took over the running of the test system, so it also has to warrant its continuous operation. How it fulfills this obligation is its own responsibility."' If this were originally in English, it would mean "We knew this would happen and we tried to tell them, but those arrogant SOBs thought they knew it all and didn't want to listen to us. So we shut up, pulled up a chair, got some popcorn, and waited for the fireworks". I'm not sure that translates, though...