23,000 HTTPS Certs Axed After CEO Emails Private Keys (arstechnica.com)

An anonymous reader quotes Ars Technica: A major dust-up on an Internet discussion forum is touching off troubling questions about the security of some browser-trusted HTTPS certificates when it revealed the CEO of a certificate reseller emailed a partner the sensitive private keys for 23,000 TLS certificates. The email was sent on Tuesday by the CEO of Trustico, a UK-based reseller of TLS certificates issued by the browser-trusted certificate authorities Comodo and, until recently, Symantec...

In communications earlier this month, Trustico notified DigiCert that 50,000 Symantec-issued certificates Trustico had resold should be mass revoked because of security concerns. When Jeremy Rowley, an executive vice president at DigiCert, asked for proof the certificates were compromised, the Trustico CEO emailed the private keys of 23,000 certificates, according to an account posted to a Mozilla security policy forum. The report produced a collective gasp among many security practitioners who said it demonstrated a shockingly cavalier treatment of the digital certificates that form one of the most basic foundations of website security... In a statement, Trustico officials said the keys were recovered from "cold storage," a term that typically refers to offline storage systems. "Trustico allows customers to generate a Certificate Signing Request and Private Key during the ordering process," the statement read. "These Private Keys are stored in cold storage, for the purpose of revocation."
"There's no indication the email was encrypted," reports Ars Technica, and the next day DigiCert sent emails to Trustico's 23,000+ customers warning that their certificates were being revoked, according to Bleeping Computer.

In a related development, Thursday Trustico's web site went offline, "shortly after a website security expert disclosed a critical vulnerability on Twitter that appeared to make it possible for outsiders to run malicious code on Trustico servers."

Comments

Bullet, Meet Foot

[mentil]

When Jeremy Rowley, an executive vice president at DigiCert, asked for proof the certificates were compromised, the Trustico CEO emailed the private keys of 23,000 certificates

Those certificates are DEFINITELY compromised now.

Re:Bullet, Meet Foot

[TechyImmigrant]

When Jeremy Rowley, an executive vice president at DigiCert, asked for proof the certificates were compromised, the Trustico CEO emailed the private keys of 23,000 certificates

Those certificates are DEFINITELY compromised now.

TFA seems to imply that he emailed the private keys in order to prove that they were compromised. Which seems like an appropriate thing to do.

Re: Bullet, Meet Foot

[Anonymous Coward]

As per Slashdot rules, i have not read the article.

Why the fuck did the CA have the Private Keys of these domains? You send the CSR to them, but keep the private key PRIVATE. That's the whole point!

Re:Bullet, Meet Foot

[mysidia]

... Which seems like an appropriate thing to do.

No... it is NOT appropriate for a CA or a reseller of a CA to retain customers' private keys in the first place --- it is even MORESO inappropriate for a CA to deliberately extract and use in any manner for any purpose a customer's private key from "secure cold storage" without that customer's specific authorization.

Basically this changed the situation from "There are security concerns related to this certificates", to --- This CA reseller deliberately compromised the security of their customer's certificates by appropriating their private keys during the ordering process and then later transmitting them insecurely over the internet.

Re: Bullet, Meet Foot

[dimko]

because... managed SSL services. For people with more money than sense. In a nutshell, you pay them for that service and installation of SSL certs is now their business. Potentially maintenance too.

Re:Bullet, Meet Foot

[Nkwe]

When Jeremy Rowley, an executive vice president at DigiCert, asked for proof the certificates were compromised, the Trustico CEO emailed the private keys of 23,000 certificates

Those certificates are DEFINITELY compromised now.

The first to shoot themselves in the foot would be anyone who doesn't generate their own private key when they purchase a certificate. The CA is only supposed to sign the public parts of your certificate, it is not supposed to ever have access to the private key. Letting your certificate vendor create a private key (and subsequently have access to it) is unwise and insecure.

Re:Bullet, Meet Foot

[Solandri]

No... it is NOT appropriate for a CA or a reseller of a CA to retain customers' private keys in the first place

I'm guessing that was the compromise Trustico was reporting. The private keys should've been deleted immediately after they were generated for the customer. But Trustico probably found during an audit that they hadn't been deleted and were still on their servers somewhere. Since they couldn't prove that those private keys hadn't been copied, they erred on the side of caution and declared them all compromised.

Re: Bullet, Meet Foot

[Anonymous Coward]

What do you mean, more money than sense? SSL is a racket, whether you do it "properly" or not. The security is just not there for anything worth attacking. There are dozens of things that you can do and have to do that have only become available recently (but not universally) in order to close the biggest holes in the TLS scheme. For example, any CA can issue a certificate for your domain, even one where the CEO sends thousands of private keys by email. (By the way, that was the responsible thing to do. He could have just sat on the keys, knowing that the previous owners probably had a copy.) To prevent that, there is certificate pinning and CAA RRs. But wait: CAA RRs are an honor system. And in many scenarios, the private key is handed over to a third party anyway, because who really terminates TLS on their own web servers in their own building anymore? Nobody, that's who. There's CDNs, DDoS-mitigators, load balancers, and reverse proxies, and they all need to be in the unencrypted path to do their work. So someone handed over their private keys to a CA, which they have to trust anyway not to issue additional certificates. Boo hoo.

Re:Bullet, Meet Foot

[Anonymous Coward]

Nope. They admitted that they store the keys intentionally specifically so they can revoke the certificates later without customer permission.

Re:Bullet, Meet Foot

[gweihir]

The level of stupidity expressed in this is staggering. I mean it is not only the fact that somebody with the least bit of clue would never email secret keys without protection, it also is that he could get them in the first place and do this. This means that DigiCert is completely compromised itself due to non-existing or easily bypassed security policies and should under no circumstances be trusted again.

Re:Bullet, Meet Foot

[thegarbz]

Those certificates are DEFINITELY compromised now.

Wrong analogy. Bullet meets foot implies that the action of the CEO achieved something other than what he was hoping. He wanted to revoke those certificates anyway, and when questioned whether they were compromised he compromised them.

Nail meets coffin.

Re: Bullet, Meet Foot

[Anonymous Coward]

But realistically, to do anything useful with someone's key you also need to control their dns. And if you control their dns you can recreate new certs anyways. I don't see the value of having those keys.

Re:Bullet, Meet Foot

[Shuntros]

You do not need a private key to revoke a certificate. You need the certificate serial number.

The issuer should NEVER set eyes on a private key which isn't theirs. If you want to make life easier for your customer, do it client-side with JavaScript and throw a PFX at them once the issuance has completed.

It still stuns me how many in tech, even CAs, have such a poor understanding of how PKI works.

Re: Bullet, Meet Foot

[Anonymous Coward]

Not true. You need the ability to influence the clients DNS. Access to their DNS server, even their hosts file etc.

Re:Bullet, Meet Foot

[mysidia]

You do not need a private key to revoke a certificate. You need the certificate serial number.

As a reseller, they wouldn't have any legal right to revoke the certificate without customer permission -- that should be an act of the CA.

Notice how in the article that DigiCert Demanded Proof of the compromise?

Hopefully now that this has happened: DigiCert will be more reluctant to permit this reseller in the future to have their own custom ordering process --- I would say they should suspend this reseller until they agree to a contract addendum that they will neither generate nor obtain or store private keys for customer certificates.

Re:Bullet, Meet Foot

[phantomfive]

It makes you wonder how the CEO got the keys to begin with.

Re:Bullet, Meet Foot

[Anonymous Coward]

If they weren't compromised then the CEO wouldn't even have those keys in the first place. The headline leads the reader to believe that the CEO was the one who compromised the keys, but in fact they were already compromised and the CEO sent those keys as proof that they needed to be revoked.

When a site-owner requests a certificate he makes his own private keys and keeps them private. The certificate request sent to the CA (or reseller, in this case) is made using the public key, not the private one.

Re:Bullet, Meet Foot

[Megane]

DigiCert didn't mail the private keys, Trustco's PHB did.

Re: Bullet, Meet Foot

[Anonymous Coward]

un^Wlawfull interception. They can sell the keys to Government / "security" appliance vendors for completely transparent interception. They could claim immunity if anything happened by saying they were hacked or something.

Second thing I do in any browser is UNTRUST EVERY FUCKING CA. First is install an adblock.

Re:Bullet, Meet Foot

[Anonymous Coward]

Not so much, if you'd read the summary.

Re:Bullet, Meet Foot

[gweihir]

Ah, sorry. Thanks for the correction. Then my statement applies to Trustco.

Re:Bullet, Meet Foot

[TechyImmigrant]

>No... it is NOT appropriate for a CA or a reseller of a CA to retain customers' private keys in the first place

Well that's not what I said, is it?

Emailing the keys to someone is an appropriate way to prove to them that the keys have been compromised. Now go and be contrarian somewhere else.

Re:Bullet, Meet Foot

[modmans2ndcoming]

ever here of secure file shares?

Re:Bullet, Meet Foot

[Anonymous Coward]

Yes... and yet, this is perfectly normal. In my experience 99% of company employees, especially those is management and above, have no actual understanding of security.

Re:Bullet, Meet Foot

[FormOfActionBanana]

I don't get the fuss. If they are compromised, they are compromised.

Re: Bullet, Meet Foot

[Anonymous Coward]

Its in the summary just fyi, no need to read the article

Re:Bullet, Meet Foot

[Anonymous Coward]

Emailing the keys to someone is an appropriate way to prove to them that the keys have been compromised.

The keys are compromised for sure so its not inappropriate, but a better method might have been to sign something with the keys as proof instead. Though either way the keys are just as bad off. Now why did digicert only suspend 23k keys instead of the full 50k as was initially reported? digicert is in essence saying they must send all 50k keys before they'll suspend them.

Re:Bullet, Meet Foot

[Anonymous Coward]

The first to shoot themselves in the foot would be anyone who doesn't generate their own private key when they purchase a certificate.

On the one hand that is right and the correct thing to do. On the other hand you already trust Trustico to handle your certificates, so giving them access to a specific private key might not make matters worse as long as you don't use it for anything else - they don't need your private key to pretend to be you, they can just sign their own.

Re: Bullet, Meet Foot

[TheRaven64]

This is only true if the domain uses DNSSEC. Most laptops / phones trust whatever DNS server the access point that they're connected to tells them to use via DHCP. If you set up a malicious AP then you can return your own server address for the compromised domains. If you deploy malware that infects a few thousand vulnerable WiFi routers then you can do this on a large scale quite easily.

Re:Bullet, Meet Foot

[TechyImmigrant]

Emailing the keys to someone is an appropriate way to prove to them that the keys have been compromised.

The keys are compromised for sure so its not inappropriate, but a better method might have been to sign something with the keys as proof instead. Though either way the keys are just as bad off. Now why did digicert only suspend 23k keys instead of the full 50k as was initially reported? digicert is in essence saying they must send all 50k keys before they'll suspend them.

My crystal ball isn't working today. It seems like a really bad idea to be a CA and have people's private keys on file. It seems like a bad idea to be taking people's word for it when you're in a position of issuing revocations. Not my CA. The real CA I set up (in a former job role) held onto nothing but its own keys and the cert list. Certs are needlessly complicated, but you can and should make the process as simple as possible.
 

Re:Bullet, Meet Foot

[mysidia]

The keys are compromised for sure so its not inappropriate, but a better method might have been to sign something with the keys as proof instead.

What they showed wasn't proof of the compromise the reseller had been claiming happened. What they showed was proof that the reseller had (IMPROPERLY and DELIBERATELY) retained customer private keys, WHICH IS INAPPROPRIATE --- and the reseller then literally compromised their keys (that might not have been compromised before) to "prove" it, which was ALSO highly inappropriate.

why did digicert only suspend 23k keys instead of the full 50k as was initially reported?

None of the keys are KNOWN compromised according to the reseller, Only suspected that they could have been, and ultimately it's the customer's decision how to proceed if there's merely a suspicion or belief ---- PROOF would have been obtaining compromised Keys through the flaw, not going to deliberately retained cold storage files and retrieving customer keys.

That's Nothing

[NicknameUnavailable]

Sophos has a trusted root CA embedded in their enterprise firewalls which allows the firewall to launch man-in-the-middle attacks against clients to spy on them. That means all you have to do to launch a successful man-in-the-middle attack yourself against HTTPS traffic is to gut a Sophos firewall and find the private key embedded in it.

Re:That's Nothing

[Pinky's Brain]

They use the same key pair for every firewall?

Re:That's Nothing

[Nkwe]

Sophos has a trusted root CA embedded in their enterprise firewalls which allows the firewall to launch man-in-the-middle attacks against clients to spy on them. That means all you have to do to launch a successful man-in-the-middle attack yourself against HTTPS traffic is to gut a Sophos firewall and find the private key embedded in it.

You would have to install the root cert certificate from the firewall CA into all your clients for that to work. In an enterprise if you want to sniff HTTPS traffic, you may chose to do this (since in an enterprise you control the client machines), but as soon as you chose to do this, you open up huge security holes.

Re:That's Nothing

[NicknameUnavailable]

They use a keypair that's embedded into all the major browsers as a trusted root CA - so chances are yeah.

Re:That's Nothing

[NicknameUnavailable]

Nope, it's included in all the major browsers (I only noticed it after installing a privacy-geared browser far from the mainstream which didn't include the bundled root CAs for Sophos.)

Re:That's Nothing

[BlueUnderwear]

Does anybody know which root CA this is? So that I can mark it "untrusted" locally in my Firefox.

Re: That's Nothing

[orbit500]

Thatâ(TM)s really not true though, is it? There is an appliance CA keyed for the machine and which generates an appliance cert. The CA is NOT in the default trust store of any browser. It has to be explicitly downloaded from the appliance and added to trust. No idea where you got that nonsense from!

Re: That's Nothing

[Anonymous Coward]

Enterprise proxies that do ssl mitm cause not only security problems, but also all kinds of software problems because many tools like npm won't know about the local CA by default. And it's not always easy to fix it, because some tools can use other tools (like mvn calling npm or pip) in public open source repos, and so forth. It becomes a fool's errand.

Re: That's Nothing

[Anonymous Coward]

Correct. My guess is he is in an enterprise where someone is pushing cert keys to the clients.

Re:That's Nothing

[Anonymous Coward]

Can you dig out the keypair and post the requisite info about it here? You can't use a root CA this way, so if they did it would be revoked immediately if it became public. Your local admin staff might have included additional CAs in the browser though, make sure that isn't the case.

Re:That's Nothing

[Anonymous Coward]

Have you looked at the CA's installed?? What do you think those Entrust certs are for. They aren't the only ones either, hell China is still trusted.

Mozilla, nor Google will do _anything_ about these. SSL extortion (the fees for certs) is a massive business.

Re: That's Nothing

[NicknameUnavailable]

Firefox, Chrome, Opera, IE, and Safari didn't complain about it - I switched to a more secure version of Firefox with locked down privacy features when I noticed it, turns out they don't include the certs allowing for corporations to override websites to monitor traffic going over their network.

Re:That's Nothing

[modmans2ndcoming]

it was a re-seller that did this, not the root. DigiCert is the root and they asked for proof from the re-seller that the keys were compromised before they revoked them. The re-seller CEO sent the private keys to DigiCert via standard email with no protection.

Re: That's Nothing

[Anonymous Coward]

Our sohpos IS a ca, generating certificates on the fly for https decryption. The root CA cert is pushed out via domain group policy. It is untrusted on workgroup machines that dont get the gp updates.

Re:That's Nothing

[Anonymous Coward]

BlueCoat Systems, Inc. and many other "re-encrypting proxy" appliance vendors do exactly the same thing. This is why SSL/TLS is absolutely worthless to anybody inside any Top 1,000 company.

I call BS

[Anonymous Coward]

The CA does not have to store customers' private keys in order to be able to revoke certificates. They just need to publish a signed list of revoked serial numbers.

Re: I call BS

[Anonymous Coward]

I agree.

I would also question why anyone would recommend the certs.

I would start by revoking the CA.

Why should I have to check every cert isn't in a list of 23,000 bad ones? Seems like it would bee better to revoke the lowest CA cert and make them reissue to everyone under it.

 

Idiots.

[Narcocide]

What, were they just loose on his desktop next to the vacation photos?

Re:Idiots.

[Anonymous Coward]

What, were they just loose on his desktop next to the vacation photos?

Probably exactly that. In a folder on his laptop he carries around.

Re:Idiots.

[Anonymous Coward]

Right next to the image of his music degree.

Is SSL just "Security Theater"?

[Anonymous Coward]

Sure feels like it. "The practice of investing in countermeasures intended to provide the feeling of improved security while doing little or nothing to achieve it."

Re: Is SSL just "Security Theater"?

[jecowa]

I think SSL is good technology, but the certificate authorities are a scam. I believe you can create your own certificates for free and without some third-party having access to the key and make money from selling it to you.

Re: Is SSL just "Security Theater"?

[Anonymous Coward]

> I believe you can create your own certificates for free

I'm pretty sure I could create a certificate purporting to be google.com. Now if I want to MITM you, you just need to trust my cert. Keeping trusted signing authorities should prevent that. I still get your scam commentary, but the idea of the system is that there's a web of trust going up to someone who can be held responsible.

Re: Is SSL just "Security Theater"?

[Anonymous Coward]

I think you need to do some reading to understand the problem CA's were created to solve.

Re: Is SSL just "Security Theater"?

[Anonymous Coward]

Now if I want to MITM you, you just need to trust my cert. Keeping trusted signing authorities should prevent that.

Nope. For that to work, you would need trustworthy certificate authorities. Of which there are approximately zero. But that doesn't even matter, because by design, if even one CA is not trustworthy, the whole system stops being trustworthy.

In security, "trusted" simply means someone who can compromise your security at will. If your trusted people are also trustworthy, that's all fine. But that's not how it works in the real world. Your trusted people come as a list bundled with you browser, and nobody even looks through the list, much less verify the trustworthiness of any CA on the list.

Re: Is SSL just "Security Theater"?

[Anonymous Coward]

The problem CA's were created to solve?

A way of standardize on a set of root certificates without care for the (lack of) trustworthiness of those issuing certificates.

Oh Well,

[rotorbudd]

Dumbasses gotta dumbass.

I might be dumb...

[Anonymous Coward]

But why the fuck isn't the PUBLIC key signed, and the end user sends a message to the private key to verify it is authentic?

This way only the public part of the signed certificate needs to be provided, but the certificate authority's signing of the public certificate would make it obvious that the remove host did/didn't match the associated metadata (ie hostname/key signature/etc.)

I will personally just stick to self signed certificates for exactly this private key threat model, but obviously the entire chain of trust is untrustworthy in this day and age.

Re:I might be dumb...

[TheRaven64]

But why the fuck isn't the PUBLIC key signed, and the end user sends a message to the private key to verify it is authentic?

It is. You generate a certificate signing request (CSR) from your certificate (which embeds the public key and the metadata fields such as organisation name, host name and so on). You send the CSR to your certificate authority (CA). The CA then gives you back a signed certificate (which may strip out some fields from the cert that the CA doesn't want to attest to). The key exchange phase of TLS then sends the certificate to the client, which can walk the certificate chain to verify that someone (hopefully someone trustworthy) is willing to attest that the public key belongs to the organisation that you think you are communicating with. The client then encrypts using the public key and the server decrypts using the private key.

I will personally just stick to self signed certificates for exactly this private key threat model

You use self-signed certs for a threat model that doesn't exist? This problem existed only because they had customers that decided to outsource certificate creation to them, which is a bad idea and would have failed a security audit.

but obviously the entire chain of trust is untrustworthy in this day and age

If you get a CA to sign your certificate then, at worst, it is no less secure than if you don't. You are still free to distribute the hash out of band and check it. You are still able to use certificate pinning to ensure that you notice if it has unexpectedly changed. And if you don't sign it, then a malicious CA (e.g. one compromised by an intelligence agency) is still able to sign a cert claiming to be yours and have other people trust it. If you use DNSSEC and publish CAA records then you can at least narrow this down to one CA that they must compromise.

Either encrypt your email or stop using it

[Murdoch5]

If you're using email in 2018 and it's not encrypted it, stop using it, simple! The average person and especially executives, have no sense of security or secure operations. I can point to numerous companies, where even the CTO's and CSO's are widely unqualified to hold those positions, and they would and have, send non-encrypted email containing very sensitive information.

If society isn't going to grow up and start encrypting all email communication, then it's time to get rid of email.

Never give your CEO...

[Tony Isaac]

Many CEOs are just technical enough to be dangerous. Never give your CEO:
- Direct access to your database server
- Administrator passwords of any kind, even to their own laptop
- Access to server rooms
- PRIVATE KEYS!

You CAN give a CEO a MacBook Air. They'll be happy with the sleek design, and they won't be able to do much damage, since not a lot of "work" software actually runs on it.

Re:Never give your CEO...

[Anonymous Coward]

Many CEOs are just technical enough to be dangerous. Never give your CEO:
- Direct access to your database server
- Administrator passwords of any kind, even to their own laptop
- Access to server rooms
- PRIVATE KEYS!

You CAN give a CEO a MacBook Air. They'll be happy with the sleek design, and they won't be able to do much damage, since not a lot of "work" software actually runs on it.

You can also give a CEO a great big níggerdick in the mouth!

And once again

[Anonymous Coward]

This is just another reason why I will NEVER trust a UK service of any kind for web hosting or cloud storage. Those idiots wouldn't take privacy seriously if each citizen were bent over and had a CCTV camera shoved up their ass, to which it probably also recognizes. Privacy is a God given right, not a privilege folks and should be taken seriously. I don't think even true AI would be happy living there.

why do executives have access to this type of data

[modmans2ndcoming]

This is why Executives are not kings. There are parts of the business that they should not have casual access to. That is not to say they do not have a right to review and inspect with appropriate parties involved in the process, but access to data and tools like this is not the same thing as keys to the front door.

Re:why do executives have access to this type of d

[Teun]

Although his action to mail the certs was not smart, the initial problem was not the executive.
The Initial problem was his company had kept copies of the private keys, an absolute no-no and when he(?) found out he wanted to communicate which keys were to be revoked.

Part of the Clinton IT camp...Bahahaha!!!

[Anonymous Coward]

Ummmm, was he a Liberal Democrat.... Bahahhaha...

Re:Part of the Clinton IT camp...Bahahaha!!!

[Anonymous Coward]

Clinton was center-right, numbnuts.

BCC: forward to 3-letter agency

[Anonymous Coward]

There's only one reason why anyone would be sending private keys in an email. All your keys belong to us.

Re:why do executives have access to this type of d

[Anonymous Coward]

I can't hold the CTO at fault, he wanted to have the certs revoked and had to prove something bad had happened. Their company shouldn't have been holding the private keys. The certs were already invalid.

Received on 2018-03-01

[Anonymous Coward]

Quoting from email:
Dear Customer,

Today many of our customers are experiencing lengthy delays when attempting to contact us via phone, e-mail and live chat. The reason for the delays was due to an unexpected e-mail that DigiCert sent to our customers containing some inaccurate information. We were not informed that the e-mail would be sent and was caught by surprise.

I sincerely apologise to our customers and partners that have been affected.

We didn't authorise DigiCert to contact our customers and we didn't approve the content of their e-mail. At no time had any private keys been compromised.

We can't go into specific details right now - though we believe the orders placed via our Symantec account were at risk and were poorly managed. In good conscience we decided it wasn't ideal to have any active SSL Certificates on the Symantec systems, nor any that didn't meet our stringent security requirements. Our concerns also relate to the upcoming distrust of all Symantec SSL Certificate brands within Google Chrome - meaning that your SSL Certificate will fail to be trusted in Chrome.

We implemented a system to ensure that all customers would receive a replacement SSL Certificate, though today it had failed to perform this function.

In our view it is absolutely critical that an SSL Certificate performs its intended function. In accordance with CAB Forum guidelines we acted to immediately revoke active SSL Certificates whereby trust was questionable.

We realize that this mass revocation is bothersome and time consuming for you. We're working to contact all customers to get orders replaced as priority and working through a backlog of enquiries. We've sent replacement coupon codes to all of our customers and we urge every customer to immediately replace any affected SSL Certificates.

Unfortunately things didn't go very well for us today and we are extremely sorry for all the confusion and inconvenience that has been caused. We were relying on systems that would easily replace and issue your SSL Certificates automatically, though that didn't occur.

We'll be following up again shortly with an update surrounding what occurred and more information about where we experienced failures. In the meantime, our staff are concentrating on getting your SSL Certificates issued as quickly as possible.

Best regards,

REDACTED
Trustico Online Limited
Customer Service Department

Who sends their PKey over the net. SMH

[Anonymous Coward]

Honestly who would transmit their private key over the internet for signing. Are these lazy people.
No public CA has the PKeys of my customers. People who do that are asking for trouble.