OpenSSH Will Feature Key Discovery and Rotation For Easier Switching To Ed25519

ConstantineM writes: OpenSSH developer Damien Miller has posted about a new feature he implemented and committed for the next upcoming 6.8 release of OpenSSH — hostkeys@openssh.com — an OpenSSH extension to the SSH protocol for sshd to automatically send all of its public keys to the client, and for the client to automatically replace all keys of such server within ~/.ssh/known_hosts with the fresh copies as supplied (provided the server is trusted in the first place, of course). The protocol extension is simple enough, and is aimed to make it easier to switch over from DSA to the OpenSSL-free Ed25519 public keys. It is also designed in such a way as to support the concept of spare host keys being stored offline, which could then seamlessly replace main active keys should they ever become compromised.

Comments

Seamlessly replace keys?

[Anonymous Coward]

How about 'no'.

Re:Seamlessly replace keys?

[stoborrobots]

Agreed - this makes sense if you want to display a message to the user: "The server is advertising updated host keys via the trusted channel. Do you want to add them to your local host key list?"; but automatically replacing them without prompting seems overeager...

Re:Seamlessly replace keys?

[stoborrobots]

Although, in hindsight - you've already authenticated the server, so you are going to treat it as a trusted party anyway...

Re: Seamlessly replace keys?

[Anonymous Coward]

Yeah, but it's also easier to compromise a host, send out fake server keys, and then patch things up so the server looks fine again but intercept targeted clients. Even if the server would push out only its current whitelisted keys and they completely replaced known_hosts, users who are already MitM'd wouldn't see that until they can establish a connection to the real server, which they might not even trust anymore due to known_hosts being changed by the MitM. In short, it sounds dangerous. By all means sign the ECDA25517 keys with existing trusted keys, but don't just overwrite anything.

Re:Seamlessly replace keys?

[Anonymous Coward]

Although, in hindsight - you've already authenticated the server, so you are going to treat it as a trusted party anyway...

Agreed, but *I* would still like to be prompted anyways. The same way when it comes to things like windows or app "automatic updates" I always pick the "notify me, but don't download/update without asking" option. I like to *know* what's happening to my systems, and have the option to say NO, even if it's probably unlikely.

Re: Seamlessly replace keys?

[unrtst]

Yeah, but it's also easier to compromise a host, send out fake server keys, and then patch things up so the server looks fine again but intercept targeted clients. ...

I didn't RTFA, but based on the summary it seems like this is unlikely.
If you can compromise a host to the point that you have read access to the private host key, then your MITM can simply use that private key. No need to change to a new key to exploit that. AFAICT, you would need access to the existing host key pair in order to sign the new keys. If this was done right, then it seems like this would be adding very little (if any) additional risk.

That said, I'd want to have the option of being prompted. I could set that for all my interactive shells (or it could be the default), and I could set my automated jobs to auto-sync the new keys.

Re:Seamlessly replace keys?

[paavo512]

It's one thing to trust the server to be who you expect it to be.

It's something completely different to trust that server to tell you how to authenticate all the other servers that you know about. That requires a LOT more trust than "this server is the one I expected"

This is not what this is about. TFA talks only about updating keys for the same server as far as I can see.

Re:Seamlessly replace keys?

[Anonymous Coward]

Please educate yourself about the difference between a public key and a private key. The keys being talked about here are the server's public keys. This is about verifying who the SERVER is. This has ZERO to do with actual authentication, just verification that you are talking to who you think you are talking to.

Re:Seamlessly replace keys?

[ToasterMonkey]

How about 'no'.

If you didn't trust your connection to the server in the first place, then what was the point of using SSH?

Re: Seamlessly replace keys?

[unrtst]

Unless there are other holes in the host that allow remote addition of server keys, but not stealing them directly. Say some logging facility that can be abused to add but not overwrite files in /etc/ssh

Nope. The new keys would need to be signed by the original key. So, even if you can add new files to /etc/ssh, you'd still need read access to the private host key. If you have read access to that, you can just steal it carte blanche and use it on your MITM.

Granted, I'd still probably keep the feature disabled [server side] until I needed to use it. For example, if you want to rotate your host key, then:
1. make the new keys and sign them with the old key
2. update sshd_config to enable the feature
3. let it run like that for a week or so
4. disable old keys and the config entry
That should make it seemless for those that want it seemless and significantly reduce admin costs while not introducing significant risk (updating the host key cache on the client side of 100's/1000's of devices is bound to introduce more risk).

Re:Seamlessly replace keys?

[BarbaraHudson]

Although, in hindsight - you've already authenticated the server, so you are going to treat it as a trusted party anyway...

There's trust, and then there's TRUST. Doing something like this automatically is like doing automatic updates ... not a good idea.

Re:Seamlessly replace keys?

[stoborrobots]

You need slightly more understanding to go with your reading:

"Authentication" means "verification that you are talking to who you think you are talking to".

In SSH, before you send your authentication information to the server (for it to verify that it is talking to you), the server first sends it's own public key, and specific message signature with the corresponding private key. Your client checks if the public key is already known as belonging to the server (by checking known_hosts), and if not, asks if you are willing to trust the key. If you say yes, the client computes the same specific message, checks if the signature sent matches the message and public key. If this succeeds, then your client has successfully authenticated the server (verified that it is right server), and can trust that it is not a "man-in-the-middle" trying to steal your password.

After this, your client sends your authentication information to the server, and the server looks up your password in the password database, or verifies your public key, or whatever, to check your info. If this succeds, the server has authenticated your client, (verifed that your client is under your control), and can trust your client to run commands under your user id.

Re:How about;

[ThatsMyNick]

OpenSSH is not buggy and has good security already. You must be thinking of the OpenSSL.

NSA would have loved this !

[Taco Cowboy]

How can NSA not loving this feature?

I mean, the two way encrypted channels between userA and serverA, if NSA would like to crack it, they would need to invest some crunch time just to take a peek

With this new protocol all NSA needs to do is to perform a MITM act ... get the info needed and then tell the user (and the server) to update keys and such

Mission accomplished !

Re:NSA would have loved this !

[thegarbz]

Correct me if I'm wrong but this can only take place after a fully authenticated session.

If the NSA is in a position to MITM this connection then they are likely already in the possession of all necessary keys at either end and gain nothing by being able to send the client and server new keys.

Re:NSA would have loved this !

[Anonymous Coward]

How can NSA not loving this feature?

I mean, the two way encrypted channels between userA and serverA, if NSA would like to crack it, they would need to invest some crunch time just to take a peek

With this new protocol all NSA needs to do is to perform a MITM act ... get the info needed and then tell the user (and the server) to update keys and such

Mission accomplished !

Let's not go over the top with this. When you say "all NSA needs to do is to perform a MITM act" you are forgetting that the default for SSH is to abort during a MITM. That means they are back at needing "to invest some crunch time just to take a peek". The only loss is that, if they did compromise a server or manage to get into the encrypted channel they could potentially leverage that for a silent compromise of the client side key without thaat being visible on the host. It's still a pretty high risk attack for pretty low gain compared to what they have to already have (the ability to compromise some SSH connections).

Firstly, let's not just talk about the NSA. There are plenty of other bad actors and don't forget the Chinese and Russians also have pretty hot capabilities. Weakening crypto helps the enemy of all kinds. People who suggest reducing encryption knowing that, and I think most of them do, are traitors.

There should be a warning on key replacement or, better, a warning when the new key is taken into use. This would be important in the case where the enemy could compromise some connections but not others and then use key replacement to escalate that to all connections.

Re:NSA would have loved this !

[DarkOx]

Right, I don't see much risk here, and probably loads of to gain in terms of replacing old DSA keys. The only place where I see something like this complicating things in terms of security is now the key store might be changing without user awareness so if you are doing something like having the key store monitored with tripwire or the like you might have an issue.

I suspect people who are going to those kind of lengths probably parse the change logs to security central items like openssh pretty carefully before updating and have the knowledge to develop appropriate solutions to cope with the change, even if that means a patch to reverse the behavior. I am not worried, I think this will make most folks better off.

Re:NSA would have loved this !

[Anonymous Coward]

Until that one time they forget to MITM them, and suddenly the client talks directly to the server and throws an error or removes the rogue key because the server no longer advertises it. It still comes down to that in order to do your attack, they need to compromise the server in the first place and they gain little benefit for a lot of risk by injecting fake keys.

As it stands, there is a very real risk posed by not being able to easily update keys. I have a lot of automated scripts that connect to remote servers where the admins are complete idiots and are constantly changing keys without warning. There is no way I can validate every key change, so I just blinding accept the new key. If this allows me to not have to take time out of my day and not have to just accept a new random key that has no relation to the last one, this is win win for me.

As long as this is optional, I see no issue.

Re:NSA would have loved this !

[Antique Geekmeister]

> ou are forgetting that the default for SSH is to abort during a MITM

With stolen hostkeys on the same IP address? Or by presenting a new host IP in DNS with their own MITM keys, connecting to an unencrypted login transaction logger, and recording the user login attempt and passwords, then using them next time to connect to forward the connection to the relevant upstream host? Or any of a dozen other MITM approaches?

I've been through just such an attack. Fortunately, the person doing the attack gave themselves away by failing to deal with 'ssh-agent' based connections, which is when I got called: key based access to the attacked server stopped working.

Re:NSA would have loved this !

[thegarbz]

If the NSA manages to MITM a session once, then they would already be past the point of encryption. At that point wouldn't they already a) be able to get the client key, and b) be able to get the server key, making it trivially easy to MITM again in the future?

I don't understand how being able to change keys at either end if you are in possession of the correct ones makes it any easier to hide what you are doing or any easier to do it again in the future.

It would seem like once you have the keys and are in position to use them the first time then the connection is forever broken unless the keys change manually.

yep, need "UpdateHostkeys Prompt". Damien?

[raymorris]

Agreed. I want to know if my servers' keys have changed unexpectedly. You can set UpdateHostkeys No to turn this off; I'd like the option of UpdateHostkeys Prompt.

I do understand that having Prompt as the default would undermine the intended use case somewhat, but I think it would be good to have the option.

Hope that code gets better

[rippeltippel]

Maybe it's off-topic, but is it just me who see potentially big problems with ed25519.c? e.g. http://bxr.su/OpenBSD/usr.bin/ssh/ed25519.c#25

Hint: no input validation, hard-coded array offsets with no clue about their expected size, etc...

I know, it's open-source (I should contribute, blablabla) but I see this kind of problems all over that code base.

Re:Hope that code gets better

[bmayer0122]

It is an internal function so the input validation should have already happened.
The hard coded offsets look like they are part of how the crypto works.

Re:Hope that code gets better

[Anonymous Coward]

The OpenBSD/OpenSSH team is regarded as being considered paranoid to paranoid people. They have a long track record of great decisions, excellent design, and excellent code quality. They are known to error on the side of caution and are the standard for security.

Re:Hope that code gets better

[gurnec]

Keep in mind that this is crypto code, which often has different properties and requirements than other types of code.

For example, the count of bytes to store a private key, or a finite field element, or a hash output, etc. is hard coded and never changes. It would be akin to asserting that the sizeof a uint32_t is 4 somewhere in your code.... not very useful. Perhaps some defines would be nice from a documentation point of view, but that's more a style choice.

Input validation must be handled very carefully due to the possibility of opening up side channel attacks (anywhere you see a piece of code which branches depending on its input is a potential vulnerability for a timing attack, and is intentionally avoided by this library).

But the big reason I'm a lot less concerned than you are is simply the reputation of the authors, who are among the most talented cryptographers on the planet. It's not that they can do no harm (such hubris would be self-defeating), but rather that the likelihood of you or I finding some vulnerability with just a cursory glance seems extremely unlikely....

Re:Hope that code gets better

[rippeltippel]

Thank you, I think you're right.
My comments above were probably due mainly to ignorance.
After all I've never bothered looking at that code before and should have studied the whole thing before judging... well... the whole thing.

Re:Hope that code gets better

[fahrbot-bot]

It is an internal function so the input validation should have already happened.

That's adorable.

Re:Hope that code gets better

[TechyImmigrant]

It's a signature algorithm. It should take any input and sign it.

Attack surface

[benjymouse]

How about 'no'.

Agreed.

Consider the situation where an attacker has actually compromised a server key - either it was leaked, brute forces, a vulnerability exploited. It happens and big parts of the certificate system such as revocation lists, OCSP, validity periods etc are concerned with this. Or consider the situation where a vulnerability allows the attacker to *fake* the fact that he has the private key.

Many systems are set up to warn or outright block if they are not able to check revocation for a certain time period. Seems to me, that with this extension an attacker can use the time from when the key is actually compromised until the breach is discovered/reported and gain permanent foothold instead of a temporary one.

This capability seems too dangerous to even have in the protocol - even if one can argue that it is protected. It broadens the attack surface in a significant way.

Of course, all security is a balance between risks, consequences and cost. Maybe, if this capability significantly lowers some other risk, it could outweigh the risk incurred by the increased attack surface. But color me skeptical.

Re:Attack surface

[iggymanz]

logic fails you. we're talking about public keys; after logging into a server via ssh you can already replace or change lines in .known_hosts anyway.

How about a better feature?

[thogard]

We need keys and host passwords checked as authentication types without having to revert to PAM hackery. Just how many systems have been exploited because some root process found a way to read some .ssh/keys and then hopped to other systems with no human intervention.

known_hosts only, not login keys

[heikkile]

If I read the article (or even the summary) correctly, this is about updating the known_hosts file, not authorized_keys. So, even with this enabled, this only affects the "The hostkey has changed" warning message, not who can log in with which keys. Although I am a tad uneasy about automatic key updates, this seems to be fairly safe, and it makes it so much easier to change a hostkey, without bothering all the users of a system.

Dangerous overkill

[drolli]

Dangerous because automatic key updates should require a great deal of verification of the new keys. I could imagine some scenarios (e.g. cloned virtual machines), which lead to the authenticated key being correctly updated (e.g old instructions/documentation) by the admin, but the EC key not overwritten (since it's not in the standard procedure). If this EC key then is copied automatically to the client, any of the cloned machines would accepted as a verified server after the login.

Overkill because exposing the public keys in terms of the sftp protocol would be less invasive and give the client full control what to do whith the keys.

Re:Dangerous overkill

[drolli]

It is dangerous to have partially redundant mechanisms. If you have only explicit redundancy, then errors will show up.

This tool automates an administrative task. But only one.

What's so wrong with a USB stick?

[buckfeta2014]

No thanks, I'll pass on this feature.

SSH

[ledow]

So, I have to trust a server to automatically replace a trusted key with a new trusted key.

Yeah, this is the type of thing I'll try when it's been in the code for five, ten years.

I'm perfectly sure, as a mathematician, that you can use some kind of secure exchange to make this work but - fuck - I won't be trusting implementations of it for a while.

Isn't this exactly the sort of thing that, half-assed, will generate security problems for years to come and yet still seems to be outside the SSH protocol and has to be a custom extension? Is there an RFC for this?

Sorry but as far as I'm concerned key management shouldn't be a part of the process that's handling connection authentications, etc. Why can't this be an outside protocol entirely? For decades, we've been waiting for some kind of automated decentralised, anonymised key-store and surely the effort going into securing this very dangerous piece of code would have been better put into moving the problem away from SSH and allowing multi-protocol use of such things.

Re:SSH

[QuietLagoon]

So, I have to trust a server to automatically replace a trusted key with a new trusted key....

You don't have to do anything. You can use this feature if it is suitable for your needs and security requirements. If it is not, then no one is forcing you to use it, so don't.

Re:SSH

[ledow]

Exactly.

And the ability to overwrite such a critical file should really be something huge and manual because it's so critical.

I'm also thinking DDoS situations - malware replacing your SSH keys with their own, stopping you logging in at all or adding their public key to all your normal ones granting them a kind of hidden back door.

Sure, they can do all that in other ways, but one way built-into the SSH protocol as an extension is something we can do without.

Re:SSH

[iggymanz]

eh? you can already overrite known_hosts or put in any extra keys you want after logging in already. Really one could do such a thing as command argument to ssh already! there's not really much new risk introduced here

Re:SSH

[Anonymous Coward]

I'm perfectly sure, as a mathematician, that you can use some kind of secure exchange to make this work but - fuck - I won't be trusting implementations of it for a while.

If only SSH had some sort of way of, say, encrypting the channel over which you communicate. And if only there were some way to know that you previously acknowledged that you trusted the server.

Maybe that would help?

Re:SSH

[ToasterMonkey]

Sorry but as far as I'm concerned key management shouldn't be a part of the process that's handling connection authentications, etc. Why can't this be an outside protocol entirely? For decades, we've been waiting for some kind of automated decentralised, anonymised key-store and surely the effort going into securing this very dangerous piece of code would have been better put into moving the problem away from SSH and allowing multi-protocol use of such things.

If you trust a server by accepting its public key, it is by definition, trusted, for as long as its private key is secure.
Only the initial trust needs to be verified by humans, and with a chain of trust, even that can be nearly automated by adding your organization's CA key when systems are deployed (I'm in an imaginary world where SSH key management has caught up with the rest of the world).
The older a private key gets, the more likely it has been compromised, maybe by VM cloning, backup media leaking, etc.

To address that, you should change the keys periodically. Prompting the user is pointless, because the connection is trusted.

WHOA, let me back up a minute, you did know your session data is actually encrypted with symmetric keys right? ... and those keys are in similar fashion changed on a regular basis without your knowledge?
If you didn't know that, well.. that explains 99% of the ignorance I'm seeing on this page.

SSH's key management is an absolute joke, but this is a step in the right direction at least. The only thing I can imagine is the authors figured people would be using kerberos in all but the smallest shops... and I'm being nice assuming SSH's kerberos integration is any good.

Re:SSH

[thegarbz]

So, I have to trust a server to automatically replace a trusted key with a new trusted key.

You have already trusted a server, and connected to it in a secure fashion, and authenticated. At that point why would you not trust it to replace the key?

Re:How about;

[Anonymous Coward]

total hogwash. examples please or you're just spouting nonsense like a raving lunatic.

Re:How about;

[fahrbot-bot]

total hogwash. examples please or you're just spouting nonsense like a raving lunatic.

Please note that he, apparently, is Batman so ...

Problem is key clearing, not multiple keys

[Anonymous Coward]

This is so typical of the current OpenSSH maintainers, it actually hurts to watch.

The actual problem is that there's no client tool for clearing replaced keys. You hve to open up your "authorized_keys" file with a text editor or the client keeps moaning "oh, no, the key doesn't match, boo-hoo-boo-hoo!!" even when you know damn well you rebuilt the host and it has new keys, or a dozen similar situations. If the software is smart enough to read and detect old and not matched recorded host keys, it should be intelligent enough to delete the old one if you say "ok, yes, I do want to replace that". But there's not even a tool to do it with. And you can't just use "ssh-keygen" to scan the relevant target and script it, because known_hosts entries are not in a canonical format or one that matches the output of 'ssh-keygen'. So scripting it turns into a right pain in the ass, partly because people ignore the RFC's about hostnames and now are using Unicode hostnames for things. (Check out Thailand and other Asia, non-compliant registrars for an introduction to non-compliant DNS).

The result is that, if you have significant system churn, you set your client to ignore host key validation (Setting "CheckHostIP no") because you *do not have the time to waste with this*. This especially happens when you are building new hosts in a small DHCP assigned address space, or if you have non-registered DHCP for a bunch of development environments that all get built from scratch rather than using a deployed, locked set of standard SSH keys. When keys expire and a new host gets the same IP address, you're screwed.

SSH hostkeys are, frankly, a joke in any case. They cannot be trusted for the initial host authentication because there is no default signature or verification method for the keys themselves. And the client behavior is so painfully awful that everyone who actually handles more than the machine room they built in their mom's basement has learned to just ignore the damn things and either set 'CheckHostIP no' or simply delete the $HOME/.ssh/known_hosts file on a regular basis, because the criping and whinging about accumulated discrepancies break things on an unpredictable basis. And yes, I reported the issue and tried to patch it with SSH 1, SSH 2, and OpenSSH, it's a fundamental architectural "practice".

So basically, this is a complex and stupid answer to what is the wrong problem. It's not a server hostkey publication problem, it's a client "we're too stupid to know how to expire old or invalid hostkeys because we arrogantly assume all servers are complete stable and you should just be able to open 'vi' and fix it yourself" problem.

Re:Problem is key clearing, not multiple keys

[gweihir]

Might be a canary, i.e. a test of how much stupidity and potentially insecure automation OpenSSH users are willing to accept. In that case the OpenSSH maintainers team is not stupid, but has been compromised. Would make sense...