Slashdot Mirror
Maintaining SSH Host Keys Across a Large Network?
skullY asks: "Like most organizations, we have a large number of servers, all of which have unique host keys. Our big problem right now is that the vast majority of our servers are in a cluster and we have one or two machines that get reinstalled every week. Every time this happens, a new SSH host key is generated, and all our existing machines bitch when they have to connect. I realize that the best approach is to install system wide ssh_known_hosts files, and keep them up to date, but in practice this isn't so easy. So my question is how does everyone else keep track of large numbers of SSH host keys that continually change, and propogate those changes to machines on the network?"
Kerberos provides security in various different ways, including authentication, authorization, and encryption. A kerberized telnet is arguable just as secure as ssh (perhaps more so).
sigs are a waste of space
When I used ktelnet years ago, it only encrypted the authentication sequence. Everything else, including typing su, as in the clear. Don't know if this is still the case....
I belive this depends on the implementation of ktelnet you have, but a fully compliant kerberos application can use the kerberos service to support both authentication and encryption.
sigs are a waste of space
You're using the wrong technology for the problem. SSH is great for connecting to the occasional remote host (e.g., to your ISP shell account, to your office computer from home and vice versa), but it was never designed to handle this type of situation.
In this case you really should be using a central authentication method. Kerberos is one well-known example, and there are others as well. In this case you would/could still generate new host keys, but you would only have to update a single server (two, including a backup server).
The flip side of this problem is that you really don't want to allow just anyone to connect to a server cluster - with SSH you can specify acceptable user keys, but it's even more of a nightmare to propogate this information to each system. With a central server, you can disable a user's account systemwide (or on some defined subnet) with a single update. No worries that the guy who had to be fired left a backdoor SSH keyfile on a server or three.
Conversion to Kerberos isn't trivial, but it has some additional benefits. E.g., SSH just gives you communications, but Kerberos authentication is also available to properly configured applications (ktelnet, kftp, ksu, plus sudo, cvs, postgres, lprng, pop/imap, etc. Even NFS, on some non-Linux systems.) I've found a Kerberized system *easier* to use because I'm rarely asked for passwords except when logging in, or asking for root privileges. No need to keep track of separate passwords, lest someone trivially break the CVS password and figure out my account's password.
For every complex problem there is an answer that is clear, simple, and wrong. -- H L Mencken
What's so hard about backing up your data files before you do a reinstall? OpenSSH likes to install them on /usr/local/etc/ssh_host_key, /usr/local/etc/ssh_host_dsa_key, /usr/local/etc/ssh_host_rsa_key, /usr/local/etc/ssh_host_key.pub, /usr/local/etc/ssh_host_dsa_key.pub, and /usr/local/etc/ssh_host_rsa_key.pub. If you back these up before you wipe things and put them back when you're done it's all better. You won't have these problems anymore. Some linux distros might want to put them in /etc rather than in /usr/local/etc, but nonetheless you just need to make backups of unreplaceable data before you do wiping the drives. It's just not that hard.
_____________
I don't want free as in beer. I just want free beer.
The root problem here is that when you reinstall machines you aren't restoring the original ssh host keys. Why don't you start making backups of those keys and restoring them to the proper places whenever you do a reinstall?
Secondly for key distribution why don't you setup a cronjob that will copy the host key list from a single, secure, server every hour or something via scp? Then just make sure that servers host key never changes by doing the above.
Or make a script along the lines of:
for i in host1 host2 host3
ssh root@$i "scp someserver:/keyslist keyslist"
end
and run it every time you need to update all of the key lists.
All those methods are quite secure as they all use scp to do the copying and will fail should the host key of the machine somehow change, say by an imposter. Unless someone manages to break into the machine itself your fine.
ssh is an excellent application. I came from a university environment and started working in a large corporate environment. Sniffers would be easy to install in both. Universities such as mine have ethernet in most classrooms at every desk. My work has ethernet in every cubicle. What was it, that you said about people not being able to install sniffers?
I will admit that you have a valid point about sysadmins writing down passwords, or better yet, everybody accessing certain machines as root. On the other hand, I can only think of one environment where sniffers wouldn't be a concern. That is computer room with the entire network contained to the room.
Keeping
Not one to normally flame, but not using ssh in ANY environment because it's inconvienent is just plain stupid. If I ever had an admin working for me who believed this drivel, I'd fire him on the spot; However, I realize security isn't the easiest thing to learn, so I will explain here. Please indulge me for a moment.
Who can you trust in a datacenter environment (by datacenter, I am refering to a co-lo facility; for most companies, this is a datacenter)? Can you trust:
The $7/hour security guards that "patrol" the area?
The script kiddez who happen to land a job in the NOC (Where they study for their MCSE all day, no doubt)?
The often-incompetent datacenter "engineer" who can't seem to get the router to your cage configured correctly?
Now, let's take your arguments one at a time:
90% of servers have 6 or less users, who are admins or senior dba's/developers. Most connectivity is done through client/server connections to databases or other applications.
Even this statement reflects a poorly designed production area. No developer should have access to the production area. The only people to have access are the release engineer (and only the access he/she needs to get the code out) and the sysadmin.Period. Anything less is just asking for trouble.
Most end-user types cannot even directly connect to the server subnets. All connections go through a broker of some sort.
A "broker"? Do you mean a firewall? The packets have to get to the firewall somehow. I'm willing to bet you don't control every hop along the way as well. I'm also willing to bet that your firewall is allowing packets in and out somewhere.
Besides...we all know that firewalls never get compromised, right?
It's a pretty new network. Everything is switched. Hard to sniff on a switch.
It's trivial to sniff a switch, especially if it's a "pretty new" network. Managed switches all have traffic redirection which can be used to direct all traffic to a specific port. All commercial switches tend towards pathetic security. (There were Cisco advisories on Bugtraq only yesterday).
And even in an office environment, you have to deal with misconfigured clients, viruses (this is a very, very big reason to use ssh), malicious employees, socially-engineered quot&;good intentioned" people, etc., etc.
Now tell me again why you don't use ssh.