Slashdot Mirror
Maintaining SSH Host Keys Across a Large Network?
skullY asks: "Like most organizations, we have a large number of servers, all of which have unique host keys. Our big problem right now is that the vast majority of our servers are in a cluster and we have one or two machines that get reinstalled every week. Every time this happens, a new SSH host key is generated, and all our existing machines bitch when they have to connect. I realize that the best approach is to install system wide ssh_known_hosts files, and keep them up to date, but in practice this isn't so easy. So my question is how does everyone else keep track of large numbers of SSH host keys that continually change, and propogate those changes to machines on the network?"
Not one to normally flame, but not using ssh in ANY environment because it's inconvienent is just plain stupid. If I ever had an admin working for me who believed this drivel, I'd fire him on the spot; However, I realize security isn't the easiest thing to learn, so I will explain here. Please indulge me for a moment.
Who can you trust in a datacenter environment (by datacenter, I am refering to a co-lo facility; for most companies, this is a datacenter)? Can you trust:
The $7/hour security guards that "patrol" the area?
The script kiddez who happen to land a job in the NOC (Where they study for their MCSE all day, no doubt)?
The often-incompetent datacenter "engineer" who can't seem to get the router to your cage configured correctly?
Now, let's take your arguments one at a time:
90% of servers have 6 or less users, who are admins or senior dba's/developers. Most connectivity is done through client/server connections to databases or other applications.
Even this statement reflects a poorly designed production area. No developer should have access to the production area. The only people to have access are the release engineer (and only the access he/she needs to get the code out) and the sysadmin.Period. Anything less is just asking for trouble.
Most end-user types cannot even directly connect to the server subnets. All connections go through a broker of some sort.
A "broker"? Do you mean a firewall? The packets have to get to the firewall somehow. I'm willing to bet you don't control every hop along the way as well. I'm also willing to bet that your firewall is allowing packets in and out somewhere.
Besides...we all know that firewalls never get compromised, right?
It's a pretty new network. Everything is switched. Hard to sniff on a switch.
It's trivial to sniff a switch, especially if it's a "pretty new" network. Managed switches all have traffic redirection which can be used to direct all traffic to a specific port. All commercial switches tend towards pathetic security. (There were Cisco advisories on Bugtraq only yesterday).
And even in an office environment, you have to deal with misconfigured clients, viruses (this is a very, very big reason to use ssh), malicious employees, socially-engineered quot&;good intentioned" people, etc., etc.
Now tell me again why you don't use ssh.
Backing up and restoring ssh keys is a very good idea, but this still does not address the issue of large scale manageability. What happens when you add a new server? What if you want to revoke a key for security purposes?
Rolling your own solution is probably the only way. The mass "scp keyfile to all machines" trick is sort of inelegant but effective. Ideally you would have a system that reads key info from DNS. I remember hearing something about extensions to BIND to do something like this, but I don't recall the details. If you do go hacking DNS (and thus also hacking the ssh client) then you have to make sure your DNS is also authenticated properly... bleh it's late and I'm not making any more sense. :)