Slashdot Mirror
How Would You Distribute Root Access?
dhanks asks: "I'm one of 10 administrators in our group. We're equally responsible for about 300 UNIX servers. We're having problems keeping track of all the root passwords and some of the administrators have taken it upon themselves to implement different security standards. (sudo with silly !SHELLS restrictions) How do other companies and system administrators handle the distribution of root access? I've been charged with coming up with a security policy and I would like to receive some feedback. I'm currently thinking of personal root accounts that would be locked via the /etc/passwd and would only be accessible via 'sudo su - adm_userid' that way each administrator may have full root access only using his regular user password instead of having to keep track of root passwords." While this is similar to an earlier question, this question deals with insuring authorized administrators have the access they need. How would you distribute root over hundreds of Unix machines to the administrators that need it?
First, create one super administrator from the 10 (sorta like Voltron).
Second, create one giant supercomputer cluster from the 300 machines.
Third, give your new super administrator root access (with their choice of password) on the new supercomputer.
..I just made them a root-user which was typically nick-root, that way things could be more easily tracked, etc. Works pretty well.
I would expect such blatant racism on Fark, but on Slashdot? Mods please ban this asshole.
I keep track of system changes on our linux servers using sudo and CVS. Admittedly, my situation is on a much smaller scale (2 admins, handful of systems). I like to use sudo because then any command that's run as root is traceable back to who did it via the system logs. (of course someone could hide their tracks by editing the logs...) I trust the other admin here, but in the past, I've set up sudo so that people could NOT USE the su command. The reason I did this is so that someone couldn't do a sudo su -, and then do whatever they want as root, unlogged. There are still workarounds, as sudo is not a be-all-end-all of security. You still need standard procedures, and you have to make sure people follow them.
As for the CVS side of things, I just keep a "sysconf" module for each server. Whenever I make any changes to a system file, I will first add it into CVS. Then all subsequent changes are made to the CVS version'd file, and notes and stuff committed to CVS. After committing to CVS, the admin then moves the file into the proper system location and does whatever else is neccessary to make the changes take effect. Once again, it doesn't work unless people use it. There's nothing I have in place that would keep someone from editing the file in the system location (since they need root to put the file into place...), but I try to discourage people from doing that.
Eventually I'd like to write some scripts and a DB backend that will hold the locations of all the files, so it's easier to move them into the proper location. But I haven't started that yet...
Place sig here.
With that many systems, the only rational access control seems to be to be using public keys and SSH agent to deal with the logon issue. Once in the system, the only way to keep track of so many cooks in the kitchen is to have sudo running and logging sysadmin actions. Finally, if there are specific files or groups of files that need special attention, I'd probably use ACLs to control access. Another thing that seems like a pain in the ass until it saves you is RCS. Especially with so many admins, being able to roll back a config change quickly is a lifesaver.
So nobody would get in touch with actual root passwords, which can be stored at a safe place.
In a perfect world, Kerberos is the way to go. Your kerberos ticket would, according to the access controls on each box, grant login and root privileges. SSH can pass along your ticket, granting you seamless access with your credentials.
In practice, Kerberos is really hard to do right and so far ssh support is very weak. But if everything was kerberized (this is in the works), then everything from logins to web access can go through your ticket. Granting root privileges is merely a matter of setting the acl properly and then letting the use ksudo.
PAM, Pluggable-Authentication Modules. The PAM Radius module, with a central radius server, has worked for me in my testing, but I wish the module was a little more featureful in doing something with return attributes. It's one of those things I kept wanted to do, never had time for. A random descriptive page I found is here One minor caveat: it's moderately easy to misconfigure PAM to allow too much access - just be careful and use a standardized configuration.
In general, nobody should EVER type the root password, only their su$NAME password. That way, if it gets compromised (accidentally typed somewhere bad) you only have to change it in one place (NIS master) rather than on all machines.
All of this seems pretty obvious, so let me know if there's something unusual about your setup that makes this unworkable.
Novell Directory Service.
Oops, that's three words. Try "eDirectory" instead.
No, wait a second - I seem to recall that Novell marketing renamed it yet again - now it's called either Ngage, exteNd, Nsure, or Nterprise - not sure which.
Frankly, I'm not even sure the people at Novell know what it's called anymore.
Maybe we should moderate Redmond "+1 Has a Clue" simply for fielding a marketing team that knows its ass from a hole in the ground...
...and ssh rsa authentication in authorized_keys of root's. peice of piss.
just all use the password 'secret' - nobody would ever think a root user could so dumb.
Join the Free Software Foundation
You may want to consider establishing a basic auditing policy, to back up any access controls you put in place.
/dev/audit for AIX, Irix, Unicos, etc.).
Depending on what operating system you are using, you could turn on execve / set*id auditing. This functionality is available in a variety of unix implementations (BSM for Solaris, Snare for Linux,
Alternatively, many OSs provide 'sulog' or equivalent.
Note though, that auditing root suers is an inherently risky process, as a root user can cover their tracks quite easily by removing audit log data; as such, you might want to consider real-time forwarding of audit data to a central server, getting it off the host machine, and away from the administrative influence of the root-level user. For basic log files, this is effectively a tail -f | send across the network. For OS-level auditing, it's generally a little more complex.
Red.
That's easy: With Post-it notes on monitor bezels.
sudoscript preserves your audit trail in root shells. It's not perfect, (there are still ways to evade the auditing) but if your concern is to have a record of root's actions so that problem diagnoses is easier, rather than keeping malicious users from doing bad stuff, then it's useful.
"Even if you are on the right track, you'll get run over if you just sit there" - Will Rogers
I'm a user dealing with this right now. Here's what I wish they'd implement at my place.
Give _everyone_ root access. These machines are behind a firewall, right? These are used by developers working to design/forward your company's projects right? If there's the slightest chance that they'll need root, give it to them.
Now, how do you deal with the chaos that results?
Simple. Write a script that reimages the drives on a regular basis. Daily, weekly, monthly, or even by command. In that way, you know the machines will always be kept up to date.
Use your existing admins to maintain and develope the image that you push down to the client machines. Every user should know that the machines will be reimaged often and that they can't plan on the machine always being in the same state. If they have an application or library that they want to persist, then have a procedure for having one of your admins add it to the master image.
User files should be kept on a file server elsewhere. Home directories may or may not be mounted to the machines as you like.
Everyone deserves root. Even those people that are going to screw the system up. (Once or twice, and they won't do it again.)
1. 2.
- Make sure that only root can execute the su command.
- Distribute access to the su command via sudo, and only allow specific syntax for loading roots
.profile - In roots profile, establish a separate history file for each original user, and a log which shows where they are connected from, and whatever else you need separated by human user
The drawback is that most users cannot su to a non root account, but they can still ssh user@localhost. This is by no means a particularly robust solution, but it is better than having the root account shared completely IMHOI find the best way to distribute root access is not to use passwords at all.
.ssh/authorized_keys file automatically, then removing someone's access or granting access to someone is a simple matter of running the script across all the systems.
Disable the root password (or set it to something nobody knows), and only allow access via ssh's public/private key system. If you have a script which will set up the
This way nobody has to remember a password(s), you don't have to worry about cycling passwords, and if someone leaves you can remove all their access in minutes.
We have a policy of requiring all ssh private keys to have a passphrase of a reasonable length, so people can't go using other people's keys.
Anything is possible, except skiing through revolving doors.
The wheel group is the traditional UNIX way of restricting root access to a privledged few.
/etc/suauth---it's in the man page.
It's not this way with GNU su, because RMS don't like it that way (too totalitarian, etc...).
You can make GNU su act like normal su by adding a line in
When I worked at UnixOps we had several different versions of
Michael.
Linux : Mac
I administrate FreeBSD machines, and by default the wheel group decided who can and cannot run the "su" command at all, and that is with the root password. In order to su successfully, a wheel member must also have the root password; other users aren't allowed to even attempt it. I am sure that this can be changed to match the passwordless authentication that you have mentioned, but I actually prefer the current model I have been using. The security implications are complicated, but, password or no, if a person has root access, they can divulge the information no matter what. The biggest benefit to keeping the root password secret to all admins, however, is that they can't do direct logins as the root user; you will have good records of who did what from which admin's account.
If you use "root", someday you will be rooted.
[user@machine:~]$ fakeroot /etc/shadow
[root@machine:~]$ whoami
root
[root@machine:~]$ rm
rm: remove write-protected regular file `/etc/shadow'? y
rm: cannot remove `/etc/shadow': Permission denied
Problem solved, right?
I used up all my sick days, so I'm calling in dead.
Trying to "restrict" sudo access via ! commands is dumb - there are too many shell escapes, etc. At some point, you MUST trust your admins, so just give 'em sudo=ALL. Having said that, I would setup syslogging to a central loghosts, and have some sort of audit process so if someone does an "su root" or a "sudo csh" (or futzs with the syslog configuration), then you beat 'em over the head with a baseball bat! ;-)
Ohhhh ... you say can't do the later ... then you are basically screwed, since if you don't have management support for this, you'll never succeed unless all of your admins realize having logging/accountability/etc. of root-type actions is a darn good thing for everyone - those type of folks work hard to make SURE whatever they do is logged ... whereas there always seems to be at least one admin who thinks they are above this stuff - some eventually learn, some don't.
BTW, note the loghosts (plural) above ... you should have this allready in place for general security purposes ... and NO admin should have access to all of the loghost machines - i.e. this allows you to deal with renegade Sysadmins who cover try to cover their tracks ... or worse yet, someone who tries to "frame" another Sysadmin.
sudoscript was allready mentioned as a nice compliment to sudo, and the sudo tools are also handy for some auditing features.
Hulk SMASH Celiac Disease
This may not scale well to your 300 servers, but I saw it used effectively for 20-30 servers at one company I worked for.
The basic idea is to use a locked box to store the passwords in. The box is secured with a simple padlock, to which every knows the combination. For each machine you want to manage the password for, you have an envelope in the box with that machine's name on it in the box, along with a bunch of empty envelopes and some blank password sheets which I'll describe later.
So, you're setting up a new server, say, "fred". You take a blank password sheet and fresh envelope out of the lockbox. The password sheet is basically a simple table with password, name, date, and reason columns on it. Write "fred" at the top of the password form, and then fill in the first line with the root password you've just chosen, your name, today's date, and the reason for the password change, in this case "new machine setup". Fold up the password sheet, put it in the envelope and write "fred" on the front of the envelope. Seal the envelope, and then write your name and the date across the seal. Put the now-sealed envelope in the lockbox and lock the padlock back in place.
Ok, that was quite a bit of work. So, what's it do for you? Suppose you need to do some administration on fred, and you've either forgotten or never knew its root password. You go to the lockbox, open it up (remember, everybody knows the padlock combination), and find the envelope labelled "fred". Now, look at the back of the envelope. Is it unsealed, or does it look like the signature/date over the seal has been disturbed in any way? If so, you might want to change the root password before you're done.
Assuming the date and signature on the seal looks good, open the envelope and pull out the password form. The current password is the last line. Fill out the next line with the same password, you name, the date, and some reason like "forgot password". Throw away the old envelope, take a new one from the box, refold the password form and seal it in the new envelope. Write "fred" on the front of the envelope, and sign and date the seal on the back. Now you know the password.
Now, what do you do if you actually want to change the root password for some reason? It's pretty much the same as above. Find the envelope for the system, and open it to find out the current password (if you don't already know it). Make a new password table entry, and seal a new envelope as described above, and put it back in the lockbox. Note that you can do this at any time, and your fellow admins can still find out the new password without any help from you.
This approach assumes a relatively low rate of password change, and that administrators have physical access to the lockbox most of the time. It also assumes that you want per-machine root passwords. In addition, it allows admins to memorize passwords for machines they use relatively frequently without having to write down passwords for machines they use rarely.
We use powerbroker to control root and limited root access. Think of it as like sudo, but the rules are maintained on a centeralized, trusted host. The trusted host then authenticates and authorizes the request, then executes the requested command on the target host. And optionally loggs both the event (user x executed command y on host z), and can also record the entire session. .ssh/authorized_keys file with the "command=..." parameter (which forces that key into executing the specified script). That script can then take the rest of the command line, which specifies a target host / command / user, parse it, authenticate/authorized it, then issue the command on the target host using a seperate keypair (the public side of which is loaded in root's .ssh/authorized_keys file, and the private side is known only to the trusted host). You can have the trusted host authorize the command by matching it against your defined rules, using either a simple shell script or a perl script.
However, powerbroker is commercial software and a bit expensive. You can accomplish the same thing by having users run a script that issues an ssh command to a trusted server, which in turn relays that command (if approved) to the target host. The way you do this is to have one keypair that all client users use to issue ssh commands to the trusted host, and the trusted host then has the public side of that key in its
We have a little over 450 systems The eight of us can login to all of them via ssh to our regular unix accounts and su to root as needed. We all know the root passwords for all of the machines for use during emergency situations where direct console or single user mode access is needed. We also have a rather complex centralized sudoers file that doles out access to functional application groups to specific commands as needed. We've often discussed going to a model where we all have to use sudo to run commands as root, but in truth, it's never been a problem over the past 5 years. Sure people make mistakes, but that will happen no matter what. Anything that makes our job more difficult without adding any real value is a waste of time. Once you get into situations where you have hundreds or thousands of production systems, you'll be trying to do everything you can to weed out complexity, not add to it. If you have someone in your group running around like a cowboy and trying to cover their tracks, you've got more serious problems. These kind of things are better dealt with with peer pressure and the occasional beat down in the parking lot ... ;-)
"Who hasn't slipped into the break room for a quick nibble on a love Newton before?" - Mr. Peterman.
> "How would you distribute root over hundreds of Unix machines to the administrators that need it?"
/etc/ssh/authorized_keys/{login} using an access control list that we distribute to every box. The control file is centrally managed (kept in CVS) and has groupings for roles, individuals and servers, so administration is a breeze. The generator is just a short perl script. Finally, we have these lines in /etc/ssh/sshd_config:
/etc/ssh/authorized_keys/%u
:)
We have a similar team size and a similar number of servers. In addition there are other teams with access to more limited (regular user) team role logins. Access also varies according to server role and location.
These systems are often located in continents away in untrusted locations. So passwords are not acceptable.
My solution:
We have a standard Debian package that updates
AuthorizedKeysFile
PermitRootLogin without-password
We make a contingency for emergencies, but I won't describe it here. Suffice to say that it's safe enough to use, analysed enough that it's not snake-oil, and inconvenient enough to stop sysadmins in a hurry from using it by default
- J
I'm surprized that nobody mentioned cfengine2. I believe there are couple of ways you can distribute your passwords across the net using it... cfengine2 is an exellent solution to keep multiple computers intact..
Hey!
Just for an educational survey
Can you give me the ip of one of those boxes that is on the net?
I find this thread very very interesting. I'm one of the 10 SysAdmins that dhanks works with. Yes, I have servers with quote the silly sudo with !SHELL endquote restrictions.
.sh_history file from the dhanks directory (sorry for the length):
/usr/local/bin/sudo vi .sh .ksh / /etc/passwd /home/root .ksh .sh_history .sh_history /home/root .sh_history .sh_history /usr/local/bin/sudo tail /home/root/.sh_history /usr/local/bin/sudo Rsh /usr/local/bin/sudo /usr /usr/local/bin/sudo /bin/tsh /usr/local/bin/sudo /bin/msh /usr/local/bin/sudo /bin/chsh /usr/local/bin/sudo /bin/bsh /bin/ksh /usr/local/bin/sudo /bin/ksh /usr/local/bin/sudo cp /bin/ksh /tmp/fuck-this /usr/local/bin/sudo /tmp/fuck-this /tmp/fuck-this /usr/local/bin/sudo rm /tmp/fuck-this /usr/local/bin/sudo cp /bin/ksh /tmp/shell /usr/local/bin/sudo /tmp/shell /etc/security/limits /etc/security
/dev/null.
First a bit of background - these are AIX servers, sudo is set up so sysadmins can't just go to a shell (plus a few other minor restrictions), there are normal password settings (8 characters, etc), modest logging is enabled, and there are Corporate Security Policies and Practices. Only the Primary and Secondary SysAdmins know the root password (plus the Supervisor in a sealed envelope). All other 8 SysAdmins are supposed to use sudo. Oh, and somewhere we've acquired 150 servers that I know nothing about....
I've been gradually putting these things in place due to Sarbanes-Oxley and the increasing security needs in our world. DHanks has been affected by this since he likes to use a common dictionary word for his password, and he absolutely does not want anyone else to be able to see what he does on a server.
Now to tie it back to the various parts of this thread - I too used to trust all our SysAdmins until one showed me how wrong that was. Take a look at the following
rm
rm
cd
grep root
( cd
ls -lt
ls -lt
tail
"sleep 10; >/home/root/.sh_history &";exit
cd
ls
tail
ls -lt
telnet
exit
exit
cd
ls *sh
cd bin
ls *sh
bash
exit
exit
sudo
set -o vi
passwd otheruser
sudo rm
set -o vi
whoami
vi
cd
ls
grep doug *
vi user
exit
exit
I'll let this speak for itself, though any comments are welcome on what you see. In other history files, there is evidence of editing of various system log files, and other non-condoned work. For a while the history file was piped to
I believe this clearly points out that sudo and almost any other technique I've seen discussed in this whole thread is only effective if the SysAdmin is trustworthy to start with. Once any kind of root level access is achieved, there are too many ways for someone to get around logging, sudo resctrictions, etc. Possibly at higher auditing levels, this is harder, but still not impossible.
If you read the original question, the last italized sentence, you'll see that the security policy is designed to give full root access to all SysAdmins. Is this wise where you have SysAdmins willing to do what you see in the history file?