Slashdot Mirror
Disempowering the Singular Sysadmin?
An anonymous reader writes "Practically every computer system appears to be at the mercy of at least one individual who holds root (or whatever other superuser identity can destroy or subvert that system). However, making a system require multiple individuals for any root operation (think of the classic two-key process to launch a nuke) has shortcomings: simple operations sometimes require root, and would be enormously cumbersome if they needed a consensus of administrators to execute. There is the idea of a Distributed Administration Network, which is like a cluster of independently administered servers, but this is a limited case for deployment of certain applications. And besides, DAN appears still to be vaporware. Are there more sweeping yet practical solutions out there for avoiding the weakness of a singular empowered superuser?"
Rule by a benevolent dictator has certain advantages, and rule by committee has certain opposite advantages. It was ever thus.
It is called: "Change Control" and usually goes along with "Revision Control" on configs.
If you change without recording the reason for change and without checking in the result so that the two versions can be compared and analysed you get a pink slip. Voila. Problem solved.
Baker's Law: Misery no longer loves company. Nowadays it insists on it
http://www.sigsegv.cx/
look at programs where there is a lot of technical activity and communication activity for time sensitive work
you can't have a nuclear missile system where one guy can invoke the bombs to go off. at the same time, the system has to be quick and responsive
so you need to engineer administrative systems where not less people are involved but MORE: you can't do this function or that function without also involving this guy over there turning a key, etc.: all admin functions invoke more than one person. that's the best way to have a system where power can't be abused. its about redundancy and layers of admins, not less admins
and if people are pursuing this question because they don't want to pay an admin or can't trust someone else with their system, then such idiots get the system they deserve: a broken one and no one willing to fix it at the money you want to pay
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
Oh, the jobs people work at! Out west, near Hawtch-Hawtch, there's a Hawtch-Hawtcher Bee-Watcher. His job is to watch... is to keep both his eyes on the lazy town bee. A bee that is watched will work harder, you see.
Well...he watched and he watched. But, in spite of his watch, that bee didn't work any harder. Not mawtch.
So then somebody said, 'Our old bee-watching man just isn't bee-watching as hard as he can. He ought to be watched by another Hawtch-Hawtcher! The thing that we need is a Bee-Watcher -Watcher!'
Well... The Bee-Watcher-Watcher watched the Bee-Watcher. He didn't watch well. So another Hawtch-Hawtcher had to come in as a Watch Watcher-Watcher!
And today all the Hawtchers who live in Hawtch-Hawtch are watching on Watch-Watcher-Watchering-Watch, Watch-Watching the Watcher who's watching that bee.
You're not a Hawtch-Watcher. You're lucky, you see.
If you were blocking sigs, you wouldn't have to read this.
would be enormously cumbersome if they needed a consensus of administrators to execute.
Thats why you leave changes to the 24x7 onsite operations team not one lone admin doin' his thing in the cube. They're the ones monitoring the systems, seems most sensible if they "push the buttons" on the things they watch. Ideally you have one team that does nothing but watch and one team that does nothing but do, and theoretically they cooperate.
And besides, DAN appears still to be vaporware.
DAN appears to be a poor reinvention of flight control software for aerospace from the 70s/80s. Those whom don't know their history are doomed to poorly repeating their past.
Next up, we'll reinvent the concept of the security office from AS/400, or maybe the idea of hard realtime control.
Maybe someone out there could could reinvent the concept of the watchdog timer so the "DAN" cluster doesn't go into deadlock? Naah, we'll let them "discover" it themselves, the hard way.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
That you have one person doing it. It's effective, and versatile.
If you have multiple people empowered to do exactly the same thing, you end up at the mercy of the one that decides to shut everyone else out.
If you then have a security admin that's the only one to be able to alter the login info, then you're at their mercy.
With the "dual key" type approach, what's to stop someone installing a back door along with a normal software upgrade? Does everyone have the same knowledge as your prime sysop? Can you afford to have one person that completely mirrors another, instead of distributing the skills across a time (with duplication covered across the team)?
What if both the key holders are in cahoots?
Interestingly, who is stopping your CEO from making those really bad decisions, or your FD from siphoning the cash, or a whole host of other areas where you trust one person to do a job?
Value the person, and make sure you treat them well enough to make it not worth their while to play you up.. Then you'll have no problem.
Screw them over at every opportunity, and you'll really have to trust their ethical views (you're still usually safe, but it's no guarantee then).
Mostly, except in very small organizations, there are several implicit safeguards to keep any one person from doing evil with the systems. They are subtle, but effective.
Peer review: Most sysadmins are hired by other sysadmins, or at the very least a technical manager. This means that you are hired based on your skills, reputation, track record, and demonstrated attitude. This means that ideally, you wouldn't even *think* about intentionally subverting a system, because that would mean breaking it or compromising it in some way, and most professional SA'a are simply too OCD to allow it.
Business continuity: Most organizations have several layers of continuity in place, such as disaster recovery scenarios, system snapshots, monitoring, and auditing. This means that unless you are VERY subtle, or work for an entirely incompetent team, you WILL get caught, and the damage will be minimized as you are being put into a police car, never to work in IT again.
There are no "indispensable people:" If you are a sysadmin, and you are the only one who knows your systems, you have not done your job. Every system and app should be documented, and there should be accountability for every change and decision.
No technical solution will ever replace good management and planning, and a design that eliminates the vulnerabilities of a system to rogue sysadmins, will also eliminate its flexibility. It's just a lot cheaper and easier to try and run a good shop.
-- lk t lv ll th vwls t f wrds. T svs lts f tm t wrt bt ts pn n th ss t rd nd mks m lk lk cmplt dpsht.
That is how all of our servers are setup. I'm just a "developer" that uses them but I believe no one knows the root password for our systems. It is a *big* random string that is printed out by the sysadmin that sets up the machine, sealed in an envelope with that person's signature on both sides and stuck in a safe. In the event that a machine is so hosed that the root password is needed it is used and then a new one is generated and sealed away again.
Everyone uses sudo for everything. All sudo access is logged.
The system isn't perfect of course, nothing is, but it goes a long way to the worry of one person having root keys for things.
First, understand that Slashdot is only going to provide a hint of what you will be doing. Security is complex and easy to get wrong, and there's a whole lot of evidence of that in the news. If security is important to your company, you should invest in a CISSP to really help you get things set up in a fashion that the industry considers to be best practices. Until then, consider these few generic suggestions.
Multiple layers of security help ensure that nothing goes astray, or if it does that it's detected before too much damage is done. And separation of duties helps make sure that one rogue actor can't do it all by himself.
Separate the admin of the box from the admin of the data. The guy who holds the root PW doesn't have to be the same guy who holds the private key for the database.
Add off-the-box auditing to the actions of root. As soon as someone signs on as root, notification is sent to a different box of the originating IP and it's timestamped. Don't let your application sysadmin be the sysadmin of the audit box! And the auditor should investigate carefully any situations that are out of the ordinary. (This box fell off the network just before root logged on? That's an odd coincidence.)
Define expected behavior with policies. If you want to run a trustworthy ship, clearly stating who has access to do what with which systems eliminates confusion, and helps avoid where one sysadmin creeps over into other systems.
Ultimately, you've placed trust your admin to do a job, and you need to trust him or her to do that job. Somebody's got to be root. But they also have to know they'll be held accountable for what they do.
John
We have several solutions which work together to minimize the risk of root at my company:
1. Powerbroker. It's in use on every single UNIX system administered by our Global IT teams. Every user has a role (or several roles), and that allows them to execute a variety of commands with elevated privileges. Once Powerbroker is invoked, however, every single keystroke is logged and can be played back. These logs are stored indefinitely; access is very restricted.
2. Automated, centralized root password management. One of the steps to setting up a UNIX machine here is ensuring the root password and remote console admin passwords match that dictated by our automated provisioning system. Then every 30-90 days (depending on policy for this type of system) the root password is changed to a very long, apparently very random string. I can look this password up if my role allows it, but the lookup is also logged.
3. A good Change Request (CR) process. Every system that exists in a data center should have a record in our systems database. Once a system has passed through the phases of deployment (Warehouse -> Data Center Install -> Sysadm Configure -> Deployed) any change made to the system must be requested and approved by the owners of the system. This approval is logged, and the date/time of the work is also logged. Sysadms must close service requests within the time window specified by the CR, or apply for an extension or reschedule if they're unable to complete it within the allotted time.
The downside to this is that you lose quite a bit of system administrator work hours filing and managing change requests. However, this loss of efficiency -- IMHO -- is better than the mayhem that ensues without an organized change process.
4. Automated forensic tools to monitor changes. Information overload is a real risk with any Tripwire-style system, though. We're still working out some of the kinks on this part of the system. Once we ensure that all normal changes due to operation of the system and scheduled maintenance get excluded, this will be the fourth leg to reduce the risk of super-user privileges.
At any company, IT must find a balance between controlling user actions and monitoring those actions. In most cases, the easiest approach is to prohibit by policy only those things that might typically result in lawsuits, but monitor everything else to the best of your ability. Combining a Powerbroker-like product with automated root password management -- both with fascistic logging -- is a reasonable approach that works well for many large companies. Combine this with a change management system, and a forensic tool to automatically monitor and notify of unauthorized changes, and super-user isn't really all that big of a concern.
Matthew P. Barnson
I learn what I think when I read what I write
Everyone treats everyone else like adults and every one acts like an adult? Honestly, if you don't trust your admins, why are they your admins?
Also, simple change management alleviates most of these problems. Even if it's just a log for what happened so that the next shift or your colleague tomorrow knows what you did today. Then again, I guess that is really back to acting like adults.
"I use a Mac because I'm just better than you are."
Not really. It's fun to think I could do anything I wanted, but I don't want to. I like my job, I like the people I work with, I don't want to screw them over. It's nice to have an employer that trusts you too. If I wasn't trusted, I would probably just leave. If they want me to be able to administer and troubleshoot everything, I obviously need full access.
which is totally what she said
This is an old problem in high assurance systems. As other posters have pointed out, as some point you have to trust someone. But you can still "trust but verify".
The standard solution is "division of privilege". Over time folks have learned that the key is a system which audits everything the admin does, and the one thing the admin can't do is modify or delete the audit trail. A separate person or team has the role of auditor.
This is one of the requirements of a B2 level system in the old Orange Book model, and you'll see if it as a requirement if you need to provide systems for most countries' military or intelligence organizations. It's rarely used elsewhere because more or less noone else is willing to pay the staffing costs. The solution there is trust someone, and be ready to fire, sue, and/or prosecute if they violate that trust.
A subset of administrative applications requiring multiple administrators may not be such a bad compromise.
ex:
* change root password (or password to any "wheel" account) - requires multiple administrators to enter the same passwords
*su/sudo'ing to a "wheel" account, or changing said account's privileges, requires the authorization of at least one other wheel'ed user.
* Alterning an active network interface, shutting down, and restarting requires authorization by other administrative users.
Stuff like that, which are things that shouldn't be done often, anyway, and could allow one admin to take over the whole system, seem like good candidates for multiple-approvals. Everything else could be left alone.
The approval process is basically - the root users needs to take the action, and then 2+ non-root (but wheel) users must approve it.
I'm using 'wheel' as that is the group in FreeBSD that is typically allowed access to sudo/su. Not sure how other systems typically work.
Self proclaimed typo king, and inventor of the bear destroying coffee table (patent not pending).
Peer Review, Change Control, Auditing, Maintenance Windows, Testing all changes in a lab before production, source and version control / maintenance. These are all best practices, work regardless of operating system and don't require any special software.
Why o why do you want to use software to take the place of established best practices? Best practices are there for good reasons, and those reasons usually have multi-million dollar lessons attached to them. You don't need special software, just a heavy that says yes you /must/ do it this way and raises hell when you try otherwise...
This.
if you can't trust the person at the top: then either they don't deserve to be there, or you need to find a new job.
when you're the person at the top: you better have earned the trust and respect of those under you. Subverting it does nobody any good in any long term.
You keep your passwords in a network share? Are you schizophrenic or just incompetent?
I hope that file is fucking well encrypted ... but even in that case, it's just a bad idea.
WTF am I doing replying to an AC at 5 A.M on a Friday night?
sudo logs are almost useless for system audit. Run sudo su - and have at it. There are no logs to follow what actions you perform. Go ahead and craft a sudoers file that eliminates all the ways to load up a shell. Have fun with that...
fine, no soup. just type sudo make me a sandwich
rewriting history since 2109
If you don't trust your sysadmin, they shouldn't be your sysadmin. Just like the accounting department probably has the ability to steal a certain sum of money before anyone will notice, your sysadmin is given responsibilities that could potentially cause grief if they are on the wrong team.
The solution here is to follow a reasonable security protocol in writing the sudoers file. Specifically, the default action is to prohibit. Permitted actions are then whitelisted. On a high-security system, no entry should allow a user to sudo su -. Problem solved.
Incidentally, I see no point in locking down users who have physical access to the DC.
www.wavefront-av.com
"trust but verify"
To get some transparency / accountability, just set up an authlog black hole that includes all of the sudo activity from your servers.
Yep. And a single malicious incident could end my career. A career I've spent many decades and countless hours on. There's no way I'd risk it. And that's assuming that my morals would allow me to seriously consider jeopardizing it in the first place.
Obviously there are those with different goals and standards and it's not always easy to identify them. I'm not sure how to prevent that -- someone who over the years gradually gets more access and one day they decide to go rogue and do something harmful. Even minimizing the attack surface you usually have that single admin account that owns everything else. Maybe I should read the article.
The tricky part comes in at the point that, while most CEOs have at least a basic understanding of accounting and other departments under their watch, IT departments are *typically* a foreign land to the understanding of those in charge. Even if they wanted to audit proper usage of root it would be difficult or impossible. Small businesses have it hardest. At least in the larger ones there's a layering system so you can have higher-ups in IT auditing the lower guys.
...spoil the soup.
The submission seems to presume that the system in question is some sort of *nix or Windows box. If we look into the world of mainframe operating systems, we'll see that this has already been fully adressed, and any number of individuals with discrete UIDs may have superuser access. This has evolved out of a history where sysadmins worked shifts, so sharing a single privileged UID/password was/is a bad idea.
The way such access is administrated needs a proper policy within the organisation, though. Back in the '90s, I worked at one outfit (an insurance company) where the vice-CEO demanded superuser privileges despite having no knowledge of system administration or any other computing background. He just wanted to act as overlord as to what staff had access to on their signons. I was very tempted to tell him to get fucked, phrased in more professional terms. Like "Go get professionally fucked".
My immediate boss was (wisely) more inclined to a diplomatic approach, however, so he pursuaded me to install a dummy program for him that was enough to convince him that he had what he wanted, without granting him any kind of command line access, or ability to change system configuration.
Well, there's /root/.bash_history
But if your sudo activity log has you doing "su -", then whatever gets borked up after that is automagically your fault as a matter of policy ^_^
Yeah, nobody's ever altered that file. Also, make sure you are watching for changes to your syslogd config, lest someone disable forwarding, do something snarky, turn it back on. But then, security is rarely something that can be solved definitively by means of one slashdot comment.
No. Now just hang on a second while I delete your user account and all your data, you presumptuous bitch.
I've fallen off your lawn, and I can't get up.
I think you're missing the point. Auditing/logging systems are not meant to provide effective defense. They are meant to let PHB's mark appropriate check boxes on compliance forms and sleep better without worrying what those evil nasty sysadmins are doing. Don't confuse them.