Slashdot Mirror
Disempowering the Singular Sysadmin?
An anonymous reader writes "Practically every computer system appears to be at the mercy of at least one individual who holds root (or whatever other superuser identity can destroy or subvert that system). However, making a system require multiple individuals for any root operation (think of the classic two-key process to launch a nuke) has shortcomings: simple operations sometimes require root, and would be enormously cumbersome if they needed a consensus of administrators to execute. There is the idea of a Distributed Administration Network, which is like a cluster of independently administered servers, but this is a limited case for deployment of certain applications. And besides, DAN appears still to be vaporware. Are there more sweeping yet practical solutions out there for avoiding the weakness of a singular empowered superuser?"
look at programs where there is a lot of technical activity and communication activity for time sensitive work
you can't have a nuclear missile system where one guy can invoke the bombs to go off. at the same time, the system has to be quick and responsive
so you need to engineer administrative systems where not less people are involved but MORE: you can't do this function or that function without also involving this guy over there turning a key, etc.: all admin functions invoke more than one person. that's the best way to have a system where power can't be abused. its about redundancy and layers of admins, not less admins
and if people are pursuing this question because they don't want to pay an admin or can't trust someone else with their system, then such idiots get the system they deserve: a broken one and no one willing to fix it at the money you want to pay
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
would be enormously cumbersome if they needed a consensus of administrators to execute.
Thats why you leave changes to the 24x7 onsite operations team not one lone admin doin' his thing in the cube. They're the ones monitoring the systems, seems most sensible if they "push the buttons" on the things they watch. Ideally you have one team that does nothing but watch and one team that does nothing but do, and theoretically they cooperate.
And besides, DAN appears still to be vaporware.
DAN appears to be a poor reinvention of flight control software for aerospace from the 70s/80s. Those whom don't know their history are doomed to poorly repeating their past.
Next up, we'll reinvent the concept of the security office from AS/400, or maybe the idea of hard realtime control.
Maybe someone out there could could reinvent the concept of the watchdog timer so the "DAN" cluster doesn't go into deadlock? Naah, we'll let them "discover" it themselves, the hard way.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
That is how all of our servers are setup. I'm just a "developer" that uses them but I believe no one knows the root password for our systems. It is a *big* random string that is printed out by the sysadmin that sets up the machine, sealed in an envelope with that person's signature on both sides and stuck in a safe. In the event that a machine is so hosed that the root password is needed it is used and then a new one is generated and sealed away again.
Everyone uses sudo for everything. All sudo access is logged.
The system isn't perfect of course, nothing is, but it goes a long way to the worry of one person having root keys for things.
A subset of administrative applications requiring multiple administrators may not be such a bad compromise.
ex:
* change root password (or password to any "wheel" account) - requires multiple administrators to enter the same passwords
*su/sudo'ing to a "wheel" account, or changing said account's privileges, requires the authorization of at least one other wheel'ed user.
* Alterning an active network interface, shutting down, and restarting requires authorization by other administrative users.
Stuff like that, which are things that shouldn't be done often, anyway, and could allow one admin to take over the whole system, seem like good candidates for multiple-approvals. Everything else could be left alone.
The approval process is basically - the root users needs to take the action, and then 2+ non-root (but wheel) users must approve it.
I'm using 'wheel' as that is the group in FreeBSD that is typically allowed access to sudo/su. Not sure how other systems typically work.
Self proclaimed typo king, and inventor of the bear destroying coffee table (patent not pending).
Sadly enough, I've had a simple drive replacement tied up in meetings and other office politics for months. Write up a proposal for change, sit in meetings where various department heads without a clue discuss the potential hazards, write up the rollback process (for changing a drive?). Your plans are torn apart and put back together. Departmental announcements, customer notifications, etc, etc. Accounting wants numbers, and proposals from 3 sources for the cost of a replacement drive (which you have 5 of in the datacenter, and a regular supplier). You're sitting there with the mind numbing noise flowing past. All you can think is "the array was set up with no hot spare. It's running in a degraded mode. Change the damned drive." Of course, complaints of slow drive performance are scattered throughout the meeting.
Two months and more meetings than you can remember later, they slate it for an arbitrary windows. Saturday at 3am. Not only change it, but you are required to stay while it rebuilds, "just in case...". Just in case? You have me working 8 to 7 Monday through Friday, weekends on demand (which are every weekend) AND you want me to blow off Saturday night to do the change? Ah who cares, I don't need sleep.
Then Thursday afternoon before the schedule change is done, a second drive in the array fails, and the whole thing is down. All the same people who were in on the meetings start screaming "How could you let this happen?!"
Thursday afternoon becomes Thursday night, and by Friday morning you have the array back up and working, through some dumb luck. (crossing fingers, praying to whatever gods may be listening, and tapping the drive with a screwdriver at boot time to make it spin up). The only planning that helped is that you keep a change of clothes and a toothbrush in the car, since you don't have time to go home once you're done. In doing the work, you notice the same thing happening to a neighboring machine. Damned aging hardware. So you just change it without the mess that accompanied the first change. Not only are you bitched out for not fixing the first array in time, but you get it twice as bad for fixing the other one before it became a problem. How could you have independent thought? How could you make a change without proper authorization?
The only thoughts still in your head are "I hate this job", "my car keys are in my pocket, and I could just leave." Is this the day you quit? Maybe, just maybe. Just one more thing, and that'll be it. I don't need this shit.
Friday afternoon, not sleeping since Wednesday night, you are told "Do [some other task] after hours tonight." No, you won't get paid any overtime since you're on salary. The task will take at least 8 hours, and they need it done before Saturday morning. Do you scratch out a resignation with a sharpie on the CEO's wall at 2am, or do you just walk out?
I really hated that job.
Serious? Seriousness is well above my pay grade.
...spoil the soup.
The submission seems to presume that the system in question is some sort of *nix or Windows box. If we look into the world of mainframe operating systems, we'll see that this has already been fully adressed, and any number of individuals with discrete UIDs may have superuser access. This has evolved out of a history where sysadmins worked shifts, so sharing a single privileged UID/password was/is a bad idea.
The way such access is administrated needs a proper policy within the organisation, though. Back in the '90s, I worked at one outfit (an insurance company) where the vice-CEO demanded superuser privileges despite having no knowledge of system administration or any other computing background. He just wanted to act as overlord as to what staff had access to on their signons. I was very tempted to tell him to get fucked, phrased in more professional terms. Like "Go get professionally fucked".
My immediate boss was (wisely) more inclined to a diplomatic approach, however, so he pursuaded me to install a dummy program for him that was enough to convince him that he had what he wanted, without granting him any kind of command line access, or ability to change system configuration.
Sadly enough, I've had a simple drive replacement tied up in meetings and other office politics for months. Write up a proposal for change, sit in meetings where various department heads without a clue discuss the potential hazards, write up the rollback process (for changing a drive?).
Not that I don't agree that some companies make change management more than it needs to be (mine does it OKAY), but I bet the guy I knew years ago who changed a drive on a RAID-5 array had thought about testing and rollback. You see, he received the replacement drive late in the day, ran into the data center, popped out a drive, popped in the new drive, and went home. Sadly, he had pulled the wrong drive.