Slashdot Mirror


Disempowering the Singular Sysadmin?

An anonymous reader writes "Practically every computer system appears to be at the mercy of at least one individual who holds root (or whatever other superuser identity can destroy or subvert that system). However, making a system require multiple individuals for any root operation (think of the classic two-key process to launch a nuke) has shortcomings: simple operations sometimes require root, and would be enormously cumbersome if they needed a consensus of administrators to execute. There is the idea of a Distributed Administration Network, which is like a cluster of independently administered servers, but this is a limited case for deployment of certain applications. And besides, DAN appears still to be vaporware. Are there more sweeping yet practical solutions out there for avoiding the weakness of a singular empowered superuser?"

11 of 433 comments (clear)

  1. In other news... by Anonymous Coward · · Score: 5, Insightful

    Rule by a benevolent dictator has certain advantages, and rule by committee has certain opposite advantages. It was ever thus.

  2. There is a well tested method for that by arivanov · · Score: 5, Insightful

    It is called: "Change Control" and usually goes along with "Revision Control" on configs.

    If you change without recording the reason for change and without checking in the result so that the two versions can be compared and analysed you get a pink slip. Voila. Problem solved.

    --
    Baker's Law: Misery no longer loves company. Nowadays it insists on it
    http://www.sigsegv.cx/
    1. Re:There is a well tested method for that by vlm · · Score: 5, Insightful

      Works, although excruciatingly slowly for planned work.
      The collision of excruciatingly slow proactive planned work, and reactive trouble tickets, always is a source of utter hilarity. Usually the end result is you only do planned proactive paper shuffling for meaningless stuff "lets change the background color to be 0.001% darker" and ram thru development as part of a trouble ticket with no oversight at all (well, to make our big customer happy, we've decided to completely redo our database schema and stored procedures this afternoon as part of the ticket).

      Another example, if it takes a month and endless meetings to replace a failing drive during scheduled maint, and a half hour to replace a failed drive at any time, this simply eliminates all proactive maintenance. Much easier / cheaper to burn the power supply out, have a nice long outage, and then replace the whole device, than to get permission to blow dust out of the air filter.

      The end result is usually much worse than it was at the beginning.

      --
      "Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
    2. Re:There is a well tested method for that by JWSmythe · · Score: 5, Interesting

      Another example, if it takes a month and endless meetings to replace a failing drive during scheduled maintenance, and a half hour to replace a failed drive at any time, ...

          Sadly enough, I've had a simple drive replacement tied up in meetings and other office politics for months. Write up a proposal for change, sit in meetings where various department heads without a clue discuss the potential hazards, write up the rollback process (for changing a drive?). Your plans are torn apart and put back together. Departmental announcements, customer notifications, etc, etc. Accounting wants numbers, and proposals from 3 sources for the cost of a replacement drive (which you have 5 of in the datacenter, and a regular supplier). You're sitting there with the mind numbing noise flowing past. All you can think is "the array was set up with no hot spare. It's running in a degraded mode. Change the damned drive." Of course, complaints of slow drive performance are scattered throughout the meeting.

          Two months and more meetings than you can remember later, they slate it for an arbitrary windows. Saturday at 3am. Not only change it, but you are required to stay while it rebuilds, "just in case...". Just in case? You have me working 8 to 7 Monday through Friday, weekends on demand (which are every weekend) AND you want me to blow off Saturday night to do the change? Ah who cares, I don't need sleep.

          Then Thursday afternoon before the schedule change is done, a second drive in the array fails, and the whole thing is down. All the same people who were in on the meetings start screaming "How could you let this happen?!"

          Thursday afternoon becomes Thursday night, and by Friday morning you have the array back up and working, through some dumb luck. (crossing fingers, praying to whatever gods may be listening, and tapping the drive with a screwdriver at boot time to make it spin up). The only planning that helped is that you keep a change of clothes and a toothbrush in the car, since you don't have time to go home once you're done. In doing the work, you notice the same thing happening to a neighboring machine. Damned aging hardware. So you just change it without the mess that accompanied the first change. Not only are you bitched out for not fixing the first array in time, but you get it twice as bad for fixing the other one before it became a problem. How could you have independent thought? How could you make a change without proper authorization?

          The only thoughts still in your head are "I hate this job", "my car keys are in my pocket, and I could just leave." Is this the day you quit? Maybe, just maybe. Just one more thing, and that'll be it. I don't need this shit.

          Friday afternoon, not sleeping since Wednesday night, you are told "Do [some other task] after hours tonight." No, you won't get paid any overtime since you're on salary. The task will take at least 8 hours, and they need it done before Saturday morning. Do you scratch out a resignation with a sharpie on the CEO's wall at 2am, or do you just walk out?

          I really hated that job.

      --
      Serious? Seriousness is well above my pay grade.
  3. how do they design nuclear missile systems? by circletimessquare · · Score: 5, Interesting

    look at programs where there is a lot of technical activity and communication activity for time sensitive work

    you can't have a nuclear missile system where one guy can invoke the bombs to go off. at the same time, the system has to be quick and responsive

    so you need to engineer administrative systems where not less people are involved but MORE: you can't do this function or that function without also involving this guy over there turning a key, etc.: all admin functions invoke more than one person. that's the best way to have a system where power can't be abused. its about redundancy and layers of admins, not less admins

    and if people are pursuing this question because they don't want to pay an admin or can't trust someone else with their system, then such idiots get the system they deserve: a broken one and no one willing to fix it at the money you want to pay

    --
    intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
  4. Eventually, you have to trust someone. by Rogerborg · · Score: 5, Funny

    Oh, the jobs people work at! Out west, near Hawtch-Hawtch, there's a Hawtch-Hawtcher Bee-Watcher. His job is to watch... is to keep both his eyes on the lazy town bee. A bee that is watched will work harder, you see.

    Well...he watched and he watched. But, in spite of his watch, that bee didn't work any harder. Not mawtch.

    So then somebody said, 'Our old bee-watching man just isn't bee-watching as hard as he can. He ought to be watched by another Hawtch-Hawtcher! The thing that we need is a Bee-Watcher -Watcher!'

    Well... The Bee-Watcher-Watcher watched the Bee-Watcher. He didn't watch well. So another Hawtch-Hawtcher had to come in as a Watch Watcher-Watcher!

    And today all the Hawtchers who live in Hawtch-Hawtch are watching on Watch-Watcher-Watchering-Watch, Watch-Watching the Watcher who's watching that bee.

    You're not a Hawtch-Watcher. You're lucky, you see.

    --
    If you were blocking sigs, you wouldn't have to read this.
  5. There are Safeguards Already by BooRadley · · Score: 5, Insightful

    Mostly, except in very small organizations, there are several implicit safeguards to keep any one person from doing evil with the systems. They are subtle, but effective.

    Peer review: Most sysadmins are hired by other sysadmins, or at the very least a technical manager. This means that you are hired based on your skills, reputation, track record, and demonstrated attitude. This means that ideally, you wouldn't even *think* about intentionally subverting a system, because that would mean breaking it or compromising it in some way, and most professional SA'a are simply too OCD to allow it.

    Business continuity: Most organizations have several layers of continuity in place, such as disaster recovery scenarios, system snapshots, monitoring, and auditing. This means that unless you are VERY subtle, or work for an entirely incompetent team, you WILL get caught, and the damage will be minimized as you are being put into a police car, never to work in IT again.

    There are no "indispensable people:" If you are a sysadmin, and you are the only one who knows your systems, you have not done your job. Every system and app should be documented, and there should be accountability for every change and decision.

    No technical solution will ever replace good management and planning, and a design that eliminates the vulnerabilities of a system to rogue sysadmins, will also eliminate its flexibility. It's just a lot cheaper and easier to try and run a good shop.

    --

    -- lk t lv ll th vwls t f wrds. T svs lts f tm t wrt bt ts pn n th ss t rd nd mks m lk lk cmplt dpsht.

  6. Re:why? by somersault · · Score: 5, Insightful

    Not really. It's fun to think I could do anything I wanted, but I don't want to. I like my job, I like the people I work with, I don't want to screw them over. It's nice to have an employer that trusts you too. If I wasn't trusted, I would probably just leave. If they want me to be able to administer and troubleshoot everything, I obviously need full access.

    --
    which is totally what she said
  7. Re:Yes by jijacob · · Score: 5, Insightful

    If you don't trust your sysadmin, they shouldn't be your sysadmin. Just like the accounting department probably has the ability to steal a certain sum of money before anyone will notice, your sysadmin is given responsibilities that could potentially cause grief if they are on the wrong team.

  8. Re:sternobread by Phreakiture · · Score: 5, Insightful

    Run sudo su - and have at it.

    The solution here is to follow a reasonable security protocol in writing the sudoers file. Specifically, the default action is to prohibit. Permitted actions are then whitelisted. On a high-security system, no entry should allow a user to sudo su -. Problem solved.

    Incidentally, I see no point in locking down users who have physical access to the DC.

    --
    www.wavefront-av.com
  9. Re:Too many cooks... by BrokenHalo · · Score: 5, Interesting

    ...spoil the soup.

    The submission seems to presume that the system in question is some sort of *nix or Windows box. If we look into the world of mainframe operating systems, we'll see that this has already been fully adressed, and any number of individuals with discrete UIDs may have superuser access. This has evolved out of a history where sysadmins worked shifts, so sharing a single privileged UID/password was/is a bad idea.

    The way such access is administrated needs a proper policy within the organisation, though. Back in the '90s, I worked at one outfit (an insurance company) where the vice-CEO demanded superuser privileges despite having no knowledge of system administration or any other computing background. He just wanted to act as overlord as to what staff had access to on their signons. I was very tempted to tell him to get fucked, phrased in more professional terms. Like "Go get professionally fucked".

    My immediate boss was (wisely) more inclined to a diplomatic approach, however, so he pursuaded me to install a dummy program for him that was enough to convince him that he had what he wanted, without granting him any kind of command line access, or ability to change system configuration.