Slashdot Mirror


Disempowering the Singular Sysadmin?

An anonymous reader writes "Practically every computer system appears to be at the mercy of at least one individual who holds root (or whatever other superuser identity can destroy or subvert that system). However, making a system require multiple individuals for any root operation (think of the classic two-key process to launch a nuke) has shortcomings: simple operations sometimes require root, and would be enormously cumbersome if they needed a consensus of administrators to execute. There is the idea of a Distributed Administration Network, which is like a cluster of independently administered servers, but this is a limited case for deployment of certain applications. And besides, DAN appears still to be vaporware. Are there more sweeping yet practical solutions out there for avoiding the weakness of a singular empowered superuser?"

66 of 433 comments (clear)

  1. In other news... by Anonymous Coward · · Score: 5, Insightful

    Rule by a benevolent dictator has certain advantages, and rule by committee has certain opposite advantages. It was ever thus.

  2. There is a well tested method for that by arivanov · · Score: 5, Insightful

    It is called: "Change Control" and usually goes along with "Revision Control" on configs.

    If you change without recording the reason for change and without checking in the result so that the two versions can be compared and analysed you get a pink slip. Voila. Problem solved.

    --
    Baker's Law: Misery no longer loves company. Nowadays it insists on it
    http://www.sigsegv.cx/
    1. Re:There is a well tested method for that by Anon-Admin · · Score: 4, Insightful

      What an Amazing Idea, now tell me who does this? I have worked for 4 fortune 10 companies and 1 financial institution. Not a single one has used Revision control, and only one has used change control. That is if you consider a meeting of 20 non-technical managers who can nix a change with out explaining why, change control.

    2. Re:There is a well tested method for that by alen · · Score: 2

      and how many people are always changing minor things without change control because they feel this is their baby and they can do anything?

    3. Re:There is a well tested method for that by vlm · · Score: 5, Insightful

      Works, although excruciatingly slowly for planned work.
      The collision of excruciatingly slow proactive planned work, and reactive trouble tickets, always is a source of utter hilarity. Usually the end result is you only do planned proactive paper shuffling for meaningless stuff "lets change the background color to be 0.001% darker" and ram thru development as part of a trouble ticket with no oversight at all (well, to make our big customer happy, we've decided to completely redo our database schema and stored procedures this afternoon as part of the ticket).

      Another example, if it takes a month and endless meetings to replace a failing drive during scheduled maint, and a half hour to replace a failed drive at any time, this simply eliminates all proactive maintenance. Much easier / cheaper to burn the power supply out, have a nice long outage, and then replace the whole device, than to get permission to blow dust out of the air filter.

      The end result is usually much worse than it was at the beginning.

      --
      "Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
    4. Re:There is a well tested method for that by JWSmythe · · Score: 5, Interesting

      Another example, if it takes a month and endless meetings to replace a failing drive during scheduled maintenance, and a half hour to replace a failed drive at any time, ...

          Sadly enough, I've had a simple drive replacement tied up in meetings and other office politics for months. Write up a proposal for change, sit in meetings where various department heads without a clue discuss the potential hazards, write up the rollback process (for changing a drive?). Your plans are torn apart and put back together. Departmental announcements, customer notifications, etc, etc. Accounting wants numbers, and proposals from 3 sources for the cost of a replacement drive (which you have 5 of in the datacenter, and a regular supplier). You're sitting there with the mind numbing noise flowing past. All you can think is "the array was set up with no hot spare. It's running in a degraded mode. Change the damned drive." Of course, complaints of slow drive performance are scattered throughout the meeting.

          Two months and more meetings than you can remember later, they slate it for an arbitrary windows. Saturday at 3am. Not only change it, but you are required to stay while it rebuilds, "just in case...". Just in case? You have me working 8 to 7 Monday through Friday, weekends on demand (which are every weekend) AND you want me to blow off Saturday night to do the change? Ah who cares, I don't need sleep.

          Then Thursday afternoon before the schedule change is done, a second drive in the array fails, and the whole thing is down. All the same people who were in on the meetings start screaming "How could you let this happen?!"

          Thursday afternoon becomes Thursday night, and by Friday morning you have the array back up and working, through some dumb luck. (crossing fingers, praying to whatever gods may be listening, and tapping the drive with a screwdriver at boot time to make it spin up). The only planning that helped is that you keep a change of clothes and a toothbrush in the car, since you don't have time to go home once you're done. In doing the work, you notice the same thing happening to a neighboring machine. Damned aging hardware. So you just change it without the mess that accompanied the first change. Not only are you bitched out for not fixing the first array in time, but you get it twice as bad for fixing the other one before it became a problem. How could you have independent thought? How could you make a change without proper authorization?

          The only thoughts still in your head are "I hate this job", "my car keys are in my pocket, and I could just leave." Is this the day you quit? Maybe, just maybe. Just one more thing, and that'll be it. I don't need this shit.

          Friday afternoon, not sleeping since Wednesday night, you are told "Do [some other task] after hours tonight." No, you won't get paid any overtime since you're on salary. The task will take at least 8 hours, and they need it done before Saturday morning. Do you scratch out a resignation with a sharpie on the CEO's wall at 2am, or do you just walk out?

          I really hated that job.

      --
      Serious? Seriousness is well above my pay grade.
    5. Re:There is a well tested method for that by sglewis100 · · Score: 4, Interesting

      Sadly enough, I've had a simple drive replacement tied up in meetings and other office politics for months. Write up a proposal for change, sit in meetings where various department heads without a clue discuss the potential hazards, write up the rollback process (for changing a drive?).

      Not that I don't agree that some companies make change management more than it needs to be (mine does it OKAY), but I bet the guy I knew years ago who changed a drive on a RAID-5 array had thought about testing and rollback. You see, he received the replacement drive late in the day, ran into the data center, popped out a drive, popped in the new drive, and went home. Sadly, he had pulled the wrong drive.

    6. Re:There is a well tested method for that by JWSmythe · · Score: 2

      hehe.

          Sorry, I had to laugh.

          That has more to do with check your work than it does with the prolonged control processes that businesses put in place. I've seen the control processes made by committee fail miserably. Sure, they want all this stuff done. The 400 point checklist frequently misses some essential piece, like "is it the right drive?" and "verify it's rebuilding properly". The last step would have screamed "You're doing it wrong".

          If there's no clear indication of which drive to change, I tend to be very cautious. Most good arrays have a nice friendly indicator light. The rest, I take my time, and make sure I'm right before yanking anything out. That would also indicate the difference between a datacenter monkey and a good SysAdmin.

          That same place was very fond of sending replacement parts to the datacenter, for their "tech" to install. I've known a lot of datacenters, where their "tech" was the security guard. Sure, I trust him to follow simple instructions (Find the machine labeled X. Reboot it.) Sometimes even those simple instructions are screwed up. There's nothing like getting a database server rebooted in the middle of the day, when you asked for some other server to be rebooted.

          Talking the "tech" through fixing the array after they swapped the wrong drive is always entertaining. "Entertaining" roughly translates to "I've pulled out all my hair, I now want to drive up there and shoot him."

      --
      Serious? Seriousness is well above my pay grade.
    7. Re:There is a well tested method for that by AchilleTalon · · Score: 2
      All this reminds me something I was fortunate enough to see happening without being involved and responsible for all the shit that results from it.

      Once a day, we had a change to do and this change need to be coordinate with a mainframe change. We were there to test the procedure on test environment, with the test mainframe partition. In the operators' room, there is three levels of desk, each level seeing what is going on on the other level. It was very like the Star Trek Enterprise command room or something at the DoD and Pentagon, anyway, so, here was all these mainframe guys watching each other for this test on the mainframe test partition. This was preparatory work for the real thing. Then, guess what happened? They shutdown the whole mainframe production live partition of a banking transactionnal system rather than doing it with the test partition. I remind you, there was three levels of authorities involved here.

      So, this kind of thing doesn't guarantee shit won't happen, because shit happens!

      I agree that a formal procedure must be followed, but don't be fooled, this will not prevent all the shit to happen. So, don't make this procedure so cumbersome that no real work can be done within a month. Because, I have seen also this counterpart where everyone hired for their technical skills ends up filling forms, attending no-ending meetings, waiting for approvals, etc. And finally, most of the work time is consume by the maniaco-bureaucratic procedures, which do not prevent shit to happen anyway. It's just to some extend ass covering.

      The real solution would be to automate the sysadmin. But we are not there yet.

      --
      Achille Talon
      Hop!
    8. Re:There is a well tested method for that by CompMD · · Score: 2

      We do it, a major publicly traded international company with thousands of employees. Its not hard: make /etc a repository (we like mercurial), have puppet manage your servers, and revision control the server config files on the puppetmaster (again, mercurial helps here).

  3. If the single SysAdmin is even half decent.. by intellitech · · Score: 2

    /etc/sudoers will handle a majority of those "simple operations" that require root.

    --
    vos nescitis quicquam, nec cogitatis quia expedit nobis ut unus moriatur homo pro populo et non tota gens pereat.
    1. Re:If the single SysAdmin is even half decent.. by JWSmythe · · Score: 2

      And the top programs run through sudo? "sudo su" and "sudo sh" :)

          The article wasn't suggesting controls for a single admin to accomplish a task. They were talking about requiring at least 3 admins to do the same thing in three identical environments to accomplish one task.

          "Ok, we need to reboot server X, all of you on my mark type 'shutdown -r now' ... 3 ... 2 ... 1 ... mark"

          "Dammit Mark, you didn't hit enter in time. Lets try again."

      --
      Serious? Seriousness is well above my pay grade.
  4. how do they design nuclear missile systems? by circletimessquare · · Score: 5, Interesting

    look at programs where there is a lot of technical activity and communication activity for time sensitive work

    you can't have a nuclear missile system where one guy can invoke the bombs to go off. at the same time, the system has to be quick and responsive

    so you need to engineer administrative systems where not less people are involved but MORE: you can't do this function or that function without also involving this guy over there turning a key, etc.: all admin functions invoke more than one person. that's the best way to have a system where power can't be abused. its about redundancy and layers of admins, not less admins

    and if people are pursuing this question because they don't want to pay an admin or can't trust someone else with their system, then such idiots get the system they deserve: a broken one and no one willing to fix it at the money you want to pay

    --
    intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
  5. yeah by Ryanrule · · Score: 2

    well, someone has to be in charge. we arent looking to get rid of the ceo, despite their abuses.

  6. Eventually, you have to trust someone. by Rogerborg · · Score: 5, Funny

    Oh, the jobs people work at! Out west, near Hawtch-Hawtch, there's a Hawtch-Hawtcher Bee-Watcher. His job is to watch... is to keep both his eyes on the lazy town bee. A bee that is watched will work harder, you see.

    Well...he watched and he watched. But, in spite of his watch, that bee didn't work any harder. Not mawtch.

    So then somebody said, 'Our old bee-watching man just isn't bee-watching as hard as he can. He ought to be watched by another Hawtch-Hawtcher! The thing that we need is a Bee-Watcher -Watcher!'

    Well... The Bee-Watcher-Watcher watched the Bee-Watcher. He didn't watch well. So another Hawtch-Hawtcher had to come in as a Watch Watcher-Watcher!

    And today all the Hawtchers who live in Hawtch-Hawtch are watching on Watch-Watcher-Watchering-Watch, Watch-Watching the Watcher who's watching that bee.

    You're not a Hawtch-Watcher. You're lucky, you see.

    --
    If you were blocking sigs, you wouldn't have to read this.
  7. Re:respect by Anonymous Coward · · Score: 2, Insightful

    It isn't about respect, necessarily. I am a sysadmin that has the keys to a lot of things and I have wondered about this very problem. It isn't about how much respect I deserve but it would be nice to a have a distributed method in the event of some sort of catastrophe or something as simple as being sick.

  8. Re:respect by 91degrees · · Score: 2

    Your reason for respecting your sysadmin should be that he or she is a compatent capable individual who keeps the network running.

    It should not be that if you don't, then you lose control of your network.

  9. Reinventing history by vlm · · Score: 4, Interesting

    would be enormously cumbersome if they needed a consensus of administrators to execute.

    Thats why you leave changes to the 24x7 onsite operations team not one lone admin doin' his thing in the cube. They're the ones monitoring the systems, seems most sensible if they "push the buttons" on the things they watch. Ideally you have one team that does nothing but watch and one team that does nothing but do, and theoretically they cooperate.

    And besides, DAN appears still to be vaporware.

    DAN appears to be a poor reinvention of flight control software for aerospace from the 70s/80s. Those whom don't know their history are doomed to poorly repeating their past.

    Next up, we'll reinvent the concept of the security office from AS/400, or maybe the idea of hard realtime control.

    Maybe someone out there could could reinvent the concept of the watchdog timer so the "DAN" cluster doesn't go into deadlock? Naah, we'll let them "discover" it themselves, the hard way.

    --
    "Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
  10. There's a reason.. by malkavian · · Score: 3, Insightful

    That you have one person doing it. It's effective, and versatile.
    If you have multiple people empowered to do exactly the same thing, you end up at the mercy of the one that decides to shut everyone else out.
    If you then have a security admin that's the only one to be able to alter the login info, then you're at their mercy.
    With the "dual key" type approach, what's to stop someone installing a back door along with a normal software upgrade? Does everyone have the same knowledge as your prime sysop? Can you afford to have one person that completely mirrors another, instead of distributing the skills across a time (with duplication covered across the team)?
    What if both the key holders are in cahoots?

    Interestingly, who is stopping your CEO from making those really bad decisions, or your FD from siphoning the cash, or a whole host of other areas where you trust one person to do a job?

    Value the person, and make sure you treat them well enough to make it not worth their while to play you up.. Then you'll have no problem.
    Screw them over at every opportunity, and you'll really have to trust their ethical views (you're still usually safe, but it's no guarantee then).

    1. Re:There's a reason.. by lwriemen · · Score: 2

      But ... make sure you have a backup in case the person gets hit by a bus.

    2. Re:There's a reason.. by Peeteriz · · Score: 3, Insightful

      who is stopping your CEO from making those really bad decisions

      The board; other executive officers, and limitations for class of big decisions that requite a vote of shareholders; (especially in non-public companies)

      or your FD from siphoning the cash,

      Periodic independent audit, as well as requirement of extra authorisation for amounts above X - in any well managed company FD can't siphon all cash without other officers getting dirty as well;

      or a whole host of other areas where you trust one person to do a job?

      There are no other areas where high-risk issues are trusted to one person without serious oversight. In most companies the IT management and auditing is either solved as well, or the only remaining weak point with this problem - that's why the article is there.

      Valuing persons and treating them well is in no way a solution - compare 'security by obscurity' vs. 'security by goodwill' vs. 'security by prayer' and you'll find some similarities.

      Four-eyes principle stops a lot of potential malice, as the likelihood of both keyholders being ethically faulty and not betraying each other is much, much lower than simple chance of one person being ethically faulty.

      Installation of back doors along with a normal software upgrade is a prime reason why someone other than 'your prime sysop' needs to periodically verify stuff; if you don't mirror, then you ask for outside audit of stuff; have secure write-only logging of 'root' tasks to a system which is completely controlled by someone else, etc.

      Of course, it depends on the risks - if the worst your sysadmin can do is shut down an informative website that you have, then it's no big deal. If it's a payment system that can fund a life-long vacation in the Bahama's for an opportunistic administrator, then we're talking about all such measures.

  11. Re:sternobread by NevarMore · · Score: 2

    Yes. Give a team of admins root access, *SNIP!* have admins use su or sudo to achieve root access.

    Given that its trivial to implement, saves a LOT of hassle with shared passwords/keys, using su/sudo should be the default case rather than the special needs case.

  12. There are Safeguards Already by BooRadley · · Score: 5, Insightful

    Mostly, except in very small organizations, there are several implicit safeguards to keep any one person from doing evil with the systems. They are subtle, but effective.

    Peer review: Most sysadmins are hired by other sysadmins, or at the very least a technical manager. This means that you are hired based on your skills, reputation, track record, and demonstrated attitude. This means that ideally, you wouldn't even *think* about intentionally subverting a system, because that would mean breaking it or compromising it in some way, and most professional SA'a are simply too OCD to allow it.

    Business continuity: Most organizations have several layers of continuity in place, such as disaster recovery scenarios, system snapshots, monitoring, and auditing. This means that unless you are VERY subtle, or work for an entirely incompetent team, you WILL get caught, and the damage will be minimized as you are being put into a police car, never to work in IT again.

    There are no "indispensable people:" If you are a sysadmin, and you are the only one who knows your systems, you have not done your job. Every system and app should be documented, and there should be accountability for every change and decision.

    No technical solution will ever replace good management and planning, and a design that eliminates the vulnerabilities of a system to rogue sysadmins, will also eliminate its flexibility. It's just a lot cheaper and easier to try and run a good shop.

    --

    -- lk t lv ll th vwls t f wrds. T svs lts f tm t wrt bt ts pn n th ss t rd nd mks m lk lk cmplt dpsht.

  13. Re:sternobread by goofy183 · · Score: 4, Interesting

    That is how all of our servers are setup. I'm just a "developer" that uses them but I believe no one knows the root password for our systems. It is a *big* random string that is printed out by the sysadmin that sets up the machine, sealed in an envelope with that person's signature on both sides and stuck in a safe. In the event that a machine is so hosed that the root password is needed it is used and then a new one is generated and sealed away again.

    Everyone uses sudo for everything. All sudo access is logged.

    The system isn't perfect of course, nothing is, but it goes a long way to the worry of one person having root keys for things.

  14. You need at least TWO good sysadmins... by DigitalSorceress · · Score: 2

    Hire admins who know their stuff and make sure you have at least two of them with the root password. Make sure they've got some kind of change control in place, and make sure you have them document what they're doing.

    I've been the sole sysadmin before, and I always felt worried that my legacy, should I be fired or quit or hit by a bus, would be "She didn't do a great job because everything fell apart after she left/was fired/was bussassinated". So, I always tried to document things and made sure the boss had the "keys to the kingdom" (document with root pw and locations of my documentation to give to my successor).

    --

    The Digital Sorceress
    1. Re:You need at least TWO good sysadmins... by Anonymous Coward · · Score: 2, Funny

      bussassinated

      I have a new word of the day! Thank you. :D

    2. Re:You need at least TWO good sysadmins... by JustOK · · Score: 2

      bussassinated has 2 possible meanings
      1) killed by bus
      2) killed by business

      --
      rewriting history since 2109
  15. It's not your only line of defense by plover · · Score: 3, Interesting

    First, understand that Slashdot is only going to provide a hint of what you will be doing. Security is complex and easy to get wrong, and there's a whole lot of evidence of that in the news. If security is important to your company, you should invest in a CISSP to really help you get things set up in a fashion that the industry considers to be best practices. Until then, consider these few generic suggestions.

    Multiple layers of security help ensure that nothing goes astray, or if it does that it's detected before too much damage is done. And separation of duties helps make sure that one rogue actor can't do it all by himself.

    Separate the admin of the box from the admin of the data. The guy who holds the root PW doesn't have to be the same guy who holds the private key for the database.

    Add off-the-box auditing to the actions of root. As soon as someone signs on as root, notification is sent to a different box of the originating IP and it's timestamped. Don't let your application sysadmin be the sysadmin of the audit box! And the auditor should investigate carefully any situations that are out of the ordinary. (This box fell off the network just before root logged on? That's an odd coincidence.)

    Define expected behavior with policies. If you want to run a trustworthy ship, clearly stating who has access to do what with which systems eliminates confusion, and helps avoid where one sysadmin creeps over into other systems.

    Ultimately, you've placed trust your admin to do a job, and you need to trust him or her to do that job. Somebody's got to be root. But they also have to know they'll be held accountable for what they do.

    --
    John
  16. Powerbroker & logging by Doc+Hopper · · Score: 4, Informative

    We have several solutions which work together to minimize the risk of root at my company:

    1. Powerbroker. It's in use on every single UNIX system administered by our Global IT teams. Every user has a role (or several roles), and that allows them to execute a variety of commands with elevated privileges. Once Powerbroker is invoked, however, every single keystroke is logged and can be played back. These logs are stored indefinitely; access is very restricted.

    2. Automated, centralized root password management. One of the steps to setting up a UNIX machine here is ensuring the root password and remote console admin passwords match that dictated by our automated provisioning system. Then every 30-90 days (depending on policy for this type of system) the root password is changed to a very long, apparently very random string. I can look this password up if my role allows it, but the lookup is also logged.

    3. A good Change Request (CR) process. Every system that exists in a data center should have a record in our systems database. Once a system has passed through the phases of deployment (Warehouse -> Data Center Install -> Sysadm Configure -> Deployed) any change made to the system must be requested and approved by the owners of the system. This approval is logged, and the date/time of the work is also logged. Sysadms must close service requests within the time window specified by the CR, or apply for an extension or reschedule if they're unable to complete it within the allotted time.
        The downside to this is that you lose quite a bit of system administrator work hours filing and managing change requests. However, this loss of efficiency -- IMHO -- is better than the mayhem that ensues without an organized change process.

    4. Automated forensic tools to monitor changes. Information overload is a real risk with any Tripwire-style system, though. We're still working out some of the kinks on this part of the system. Once we ensure that all normal changes due to operation of the system and scheduled maintenance get excluded, this will be the fourth leg to reduce the risk of super-user privileges.

    At any company, IT must find a balance between controlling user actions and monitoring those actions. In most cases, the easiest approach is to prohibit by policy only those things that might typically result in lawsuits, but monitor everything else to the best of your ability. Combining a Powerbroker-like product with automated root password management -- both with fascistic logging -- is a reasonable approach that works well for many large companies. Combine this with a change management system, and a forensic tool to automatically monitor and notify of unauthorized changes, and super-user isn't really all that big of a concern.

    1. Re:Powerbroker & logging by phyrexianshaw.ca · · Score: 2

      These logs are stored indefinitely; access is very restricted.

      to whom? what you have to keep in mind is that computers operate as single minded entities. when you approach a machine like that: security is currently an afterthought. this tells me that there is somebody that holds access above the other users, basically missing the point here.

      I can look this password up if my role allows it, but the lookup is also logged

      Again, that means that there's somebody administering the logging system. and I almost assure you that even if their logins are listed somewhere: they have full access to remove those entries and make it look like it never happened.

      as a hypothetical situation, say I have a machine that stores credit card numbers on a DSS approved network that's locked down in the ways you describe above. at the admin level, it would take me minutes to provision a machine to replicate the target. I don't mean replicate as in contents, I mean replicate to the network view.

      the replicated machine can be tunneled into place and act as if it was the machine in question. as the admin: I already know what traffic flows the machine needs to produce on a regular basis (SNMP uptime's, network traffic counters, heartbeats, etc) so I can inject artificial traffic in it's place.

      at this point, I can reverse firewall the unit preventing it for calling for help or reporting the changes I make. I can snapshot the drive and move it offsite, while making the changes to the snapshot to remove my presence from the machine and set the loader to write over itself with the snap. reboot into the snap and pull the zombie as the machine comes back up:

      and what will the monitoring/auditing/reporting software see? nothing. everything will check out, MAC addresses will match, SNMP keys will match, even the statistics reported will look like they fit into the graphs.

      Until CPU's are made to understand the "two key" approach to authentication, any machine will be susceptible to weak physical security.

    2. Re:Powerbroker & logging by Doc+Hopper · · Score: 4, Informative

      You've tossed out a few red herrings and a couple of valid points. I'll try to address them in order.

      this tells me that there is somebody that holds access above the other users, basically missing the point here.

      No, I haven't missed the point at all. The point is to distribute the responsibility with sufficient checks in order to ensure that misbehavior will be caught and dealt with in a timely fashion. Is it possible someone could scheme up a way to slide abuses past the admins? Of course it is. But between good backups, fascistic logging, role-based access control, and routine audits by the change control committee, the risk is minimized.

      There's no one person who holds the "keys to the kingdom". No critical data is stored on the machines themselves; it's all stored on centralized storage. The folks who admin the automated root password changes don't have any access to storage; the storage folks typically don't have any access to the systems.

      Again, that means that there's somebody administering the logging system. and I almost assure you that even if their logins are listed somewhere: they have full access to remove those entries and make it look like it never happened.

      Incorrect. I didn't cover this in my original post, but logs are (and should be) stored on write-once media. You can designate volumes on modern storage media so that, once written, it can never be altered without destroying the entire volume. We use this extensively.

      say I have a machine that stores credit card numbers on a DSS approved network that's locked down in the ways you describe above. at the admin level, it would take me minutes to provision a machine to replicate the target. I don't mean replicate as in contents, I mean replicate to the network view.

      Once again, distributed access can prevent this. The network team and the sysadm team aren't the same teams. Every port on your switch is disabled until it's enabled by the network team. Even once enabled, that port must be on the same VLAN as the hypothetical credit-card storage system.

      That's once again where fascistic logging and automated reporting come into play. If a port is disconnected, unless a host has been blacked out with an appropriate change control ticket filed, the port disconnection generates an immediate Priority 1 service request to investigate.

      If a drive is removed from centralized storage, that also generates an immediate P1 ticket. The sysadm's access would have been logged the moment he swiped his badge, and cameras throughout the data center capture the switch-over.

      A corrupt admin can do a lot of damage, I admit. There's no getting around it. But with sufficient logging -- and yes, I include physical surveillance as "logging" too -- they're not going to get away with it.

      the replicated machine can be tunneled into place and act as if it was the machine in question.

      Now this is the red herring. If you've ever done ANYTHING major with credit cards in a data center, you are aware that you're subject to yearly audits of your infrastructure by Payment Services. They do a deep-dive of your systems to enforce a huge number of requirements. I can't go into it here. It literally fills a large book, and they go over it line-by-line with all the admins involved, every single year. I've been through several of these, and each year it gets broadened to cover more potential issues.

      Chief among these requirements? A separate admin/management network from the front-end/back-end network. You can't "tunnel in" to that network and make it "act like" another system. The network is an unroutable private VLAN or fibre-channel connection.

      at this point, I can reverse firewall the unit preventing it for calling for help or reporting the changes I make. I can snapshot the drive and move it offsite

      Ye

  17. The eternal efficiency/polarization trade-off by OneAhead · · Score: 2

    It will always be a trade-off between efficiency and polarization of power. Just like in politics. Decision-making is extremely efficient in an absolute dictatorship, but people have to rely on the dictator being benevolent, staying benevolent, staying healthy (I've seen people's character completely change because of illness), and having benevolent successors (this almost never happens in real-life politics). On the other side of the spectrum, you have democracies with a proper implementation of Montesqieu's separation of powers, which in some instances get plagued by indecisiveness, gridlock, and half-assed compromises that really are not helpful for anyone. (*) You'll find the same principles apply to system administration, or anything that involves power.

    (*)<opinion>Obamacare is a perfect example: started off as a radical reform that would allow America to take back its place amongst civilized nations, was watered down to a legislative abomination. Note that I'm not implying that the president has too little power - this is merely an example of one of the side-effects of the democratic system America is using. I can give some examples of presidential abuse of power as well.</opinion>

  18. Driving a car by petes_PoV · · Score: 2
    Trying to get 2 sysadmins to cooperate would be like insisting every car has 2 drivers (and not like a plane has a copilot). There are at least four possible outcomes: one sysadmin becomes dominant and you're back to where you started, but paying two salaries. They continually bicker about the best way to do things and nothing ever gets done (or worse: they sabotage each others' efforts), one just slacks off and causes decision-making bottlenecks or they spend so long reaching a consensus that even the most trivial task takes a week of decision making, timetabling, agreeing and finally doing it.

    The only solution I can think of that would stand a chance is to require:
    a) everything gets documented (you'll know this is the correct way, as all the techies will hate it)
    b.) every week / month all the roles change, if an admin coming into a role finds that things aren't as they were documented, someone gets yelled at
    This also has the advantage that you're no longer completely screwed if someone leaves, goes sick or gets promoted. it also makes it clear to the people in question that the company can get along quite nicely without them.

    --
    politicians are like babies' nappies: they should both be changed regularly and for the same reasons
  19. How about by 0racle · · Score: 3, Insightful

    Everyone treats everyone else like adults and every one acts like an adult? Honestly, if you don't trust your admins, why are they your admins?

    Also, simple change management alleviates most of these problems. Even if it's just a log for what happened so that the next shift or your colleague tomorrow knows what you did today. Then again, I guess that is really back to acting like adults.

    --
    "I use a Mac because I'm just better than you are."
  20. Re:why? by somersault · · Score: 5, Insightful

    Not really. It's fun to think I could do anything I wanted, but I don't want to. I like my job, I like the people I work with, I don't want to screw them over. It's nice to have an employer that trusts you too. If I wasn't trusted, I would probably just leave. If they want me to be able to administer and troubleshoot everything, I obviously need full access.

    --
    which is totally what she said
  21. The Orange Book solution by McMuffin+Man · · Score: 3, Interesting

    This is an old problem in high assurance systems. As other posters have pointed out, as some point you have to trust someone. But you can still "trust but verify".

    The standard solution is "division of privilege". Over time folks have learned that the key is a system which audits everything the admin does, and the one thing the admin can't do is modify or delete the audit trail. A separate person or team has the role of auditor.

    This is one of the requirements of a B2 level system in the old Orange Book model, and you'll see if it as a requirement if you need to provide systems for most countries' military or intelligence organizations. It's rarely used elsewhere because more or less noone else is willing to pay the staffing costs. The solution there is trust someone, and be ready to fire, sue, and/or prosecute if they violate that trust.

    1. Re:The Orange Book solution by Animats · · Score: 3, Informative

      This is one of the requirements of a B2 level system in the old Orange Book model, and you'll see if it as a requirement if you need to provide systems for most countries' military or intelligence organizations. It's rarely used elsewhere because more or less noone else is willing to pay the staffing costs.

      Right. I developed an OS for that model many years ago.

      The key to this is a mandatory security/integrity model. At a given privilege level, you can only run programs trusted at that privilege level. So, if you're running as some kind of administrator, you can only run trusted administrator tools. You can't use a text editor on the password file, for example.

      Then you have compartments, and some tools are accessible only in some compartments. For example, the person or program that makes backups needs the ability to read almost everything, but to write almost nothing. (Restoring from backups, which is done less often, requires different privileges.) The security officer can add and delete users, but can't install programs. All this is enforced by the OS, looking at privileges associated with files, users, and programs, not by the applications themselves. A few applications are trusted, and they have to go through an elaborate approval process, which means they're usually rather dumb apps.

      The "control panels" used by hosting services are a step in this direction. Users can do some things, and first-line tech support people can do others.

      Currently, the big hole is program installation. Installers typically demand far more privileges than they should. In a mandatory security model, installation of an ordinary "application" should mean that the installer has write permission for the vendor's compartment and nothing else.

  22. Re:Yes by ByOhTek · · Score: 4, Interesting

    A subset of administrative applications requiring multiple administrators may not be such a bad compromise.

    ex:
    * change root password (or password to any "wheel" account) - requires multiple administrators to enter the same passwords
    *su/sudo'ing to a "wheel" account, or changing said account's privileges, requires the authorization of at least one other wheel'ed user.
    * Alterning an active network interface, shutting down, and restarting requires authorization by other administrative users.

    Stuff like that, which are things that shouldn't be done often, anyway, and could allow one admin to take over the whole system, seem like good candidates for multiple-approvals. Everything else could be left alone.

    The approval process is basically - the root users needs to take the action, and then 2+ non-root (but wheel) users must approve it.

    I'm using 'wheel' as that is the group in FreeBSD that is typically allowed access to sudo/su. Not sure how other systems typically work.

    --
    Self proclaimed typo king, and inventor of the bear destroying coffee table (patent not pending).
  23. Smack * by onyxruby · · Score: 3, Insightful

    Peer Review, Change Control, Auditing, Maintenance Windows, Testing all changes in a lab before production, source and version control / maintenance. These are all best practices, work regardless of operating system and don't require any special software.

    Why o why do you want to use software to take the place of established best practices? Best practices are there for good reasons, and those reasons usually have multi-million dollar lessons attached to them. You don't need special software, just a heavy that says yes you /must/ do it this way and raises hell when you try otherwise...

  24. Re:why? by phyrexianshaw.ca · · Score: 3, Insightful

    This.

    if you can't trust the person at the top: then either they don't deserve to be there, or you need to find a new job.

    when you're the person at the top: you better have earned the trust and respect of those under you. Subverting it does nobody any good in any long term.

  25. Re:respect by GNUALMAFUERTE · · Score: 3, Insightful

    You keep your passwords in a network share? Are you schizophrenic or just incompetent?

    I hope that file is fucking well encrypted ... but even in that case, it's just a bad idea.

    --
    WTF am I doing replying to an AC at 5 A.M on a Friday night?
  26. Re:sternobread by s4ltyd0g · · Score: 4, Informative

    sudo logs are almost useless for system audit. Run sudo su - and have at it. There are no logs to follow what actions you perform. Go ahead and craft a sudoers file that eliminates all the ways to load up a shell. Have fun with that...

  27. Re:respect by Minwee · · Score: 2

    I typically keep that kind of information written down and sealed inside a plain white envelope labelled "Plain White Envelope" in my handwriting and placed in a secure location. If anything happens and someone needs access all they need to do is open that up and use the login information they find inside.

    If the envelope is ever opened and I still work there then I need to do a security audit and change all of the passwords. If I don't work there any more then either I have been hit by a bus, or my manager has done something unimaginably stupid like letting me go and either way it's no longer my problem.

    That helps me feel more comfortable about the business and if my replacement can't figure out how to use what I have left for him or her then I can be secure in the knowledge that the problem is with the hiring process and not my documentation.

  28. Re:Too many cooks... by JustOK · · Score: 3, Funny

    fine, no soup. just type sudo make me a sandwich

    --
    rewriting history since 2109
  29. Re:Yes by jijacob · · Score: 5, Insightful

    If you don't trust your sysadmin, they shouldn't be your sysadmin. Just like the accounting department probably has the ability to steal a certain sum of money before anyone will notice, your sysadmin is given responsibilities that could potentially cause grief if they are on the wrong team.

  30. Re:sternobread by Phreakiture · · Score: 5, Insightful

    Run sudo su - and have at it.

    The solution here is to follow a reasonable security protocol in writing the sudoers file. Specifically, the default action is to prohibit. Permitted actions are then whitelisted. On a high-security system, no entry should allow a user to sudo su -. Problem solved.

    Incidentally, I see no point in locking down users who have physical access to the DC.

    --
    www.wavefront-av.com
  31. Re:Answer in kdawson's tagline by rwa2 · · Score: 3, Interesting

    "trust but verify"

    To get some transparency / accountability, just set up an authlog black hole that includes all of the sudo activity from your servers.

  32. Re:why? by StuartHankins · · Score: 3, Insightful

    Yep. And a single malicious incident could end my career. A career I've spent many decades and countless hours on. There's no way I'd risk it. And that's assuming that my morals would allow me to seriously consider jeopardizing it in the first place.

    Obviously there are those with different goals and standards and it's not always easy to identify them. I'm not sure how to prevent that -- someone who over the years gradually gets more access and one day they decide to go rogue and do something harmful. Even minimizing the attack surface you usually have that single admin account that owns everything else. Maybe I should read the article.

  33. Re:Yes by jijacob · · Score: 4, Insightful

    The tricky part comes in at the point that, while most CEOs have at least a basic understanding of accounting and other departments under their watch, IT departments are *typically* a foreign land to the understanding of those in charge. Even if they wanted to audit proper usage of root it would be difficult or impossible. Small businesses have it hardest. At least in the larger ones there's a layering system so you can have higher-ups in IT auditing the lower guys.

  34. Re:Yes by ByOhTek · · Score: 2

    I disagree. You can instead trust some /people/ with proper checks and balances. This can, in some situations, reduce the risk (for example, if more than one is required for approval of certain things)

    --
    Self proclaimed typo king, and inventor of the bear destroying coffee table (patent not pending).
  35. Re:Too many cooks... by BrokenHalo · · Score: 5, Interesting

    ...spoil the soup.

    The submission seems to presume that the system in question is some sort of *nix or Windows box. If we look into the world of mainframe operating systems, we'll see that this has already been fully adressed, and any number of individuals with discrete UIDs may have superuser access. This has evolved out of a history where sysadmins worked shifts, so sharing a single privileged UID/password was/is a bad idea.

    The way such access is administrated needs a proper policy within the organisation, though. Back in the '90s, I worked at one outfit (an insurance company) where the vice-CEO demanded superuser privileges despite having no knowledge of system administration or any other computing background. He just wanted to act as overlord as to what staff had access to on their signons. I was very tempted to tell him to get fucked, phrased in more professional terms. Like "Go get professionally fucked".

    My immediate boss was (wisely) more inclined to a diplomatic approach, however, so he pursuaded me to install a dummy program for him that was enough to convince him that he had what he wanted, without granting him any kind of command line access, or ability to change system configuration.

  36. Re:sternobread by sglewis100 · · Score: 3, Funny

    Well, there's /root/.bash_history

    But if your sudo activity log has you doing "su -", then whatever gets borked up after that is automagically your fault as a matter of policy ^_^

    Yeah, nobody's ever altered that file. Also, make sure you are watching for changes to your syslogd config, lest someone disable forwarding, do something snarky, turn it back on. But then, security is rarely something that can be solved definitively by means of one slashdot comment.

  37. Re:Too many cooks... by intellitech · · Score: 2

    $ sudo make sandwitch
    sandwich: target not found

    Was pretty funny until I realized you typed it in by hand. Too bad you misspelled sandwich.. ;)

    --
    vos nescitis quicquam, nec cogitatis quia expedit nobis ut unus moriatur homo pro populo et non tota gens pereat.
  38. My proposal by HikingStick · · Score: 2

    As a one-person IT department, I made a recommendation to management, reflecting a practice that is used in some other high-trust industries, like banking: audit me.

    Really. Give me a couple of paid weeks off each year, and have our auditing firm come in and look at the logs, my access, and the security model. Not only would it help the company feel good about the network controls and their network administrator (yours truly), but it would also give me a couple of weeks off without being hounded every day for me to fix something or other, as usually happens on my days off.

    --
    I use irony whenever I can, but my shirts are still wrinkled...
  39. Re:Yes by Dancindan84 · · Score: 2

    The concept is sound, but in practice the first time there's an emergency where something in the subset needs to be done and 2+ admins are required causing even a small delay, the PHBs will toss it out the window (and not be entirely wrong in doing so). There's always a trade off for greater security/accountability, and IMHO this will cross the line of what's acceptable to management often enough that it won't happen broadly.

    --
    "Always forgive your enemies; nothing annoys them so much." - Oscar Wilde
  40. Re:sternobread by Sancho · · Score: 2

    Why do you think that /bin/bash would be whitelisted?

    That said, getting this kind of security is fairly tough because you have to ensure that any utilities can't escape to shell or open files that would in turn allow circumvention. For example, if vim is whitelisted, you can :shell. That can be disabled as a compile-time option. But :r /usr/local/etc/sudoers will allow the person running vim as root to modify sudoers. I don't recall if :r can be disabled, because it's mostly irrelevant--you can modify the contents of the buffer and :w! /usr/local/etc/sudoers

    SELinux (or equivalent) is really required to be absolutely sure. Of course, you still do the sudo whitelist, because you want to do these things in layers.

  41. Re:Yes by Red+Flayer · · Score: 2

    Good point you make there.

    I think there are gaps in management knowledge for most small companies, so they outsource it. Basic accounting is near universal, but tax, for example, is typically outsourced for small companies. Tax prep, however, is via an accredited institution most of the time.

    So for IT, do we turn to accreditation of outside providers? Or do we wait a couple generations until basic knowledge of IT is assumed necessary for non-CIO CxOs?

    --
    "Trolls they were, but filled with the evil will of their master: a fell race..." -- J.R.R. Tolkien on Olog-hai
  42. Re:sternobread by hawguy · · Score: 2

    Unless you are very careful with what commands an admin can run with sudo, there are many ways for him to run a command without it appearing in the sudo log:

    sudo vi /etc/hosts
    :sh

    Now I'm in a root shell and sudo doesn't know anything about it.

  43. Re:sternobread by s4ltyd0g · · Score: 2

    With a team of administrators, you'll have no way of learning for certain who has done what. As you said sudo su - is only one of the many trivial ways. Discretionary access controls as you have described are no better than trusting your admins with the real root password and telling them if you abuse the power you will be fired. At that point, why bother? It's just gonna eat up budget to implement and you are still stuck with the same problem which is accountability. That is to say, who has done what, where and in which manner.

    regards

  44. Re:Yes by Toe,+The · · Score: 2

    That is precisely the point of the original question.

    We trust politicians with our governance, and over and over and over again, they violate that trust.

    Collaborative governance is a way to remove the need for politicians. But it is pointless if we just shift the trust over to sysadmins. They are just as susceptible to corruption as politicians.

    We are looking for a way to remove the need for trust in governance: of governments (and any other kind of administration) and of the systems that run them.

  45. Re:sternobread by alcourt · · Score: 2

    At work, I worry about these and other issues on a very large scale.

    The logging shell must be non-blocking. Will it capture everything? No. If it logs every keystroke, it fails. If it logs everything sent to your screen, it fails. The reason why is left as an exercise for the reader.

    The audit requirements have been satisfied for our auditors with command line history logging done in a reliable manner (reliable as in the data once captured is not alterable by the user).

    As for X11 only configs, I tell the vendor they can implement a command line or they can lose a sale at a very large company. Maybe I've been lucky. But so far, it's worked.

    The key is that the end user must not be greatly inconvenienced by the logging so that they do not feel like there is a reason to go around it. That's why I like the ksh-93 with auditing approach. It's a comfortable and familiar shell, but it logs their command line history in real time off-host. (Built into ksh-93, but not enabled in a default build).

    What I look for isn't necessarily what they did after they took actions to evade logging, but did an action occur that could reasonably allow them to bypass logging?

    As a nice side effect, some practices that were bad before are no longer allowed. Don't edit that file in place, check out a copy from your repository into place. Minimize the work done as root.

    --
    "I may disagree with what you say, but I will defend unto the death your right to say it." -- Voltaire
  46. Superuser by fyngyrz · · Score: 4, Funny

    Are there more sweeping yet practical solutions out there for avoiding the weakness of a singular empowered superuser?"

    No. Now just hang on a second while I delete your user account and all your data, you presumptuous bitch.

    --
    I've fallen off your lawn, and I can't get up.
  47. Re:Depends on what kind of people you want to attr by Doc+Hopper · · Score: 2

    A very talented, and very honest person will not put up with layers of approvals and constant monitoring.

    Have you actually worked at a company composed of very talented, very honest people who put up with this very thing every day? Setting up an ITIL-compliant change management system -- and getting everybody on-board with using it -- is a very daunting procedure. Speaking as someone who has been on both ends of this, I can say that in the end, it's worth it. My day as a sysadmin is no longer all about putting out fire after fire, dashing around and pulling crazy hours at the whims of vice-presidents who think the latest thing they heard about is the "highest priority".

    Work is scheduled, executed, and followed-up on. It's not perfect, but requiring approval from the stakeholders prior to making changes has been a HUGE improvement in my quality-of-life and that of my fellow admins. The principal cost is manpower: I spend far more time managing changes than I used to.

    It bugs me when people equate "managing change" with "being a bureaucrat". The process of getting to the point that we could manage and track all our changes was a royal pain in the butt. Five years later, even if you factor in the time-cost of documenting all changes to all systems, we're running more efficiently than before.

    I've played the game of being the chief sysadmin in a startup before. Heroic effort, hectic schedules, and obscene hours at low pay aren't what I'm interested in anymore. And truthfully, the average quality of sysadmin I work with in a Fortune 500 company is head-and-shoulders above the admins I worked with before. There's a minimum standard we expect, and if you can't hack it, you're out.

    That minimum standard includes knowing that any change you make as root on a system will be monitored, catalogued, and may be subject to a later Root Cause Analysis (RCA). Learn to behave ethically, carefully, and competently at all times when you're working as root, and it's no big deal. In truth, I'm GLAD for the monitoring. During any audit -- and I've been through many! -- I can point to the service request number, change request number, and approvals for the work I performed. It's good pay, my ass is always covered when I make important changes, and I get to work on machines in a data center that makes the warehouse at the end of Raiders of the Lost Ark look small. What's not to love?

  48. Re:sternobread by ahodgson · · Score: 3, Insightful

    I think you're missing the point. Auditing/logging systems are not meant to provide effective defense. They are meant to let PHB's mark appropriate check boxes on compliance forms and sleep better without worrying what those evil nasty sysadmins are doing. Don't confuse them.

  49. Re:Yes by Lumpy · · Score: 2

    You nailed it on the head.

    Companies like accountability. If the thing blew up, they want someone to fire.

    with 10+ admins, you cant point a finger in a heat of passion and say "Escort him from the building!", it would take weeks to figure out what happened, and if the 10+ admins were wise they would cover each others asses.

    --
    Do not look at laser with remaining good eye.
  50. Different Objectives for Dividing Responsibility by billstewart · · Score: 2

    There are a number of different objectives people might have for splitting up superuser powers, and depending on what you're trying to accomplish, there are different kinds of solutions out there. For instance

    • You don't trust unwatched individuals acting alone with the power to do stuff and change logfiles to cover their tracks? - That sounds like what you're asking here? But is that what you really want?
    • You trust your individual superuser as a person, but don't trust his remote access environment not to be hacked, compromised by the cross-site-scripting-virus-wireless-keyboard-logger of the month? - You might be insufficiently paranoid, or you might have much more serious problems than you think?
    • You want multiple people to have sysadmin capabilities so you can get stuff done even if the main lead isn't around or it's night shift? That's what wheel group or equivalent is for.
    • Having One Super Root is too powerful, and you want to split up the different admin functions? There have been a bunch of Secure Unix projects, typically B2 or B3 Orange Book things funded by the NSA, which do that.
    • Root is more powerful than most admins need most of the time, so you'd like to be able to split up admin functions without giving everybody root. - Sometimes setuid and sudo are enough, and the traditional Unix approaches of having different admin user ids (lp/lpr, mail, etc.) worked relatively well but need a bit of help on TCP/IP permissions, and there have been various projects to build tools like that.

    You really need to nail down your problems and objectives carefully before looking for a solution. Security can really improve your operations if it matches your goals, but it can also really interfere with work if it's preventing you from doing things you need, whether that's directly blocking appropriate actions or whether it's by making you use a Linux distribution that's inappropriate because it's the only one with your required security buzzwords all checked.

    --

    Bill Stewart
    New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks