Slashdot Mirror
Disempowering the Singular Sysadmin?
An anonymous reader writes "Practically every computer system appears to be at the mercy of at least one individual who holds root (or whatever other superuser identity can destroy or subvert that system). However, making a system require multiple individuals for any root operation (think of the classic two-key process to launch a nuke) has shortcomings: simple operations sometimes require root, and would be enormously cumbersome if they needed a consensus of administrators to execute. There is the idea of a Distributed Administration Network, which is like a cluster of independently administered servers, but this is a limited case for deployment of certain applications. And besides, DAN appears still to be vaporware. Are there more sweeping yet practical solutions out there for avoiding the weakness of a singular empowered superuser?"
...spoil the soup.
I have been wondering the same thing.
Hebrews 11:8
Jeremiah 33:3
does anyone think about the abuse potential anymore?
Yes. Give a team of admins root access. This is common practice. If you need additional auditablilty, deny direct login as root, and have admins use su or sudo to achieve root access.
Maybe instead of worrying about your sysadmin, you should treat him/her with respect.
Try and remember how valuable they are when your stuff is NOT broken and remember THEY keep it humming along day after day after day........
We just have an account in Active Directory called "Support" with Administrative Rights across the domain, and the Sysadmin holds the Root and Administrative Passwords to effectively hold total control if need be. He can change the Support Password and lock all of us out if he wishes, or give us the info to let us back in. But pretty much anything that needs to be done, we can do on that account, including adding the PC to the domain itself.
Or am I going to be laughed at for posting the Microsoft answer?
Rule by a benevolent dictator has certain advantages, and rule by committee has certain opposite advantages. It was ever thus.
we have a few databases where selected developers can do anything they want since they do most of the work there and there is no SOX requirement for those databases. every week mysterious things happen where column schemas are changed, stored procedures are updated, etc with no notification to anyone except when trouble tickets come in because some other application broke
What is this bullshit. Make simple common operations NOT REQUIRE ROOT. No, I know, LET'S ADD MORE MANAGEMENT.
It is called: "Change Control" and usually goes along with "Revision Control" on configs.
If you change without recording the reason for change and without checking in the result so that the two versions can be compared and analysed you get a pink slip. Voila. Problem solved.
Baker's Law: Misery no longer loves company. Nowadays it insists on it
http://www.sigsegv.cx/
You can use sudo to grant privileges to run the commands for day-to-day administration to certain individuals with user level accounts. This will allow root level privileges that are logged for only specific commands.
You can then have a 'proxy' account with the ability to sudo to full root with its password locked away that requires some sort of management approval (perhaps through change control?) to access it. Then, if you have a system go tits up, you can get that password from the safe or bosses office to fix it.
/etc/sudoers will handle a majority of those "simple operations" that require root.
vos nescitis quicquam, nec cogitatis quia expedit nobis ut unus moriatur homo pro populo et non tota gens pereat.
look at programs where there is a lot of technical activity and communication activity for time sensitive work
you can't have a nuclear missile system where one guy can invoke the bombs to go off. at the same time, the system has to be quick and responsive
so you need to engineer administrative systems where not less people are involved but MORE: you can't do this function or that function without also involving this guy over there turning a key, etc.: all admin functions invoke more than one person. that's the best way to have a system where power can't be abused. its about redundancy and layers of admins, not less admins
and if people are pursuing this question because they don't want to pay an admin or can't trust someone else with their system, then such idiots get the system they deserve: a broken one and no one willing to fix it at the money you want to pay
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
well, someone has to be in charge. we arent looking to get rid of the ceo, despite their abuses.
Oh, the jobs people work at! Out west, near Hawtch-Hawtch, there's a Hawtch-Hawtcher Bee-Watcher. His job is to watch... is to keep both his eyes on the lazy town bee. A bee that is watched will work harder, you see.
Well...he watched and he watched. But, in spite of his watch, that bee didn't work any harder. Not mawtch.
So then somebody said, 'Our old bee-watching man just isn't bee-watching as hard as he can. He ought to be watched by another Hawtch-Hawtcher! The thing that we need is a Bee-Watcher -Watcher!'
Well... The Bee-Watcher-Watcher watched the Bee-Watcher. He didn't watch well. So another Hawtch-Hawtcher had to come in as a Watch Watcher-Watcher!
And today all the Hawtchers who live in Hawtch-Hawtch are watching on Watch-Watcher-Watchering-Watch, Watch-Watching the Watcher who's watching that bee.
You're not a Hawtch-Watcher. You're lucky, you see.
If you were blocking sigs, you wouldn't have to read this.
would be enormously cumbersome if they needed a consensus of administrators to execute.
Thats why you leave changes to the 24x7 onsite operations team not one lone admin doin' his thing in the cube. They're the ones monitoring the systems, seems most sensible if they "push the buttons" on the things they watch. Ideally you have one team that does nothing but watch and one team that does nothing but do, and theoretically they cooperate.
And besides, DAN appears still to be vaporware.
DAN appears to be a poor reinvention of flight control software for aerospace from the 70s/80s. Those whom don't know their history are doomed to poorly repeating their past.
Next up, we'll reinvent the concept of the security office from AS/400, or maybe the idea of hard realtime control.
Maybe someone out there could could reinvent the concept of the watchdog timer so the "DAN" cluster doesn't go into deadlock? Naah, we'll let them "discover" it themselves, the hard way.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
They are only empowered on certain systems - mail admin doesn't have root on db for example.
Generally a contract (that you have anyway), an external audit trail (that you should have anyway and that they don't have access to) plus a few lawyers pretty much covers it.
Also when process gets in the way of business business usually wins - passwords get written down, lending hardware tokens to others etc...
That you have one person doing it. It's effective, and versatile.
If you have multiple people empowered to do exactly the same thing, you end up at the mercy of the one that decides to shut everyone else out.
If you then have a security admin that's the only one to be able to alter the login info, then you're at their mercy.
With the "dual key" type approach, what's to stop someone installing a back door along with a normal software upgrade? Does everyone have the same knowledge as your prime sysop? Can you afford to have one person that completely mirrors another, instead of distributing the skills across a time (with duplication covered across the team)?
What if both the key holders are in cahoots?
Interestingly, who is stopping your CEO from making those really bad decisions, or your FD from siphoning the cash, or a whole host of other areas where you trust one person to do a job?
Value the person, and make sure you treat them well enough to make it not worth their while to play you up.. Then you'll have no problem.
Screw them over at every opportunity, and you'll really have to trust their ethical views (you're still usually safe, but it's no guarantee then).
Disemboweling the Cingular Sysadmin?? Huh?!
>heads on in search of caffeine...
Really though if they have physical access to something they can do whatever they like. Auditing and logs can go pretty far but at some point you have to trust the people that run things.
No sir I dont like it.
You need to decide on the access the person needs when they are hired. If you are getting a system administrator, then make sure you can trust them from the beginning. Even if you figure out a way to prevent any one individual from being the sole point root admin on your system, you still have a brilliant person well versed in how that security is set up that can still subvert it. The key is to trust your root admins and hope they are doing the right thing; the other option is to not trust them, which will cause stress and animosity, and they'll probably still figure out a way to burn you even with "safe guards" in place.
Mostly, except in very small organizations, there are several implicit safeguards to keep any one person from doing evil with the systems. They are subtle, but effective.
Peer review: Most sysadmins are hired by other sysadmins, or at the very least a technical manager. This means that you are hired based on your skills, reputation, track record, and demonstrated attitude. This means that ideally, you wouldn't even *think* about intentionally subverting a system, because that would mean breaking it or compromising it in some way, and most professional SA'a are simply too OCD to allow it.
Business continuity: Most organizations have several layers of continuity in place, such as disaster recovery scenarios, system snapshots, monitoring, and auditing. This means that unless you are VERY subtle, or work for an entirely incompetent team, you WILL get caught, and the damage will be minimized as you are being put into a police car, never to work in IT again.
There are no "indispensable people:" If you are a sysadmin, and you are the only one who knows your systems, you have not done your job. Every system and app should be documented, and there should be accountability for every change and decision.
No technical solution will ever replace good management and planning, and a design that eliminates the vulnerabilities of a system to rogue sysadmins, will also eliminate its flexibility. It's just a lot cheaper and easier to try and run a good shop.
-- lk t lv ll th vwls t f wrds. T svs lts f tm t wrt bt ts pn n th ss t rd nd mks m lk lk cmplt dpsht.
It is called: "Change Control" and usually goes along with "Revision Control" on configs.
If you change without recording the reason for change and without checking in the result so that the two versions can be compared and analysed you get a pink slip. Voila. Problem solved.
Um, excuse me. What's this? Bringing up tried and true old methods of doing things to solve a problem?!?
The is IT! We're supposed to reinvent the wheel, give it some new whiz bang buzz word, and then market it as some "new" technology and in the meantime, we get to put yet another term on our resumes which then adds "value". The PHBs and HR mor...staff will then think we have more skills and hire us over others that are perfectly able to do the job but just don't have paid experience in said "technology".
If all of us had your attitude, the IT industry would lose hundreds of billions of dollars in paper worth!
Hire admins who know their stuff and make sure you have at least two of them with the root password. Make sure they've got some kind of change control in place, and make sure you have them document what they're doing.
I've been the sole sysadmin before, and I always felt worried that my legacy, should I be fired or quit or hit by a bus, would be "She didn't do a great job because everything fell apart after she left/was fired/was bussassinated". So, I always tried to document things and made sure the boss had the "keys to the kingdom" (document with root pw and locations of my documentation to give to my successor).
The Digital Sorceress
First, understand that Slashdot is only going to provide a hint of what you will be doing. Security is complex and easy to get wrong, and there's a whole lot of evidence of that in the news. If security is important to your company, you should invest in a CISSP to really help you get things set up in a fashion that the industry considers to be best practices. Until then, consider these few generic suggestions.
Multiple layers of security help ensure that nothing goes astray, or if it does that it's detected before too much damage is done. And separation of duties helps make sure that one rogue actor can't do it all by himself.
Separate the admin of the box from the admin of the data. The guy who holds the root PW doesn't have to be the same guy who holds the private key for the database.
Add off-the-box auditing to the actions of root. As soon as someone signs on as root, notification is sent to a different box of the originating IP and it's timestamped. Don't let your application sysadmin be the sysadmin of the audit box! And the auditor should investigate carefully any situations that are out of the ordinary. (This box fell off the network just before root logged on? That's an odd coincidence.)
Define expected behavior with policies. If you want to run a trustworthy ship, clearly stating who has access to do what with which systems eliminates confusion, and helps avoid where one sysadmin creeps over into other systems.
Ultimately, you've placed trust your admin to do a job, and you need to trust him or her to do that job. Somebody's got to be root. But they also have to know they'll be held accountable for what they do.
John
Only hire admins who have a family. Make it a requirement of their contract that their child have a small explosive device implanted in them that is tied to the health of the systems that are administrated. Will not only ensure against nefarious activities, but make five-nines and above almost guaranteed.
In our IT department, root admins are required to sign a notarized document telling their deepest, darkest secrets in order to be given the password. This keeps them in line for eternity, and seems to be working quite well...
He who knows best knows how little he knows. - Thomas Jefferson
how about: a singular sysadmin can make any changes he wants (a policy can decide whether they take effect immediately), but it takes a second account to confirm/commit/consolidate those changes?
We have several solutions which work together to minimize the risk of root at my company:
1. Powerbroker. It's in use on every single UNIX system administered by our Global IT teams. Every user has a role (or several roles), and that allows them to execute a variety of commands with elevated privileges. Once Powerbroker is invoked, however, every single keystroke is logged and can be played back. These logs are stored indefinitely; access is very restricted.
2. Automated, centralized root password management. One of the steps to setting up a UNIX machine here is ensuring the root password and remote console admin passwords match that dictated by our automated provisioning system. Then every 30-90 days (depending on policy for this type of system) the root password is changed to a very long, apparently very random string. I can look this password up if my role allows it, but the lookup is also logged.
3. A good Change Request (CR) process. Every system that exists in a data center should have a record in our systems database. Once a system has passed through the phases of deployment (Warehouse -> Data Center Install -> Sysadm Configure -> Deployed) any change made to the system must be requested and approved by the owners of the system. This approval is logged, and the date/time of the work is also logged. Sysadms must close service requests within the time window specified by the CR, or apply for an extension or reschedule if they're unable to complete it within the allotted time.
The downside to this is that you lose quite a bit of system administrator work hours filing and managing change requests. However, this loss of efficiency -- IMHO -- is better than the mayhem that ensues without an organized change process.
4. Automated forensic tools to monitor changes. Information overload is a real risk with any Tripwire-style system, though. We're still working out some of the kinks on this part of the system. Once we ensure that all normal changes due to operation of the system and scheduled maintenance get excluded, this will be the fourth leg to reduce the risk of super-user privileges.
At any company, IT must find a balance between controlling user actions and monitoring those actions. In most cases, the easiest approach is to prohibit by policy only those things that might typically result in lawsuits, but monitor everything else to the best of your ability. Combining a Powerbroker-like product with automated root password management -- both with fascistic logging -- is a reasonable approach that works well for many large companies. Combine this with a change management system, and a forensic tool to automatically monitor and notify of unauthorized changes, and super-user isn't really all that big of a concern.
Matthew P. Barnson
I learn what I think when I read what I write
There are two whole categories of products to help with this problem, and as others have pointed out, the problem is decades old - not news.
(1) delegate "sub-root" rights, with sudo or a similar tool. There are commercial equivalents on Unix/Linux and even equivalents on WIndows.
The idea here is to limit the frequency with which an actual root login is required and to enable routine work to be done with lesser privileges, perhaps by other people.
(2) Control access to the "root" account. There is a whole category of "privileged password management" products out there - also known as privileged account management and privileged ID management, based on the respective vendor's marketing department's whim. Basically you scramble the root password every so often and control disclosure of access to that account via some combination of ACLs and one-off approvals workflow. There is typically also an audit trial here, and in some cases a full record of whatever the root session displayed and accepted as input (i.e., VCR playback of the session).
My company happens to make one of the latter products: http://privileged-password-manager.hitachi-id.com/ - we are by no means the only company to do so, however.
Ultimately, you have to develop some level of trust in your admins. All of these solutions boil down to "trust but verify."
Cheers!
-- Idan
Simple.
Hire two sysadmins. Give both the super user credentials, unix, linux, windows, osx server whatever.
Make sure they co-operate and have no say so in hiring processes for their co-worker.
If they won't work in such an environment, don't hire them.
It will always be a trade-off between efficiency and polarization of power. Just like in politics. Decision-making is extremely efficient in an absolute dictatorship, but people have to rely on the dictator being benevolent, staying benevolent, staying healthy (I've seen people's character completely change because of illness), and having benevolent successors (this almost never happens in real-life politics). On the other side of the spectrum, you have democracies with a proper implementation of Montesqieu's separation of powers, which in some instances get plagued by indecisiveness, gridlock, and half-assed compromises that really are not helpful for anyone. (*) You'll find the same principles apply to system administration, or anything that involves power.
(*)<opinion>Obamacare is a perfect example: started off as a radical reform that would allow America to take back its place amongst civilized nations, was watered down to a legislative abomination. Note that I'm not implying that the president has too little power - this is merely an example of one of the side-effects of the democratic system America is using. I can give some examples of presidential abuse of power as well.</opinion>
The only solution I can think of that would stand a chance is to require:
a) everything gets documented (you'll know this is the correct way, as all the techies will hate it)
b.) every week / month all the roles change, if an admin coming into a role finds that things aren't as they were documented, someone gets yelled at
This also has the advantage that you're no longer completely screwed if someone leaves, goes sick or gets promoted. it also makes it clear to the people in question that the company can get along quite nicely without them.
politicians are like babies' nappies: they should both be changed regularly and for the same reasons
There are commercial solutions where a central system manages all of your root passwords on each machine. You have to login and checkout the root password to access the machine. You provide a reason and a time limit (1 hour, 4 hours, 1 day, etc) after which the central system changes the root password.
So you get all the audit controls for logins and the root password is changed often.
Everyone treats everyone else like adults and every one acts like an adult? Honestly, if you don't trust your admins, why are they your admins?
Also, simple change management alleviates most of these problems. Even if it's just a log for what happened so that the next shift or your colleague tomorrow knows what you did today. Then again, I guess that is really back to acting like adults.
"I use a Mac because I'm just better than you are."
This is an old problem in high assurance systems. As other posters have pointed out, as some point you have to trust someone. But you can still "trust but verify".
The standard solution is "division of privilege". Over time folks have learned that the key is a system which audits everything the admin does, and the one thing the admin can't do is modify or delete the audit trail. A separate person or team has the role of auditor.
This is one of the requirements of a B2 level system in the old Orange Book model, and you'll see if it as a requirement if you need to provide systems for most countries' military or intelligence organizations. It's rarely used elsewhere because more or less noone else is willing to pay the staffing costs. The solution there is trust someone, and be ready to fire, sue, and/or prosecute if they violate that trust.
It is called sudo.
I hate to say this, but Plato raised the underlying question in 380BC, and it was rephrased in 2AD by Juvenal, "Quis custodiet ipsos custodes"
We have generally relied on Plato's solution. We have convinced sysadmins that they are somehow more noble than the poor (l)users, and must keep the root password in trust to protect users from themselves.
But enough history, on to the geeky stuff. Unfortunately, this is one area where it looks like Microsoft may have stolen a march on the Unix community. Active Directory allows the delegation of the vast majority of administrative functions to other accounts. So, while it still needs the 'all powerful' domain admin at it's core you can create a group of Account Admins and delegate the account administration rights to them. The ability to add and remove machines from domains can also be delegated. Using delegation means that full domain administration rights will seldom be needed (in theory anyway).
Unfortunately, many of the Linix core administrative functions are still property of 'root' and can only be delegated with difficulty. Has anyone seen a good case study of Linux rights delegation?
It may require a bit of re-thinking to do this for sys admin, but double entry bookkeeping has handled this problem for a very long time.
The basic idea is that you set things up so it takes more than one person to rip you off.
For example:
The person who orders stuff is not the same person who cuts the check to pay for it. Any payments have to have a corresponding entry somewhere else in the books. If the guy who cuts the checks cuts a bogus check, there will not be an entry in the books from someone who ordered supplies (or whatever). (It is important that each function only gets to modify the part of the books that is their direct responsibility. Only the auditors can see all the books and they can't change them.) The books won't balance and the dishonest bookkeeper will be quickly caught.
I'm not sure how this would work for sys admins and it would create extra paperwork. On the other hand, for critical systems, it would insure against sabotage or dumb mistakes.
And you clearly have no place being a sysadmin.
With currently available technologies and administration methods, it's not possible to totally disempower a system administrator. However, it's not entirely necessary either. Disaster recovery processes should be designed around the possibility of an administrator going rogue. In this event, a rogue administrator can certainly do damage, but the organization has an established and validated set of processes for regaining control and reestablishing services after the worst happens, and also has systems in place for logging and auditing that will allow legal action to be taken after the fact.
Separate roles appropriately with regard to disaster recovery, damage mitigation, and audit log integrity. This means, for example, if one administrator runs the backups, another administrator tests the recovery process - ideally by replicating the production environment at a disaster recovery site. Similarly, where this is a risk, all systems should be logging to secure log hosts, which are under separate administrative control - preventing a rogue administrator from erasing evidence of their activity.
Audits should be performed periodically to insure that management can regain control of all systems, even after a deliberate attempt by a rogue administrator to lock them out.
RSBAC
Rule set based access control. Who is root anyway?
http://www.rsbac.org/why
Peer Review, Change Control, Auditing, Maintenance Windows, Testing all changes in a lab before production, source and version control / maintenance. These are all best practices, work regardless of operating system and don't require any special software.
Why o why do you want to use software to take the place of established best practices? Best practices are there for good reasons, and those reasons usually have multi-million dollar lessons attached to them. You don't need special software, just a heavy that says yes you /must/ do it this way and raises hell when you try otherwise...
There's a relatively simple way to handle this. First, you set up a call center with operators on the phone who each have access to the servers under management. They each have sudo rights to a single command: to create an xterm as root. Their workstations are locked down and do not have an X server installed. (You can take this further and restrict X from reaching them via firewall policies.)
Second, the admins who need root access do have an X server installed. When they need root access to a system, they first call the operators above and open a ticket, detailing exactly why they need access, for how long, and what other ticket is being worked that justified this in the first place. Then the operator invokes an xterm, using the admin's workstation as their display. The admin now has a root session on the server in question.
A few other policy guidelines (no more than one root window to a given server at a time, all requests logged to a central database, auditing conducted to make sure the same admin and the same operator aren't talking more often than statistics say they should, timeouts on all sessions with active monitoring of /etc/profile to make sure the timeout isn't tampered with) and this should at least let you know exactly who is becoming root, when, and why.
The drawback to this scheme is that it doesn't cover those instances where a machine is not on the network for some reason. But in those cases, you institute a policy of two people being required to sign in and sign out of a cage before getting physical access to hardware. Two-person integrity should mitigate against any nefarious goings sufficiently (not perfectly, of course, but sufficiently).
God invented whiskey so the Irish would not rule the world.
No doubt many sites are "lazy", or "old-fashioned" Solaris has had "Role Based Access Control" for many years. Different tasks can be farmed out/delegated to different people.
Auditing, etc. all provided.
In my household I have been implementing SAP (Spousal Approval Protocol) for years.
Set your phasers on "funky"!
about 10 years ago i got hit with the question if it was possible to deny root access to certain files. The root user had access to some financial tranfering records that he did not need for the funtion of administrator.
I Failed to come up with a reasonable scenario for this, it seems to be impossible under unix (that was AIX 4.2 i think) to hide some file for the root user. (and still to be able to access them from user space and to data transfer with them.)
I think this would provide the level of control you are looking for
http://www.e-dmzsecurity.com/
Yes, you do, even if that someONE is YOU.
For all you quivering sysadmins out there, and you lusers that question their authority and trustworthiness:
Do you trust your CIO to not shackle you to poor choices just for their kickbacks?
Do you trust the Board to not limit your opportunities by failing to act on corporate goals?
Do you trust your CEO to not collude with your accountants and cook the books?
Trust.
deleting the extra space after periods so i can stay relevant, yeah.
Are there more sweeping yet practical solutions out there for avoiding the weakness of a singular empowered superuser?
Give the responsibility back to the users.
By removing the responsibility from users, one keeps them oblivious to the infrastructure problems. That perpetuates arrogance and the "not my responsibility" mentality.
By moving the responsibility to admins, to the people who do not use (for their primary purpose) the services and infrastructure they are responsible for, make them oblivious to the actual user needs. And self servingly often make the infrastructure more complex than it really needs to be - feeding off the "not my responsibility" mentality.
I'm strong believer that people should know their tools. It is either one wastes time in the extra bureaucracy which IT brings with itself - or spends time learning what actually is going on and how to manage it. Learning > bureaucracy.
My 0,02€.
All hope abandon ye who enter here.
It's called SUDO
Normal admin tasks can still be preformed and anyone that need higher privilege escalation can go to the other people that can grant that access.
"trust but verify"
To get some transparency / accountability, just set up an authlog black hole that includes all of the sudo activity from your servers.
Use the wikipedia-approach:
Don't restrict, but instead log exactly who does what change when and why, and make it trivial to undo any change.
For example, for /etc use revision-control, and require that all changes be comitted.
That way, yes, the one doing something may screw up, but you can easily undo it. And when the customer calls and goes "why doesn't that work, it worked last thursday!" you can trivially get a list of all changes since then.
As an added bonus looking at logs and commit-diffs give a new admin an excellent way of seeing how a task is typically done, and so has benefits in training.
Nothing is perfect but what you described is pretty close. I used to work for an ISP in the 90's and we had something similar. We figured if things ever got "so hosed" we would be booting into single user mode from the console so root's encrypted password was disabled (started with an *) so there was no logging in as root. The closest thing to root was accounts that had the ability to sudo visudo but all sudo was logged remotely via syslog.
This does point out the one security risk that has to be accepted and that is: physical access directly equates to root access.
You also have to be careful about what kind of access you allow with sudo. For example:
If you give a user the ability to "sudo vi /somedir/somfile.txt" there is nothing to keep them, nor any log of them, reading in any other file and writing it back out to any other file once the vi has launched... so doing so is the same as giving them access to "sudo vi". Believe me this gets tricky!!
Encryption: I may not agree with what you say, but I will defend your right to encrypt it...
KeePassX , with a master password, and hell, if you want, maybe another secret keyfile only distributed via encrypted e-mail or by hand.
You FOOL. How could you so openly reveal the secret to successful business?
If the Americans re-learn this basic principle, they will throw out SOX, HIPAA, GLB, and FDA regulations and simply hold all human beings responsible for their own actions, rather than their intentions or lack of documentation. That would return the Americans to political, economic and social dominance within 20 years!
WE CAN'T HAVE THAT!!!! Keep the Americans weak and childish, and everyone else wins.
You insensitive clod.
Maybe Computers will never be as intelligent as Humans.
For sure they won't ever become so stupid. [VR-1988]
BMC Bladelogic provides a nice solution that enables you to perform root operations without being root in a supervised manor, rules etc can then be audited automatically or using your two factor admin approach. :)
It is most likely one of the futures out there
Only the largest companies are willing to actually pay for three competent admins... and now all three of these guys will have to be on call 24-7 in case any changes need to be made. Great idea!
Gamingmuseum.com: Give your 3D accelerator a rest.
There's a fancyschmancy name for what you want, that I forget. Old solutions for this sort of thing can be found in the old "orange book" security levels.
Ye old "C1", "C2", "B1", "B2" certified levels.
A proper B1 level (or is it B2 level?) system does not have a single uber-root. It has "role based access".
Sun used to sell a "separate" variant of Solaris, called "trusted solaris", that was out -of-the-box compliant with that sort of thing.
It had a huge pile o manuals, and admining it was "a little different".
Nowadays, they've rolled all the features into regular solaris. But it ships in normal solaris mode by default. You have to do a buncha stuffs to de-rootify it. Make sure that you have enough people who have all the appropriate, required "roles" covered.
Unfortunately, I cant seem to google a good guide for setting it up that way. You'd have to go through the online solaris docs to find out specifics. I suspect you arent serious enough about this idea to really pursue it though. Virtually no-one is, once they find out what it involves. they decide, "Eh, we can live with the root account after all". Which is why trusted solaris did not do a huge amount of business :)
In my experience most sysadmins would refuse to give up control. The only other control they usually have in there lives is telling there mum what type of pizza to bring down to the basement of an evening. :-)
America, Home of the Brave.
"Practically every computer system appears to be at the mercy of..."
That's an interesting perspective. They're usually "at the mercy of" because they dump responsibility by the truckload on said individual. And then they complain when said individual has accumulated too much power. Personally, I don't like having all this responsibility and the uncomfortable level of power that comes with it. The whole joke behind the BOFH is that he *loves* the power that comes with the responsibility.
A very talented, and very honest person will not put up with layers of approvals and constant monitoring.
They will find a place where they have respect and freedom.
You will end up with a mid-to-low talent bureaucrat.
As a one-person IT department, I made a recommendation to management, reflecting a practice that is used in some other high-trust industries, like banking: audit me.
Really. Give me a couple of paid weeks off each year, and have our auditing firm come in and look at the logs, my access, and the security model. Not only would it help the company feel good about the network controls and their network administrator (yours truly), but it would also give me a couple of weeks off without being hounded every day for me to fix something or other, as usually happens on my days off.
I use irony whenever I can, but my shirts are still wrinkled...
Most of the time privileged identities are not properly managed because IT wants share credentials out of convenience. In other cases credentials are not properly managed because it is not well understood where the credentials are used and any attempt to change them will result in outages. To properly solve the problem, a technology is needed to take away the knowledge of all sensitive credentials for all admins and only release them in a controlled way and only for a limited amount of time. After the time expires, the credentials MUST be changed automatically without human intervention.
The management of privileged identities is a well understood process and is covered by all of the analyst firms and forms a subsegment of the identity management field. In the case of our product for privileged identity management (PIM), we already support multi-factor authentication, segregation of duties, complex delegation rules that factor in IP address, certificates, time of day, multi-party approval, trouble ticket verification, system sensitivity level, and all of the transactions are fed into SIEM systems using SYSLOG and other event systems.
The control mechanisms for PIM to make sure that only the right person with the right access can access systems. We think (www.liebsoft.com) we do this better than any other company, but we are biased.
There are no "indispensable people:" If you are a sysadmin, and you are the only one who knows your systems, you have not done your job. Every system and app should be documented, and there should be accountability for every change and decision.
If the sysadmin that didn't do his job leaves, the company still has a problem.
It's particularly dangerous to trust each individual to do his job well if a bad job will only be spotted when the bad employee leaves the company, which is the case of sysadmins.
The real safeguard is to have somebody check that the documentation exists and can be used. Ideally the sysadmin's manager.
I first encountered the "wheel" group on the PDP-10 (mid 70s) where the command to become superuser was "pivot". Wikipedia says the "wheel" name is from slang http://en.wikipedia.org/wiki/Wheel_(Unix_term)
Typing in your password 300 times per day does nothing. The point of logging is to be able to read the logs afterwards and parse them. :wq! ./bob
Security asked me once, "well, what commands do you need." I mailed him the contents of all the bin directories.
Sudo is a worthless circle jerk that does nothing but complicate an environment and add additional workload to get a checkbox that is absolutely fucking worthless.
Other than that, I have no opinion...
vi bob
su - guyihate
rm -rf /
chmod +x bob
sudo
More fun to tack it on to a DBA cron job, but I'm just saying.
Sounds like an HR problem to me. Hire people who are competent and trustworthy, compensate them well. Problem solved.
Two admins. Two keys. Two guns. One set of orders. Problem solved!
-kgj
Not sure if SELinux can be used for this or not, but I think so. I don't think that helps you on AIX 4.2, sorry.
I am absolutely freaking tired of living in fear of what if. I have been at this for more than 15 years and I have seen it all. I work hard, I dedicate myself to the systems, my employer and employees only to be studied like I am some kind of loose cannon. I am tired of it! Rant over...
A simpler, yet often not employed strategy, would be to hire competent people, and respect them for their abilities and role within an organization. If you shit down your SA/SE's neck on a regular basis, hire them for their ability and then second guess every idea they ever have or constantly create more work for them, because your IT budget has been supplanted by your "I want a boat" budget, you get people who resent you and want to fuck you hard for what they perceive as your gross incompetence that causes them grief and stress.
The way to avoid this is to give them the respect due, or hire somebody who can be respected. Compensate them for their time and effort and make their rewards seem to be corresponding to their work and help to the organization.
Provide some logging facility protected from access by the root user to track administrative actions. The root user can do whatever s/he wants, but will be answerable to some authority following an audit.
Have gnu, will travel.
There are a number of products available to do it..
One of the best is http://www.novell.com/priviledgedusermanager
There are some open source implementations as well.
No matter what you do, someone needs to know the all powerful password to the system (And they should be written down and stored in a safe so that if that person dies/quits/terminated, someone else can get them).
Even if you make the most perfect system there will be the occasions when you something out of the ordinary happens and you will need root.
I personally prefer an environment setup, so that people have to log in as themselves and switch to root. That way you have some audits of who did what. And on critical systems, I've actually installed a key logger so that we knew even what commands were run.
And if it is possible setting up Roles or sudo so that common tasks are can be done as themselves.
But in the end you will need to completely trust at least one or two of your IT people. They will have the keys to all your critical systems and know all the security/checks and balances around your systems. In medium/large companies, you might have silos, so that specific groups own systems, which makes it so that one set of IT people don't control everything, but even then you need to trust the people running your ERP system.
From the makers of phpMyAdmin, here's wikiSysAdmin! It's a wiki where the wisdom of the crowd is used to determine the config files and process states. Your system will have one 9 of up time, but it will truly be an egalitarian server. RMS would love it.
Rotate admins between systems/responsibilities and have a third party do random audits.
In the end, logs can be falsified and nasty activity hidden. You have to trust your admins at some point. At least by rotating admins they can basically keep an eye on each other and cover for each other as needed. As an IT guy for a small company, I welcomed having others that I trust do some IT stuff for me while I'm busy or gone. Be wary of admins who get defensive when asked to allow somebody else to take over for them from time to time. Admins who feel they need 'job security' or are allowed to 'play' on their own too much can be dangerous.
As a CS prof told us, "You'll spend the first year of your job trying to get root, and the rest of your career trying to get rid of it." ;')
Lastly, if you are paranoid, do random third party audits. I've been through a few detailed audits, and I actually enjoyed having an outside source confirm I'm doing things right (or wrong). They can actually be empowering as they can justify an increase in IT budgets that otherwise would be a hard sell. Again an admin who goes apesh!t at the idea of having somebody else check their work is a bad sign.
SELinux can block root from accessing files.
Of course, people who are root usually have the ability to get to the console of the system and boot from other media, but it's pretty obvious when a server reboots if you have appropriate monitoring.
"Alcohol, Tobacco, Firearms, and Explosives" should be a convenience store, not a government agency.
That's a job for encryption, not file attributes.
"However, making a system require multiple individuals for any root operation (think of the classic two-key process to launch a nuke) has shortcomings: simple operations sometimes require root, and would be enormously cumbersome if they needed a consensus of administrators to execute. "
--------------------
This is also ineffective. What happens if one of the multiple individuals rage-quits or several of them are involved in a plane crash? To safegaurd against this you need administrators to be able to reset other administrator's passwords. When you can do this, it eliminates the safeguard.
--------------------
"Are there more sweeping yet practical solutions out there for avoiding the weakness of a singular empowered superuser?"
--------------------
1. don't hire foul balls
2. have multiple administrators with super user access. only giving this to one person is stupid. What if they die of a heart attack?
You can't fix this problem with software. You need common sense and a knack for hiring good employees. It also requires a probationary period for you to feel out new employees and make sure they aren't nut-cases, and that there's nothing "off" about them. If you have suspicions about someone, it's usually for a good reason. Listen to them.
Don't kid yourself. It's the size of the regexp AND how you use it that counts.
if it was possible to deny root access to certain file
---
the only way to do this is to encrypt the file with a key that is closely guarded by the financial employee who uses the data. It can't be on their workstation. Presumably the administrator could get it if it was on the finance employee's workstation.
You can't deny root access to data using the operating system controls. Root owns the operating system controls.
Don't kid yourself. It's the size of the regexp AND how you use it that counts.
Use an authentication server and password escrow, backed up by a good HIDS.
One site I know of uses a RADIUS server and sudo-enabled user accounts. If you need to log in as root, you need to hit up a manager for the password, which he gets from an escrow system that logs in after you're done and resets it to something new.
Accountability - if the BOfH does something stinky, it will show up in the HIDS logs, which are hopefully maintained by another department and reviewed regularly. Authentication logs will then show who the culprit was.
... disemboweling a singular sysadmin?
After logging in slashdot still does not take you back to the page you were on. It's been that way for 20 years.
System Subverters worry about system subversion obsessively, just as nobody is as paranoid about being stolen from as a thief.
When I find myself working for a person who is repeatedly asking how "we" are going to make sure that (some guy) doesn't have too much power I immediately ask two questions: (1) Does (some guy) already have too much power that he is apparently ready to misuse [kinda common] so obviously that he has already implicitly threatened the good function of the office; and (2) is the person asking the question engaged in personal entrenchment and bad dealing to a degree that they are expressing the worry that others are able to do to them what they are already doing to others [far more common that item 1].
Both conditions are toxic. The question itself is a warning sign of bad things to come.
A good system admin will have _already_ established some kind of key escrow and should already be training his replacement by the end of day one on the job. That's because a good system administrator knows that he doesn't want to be called at all hours of the day and night for pointless tasks, nor does he want to be so indefensible that he will never be able to advance out of his current position.
There are plenty of acceptable system admins out there who don't have these personal policies, and that isn't that bad.
But when your boss doesn't trust, that is usually a sign that your boss is not worthy of trust himself. This isn't always the case, but it is the safest bet to make.
Besides, "disempowring" users or admins _never_ really works. It just leads to multiplied consequences. Set up a mail retention policy that discards mail after ninety days? You'll find people saved their mail all over the place by dragging it out of the mail system. Lock down your USB ports and you will get things emailed will-he nill-he and find burnt CDs everywhere. Try to lock out your administrators and they will spread undocumented hacks all through your system. All this just so these people can meet the minimum requirements of their jobs.
"The more you tighten your grip, the more systems will slip through your fingers" -- This is a truth, not just a Star Wars quote.
Innocent people shouldn't be forced to pay for inferior software development.
--"Code Complete" Microsoft Press
No. Now just hang on a second while I delete your user account and all your data, you presumptuous bitch.
I've fallen off your lawn, and I can't get up.
Have an authorization dialog box with 2 pairs of username and password fields. Allow both a single root privileged account to authenticate, and when the single admin is not available, allow two IT group staff accounts the ability to escalate authorization levels when they both enter their separate authentication credentials. And of course, log the accounts used in the security log.
This gets to the core of this post. Its not about su/sudo, its about trust.
I think giving multiple people root access is the opposite direction the original question intended. I think the idea is that you don't want ANY one person to have the ability to bring down, corrupt, or steal from a single system. Considering the recent madness around congressional data, passwords, credit card databases, and thousands of other examples this sort of security seems more and more reasonable.
I work on some unclassified Department of Defense machines, and even to get root access to those machines our admins have to get Top Secret clearances. Of course, this does nothing to ward off incompetence and is only useful for providing increased trust in your single point of failure, (but that trust is important). I assume you're looking for assurances beyond this sort of social-engineering aspect.
The two-key (nuclear) option mentioned is probably workable for a reasonably administered production system. There are several ways to implement a two-person system that could work.
First, routine tasks, or tasks which can be pre-planned (i.e. non-emergency situations) could be scripted on backup environments without the sensitivity or criticality of the production system. This should be standard practice anyway. Once a working tested script is prepared (and that can be a script in the programming sense a la .ksh/.perl or a script in the user-documentation meaning). The production system can require dual authentication for logon before the script could be executed. In a fully automated setting this is a completely technological solution (although I admit that I don't know of a working implementation that is publicly available), but even if there are some manual steps required, holding both key holders responsible and requiring at least over-the-shoulder confirmation is a useful policy.
For more complex tasks, such as recovering physically corrupted data, which really can't be simulated and scripted, dividing responsibilities and ensuring multiple people are in attendance at all times can significantly reduce risk exposure with minimal impact (it doubles your staffing requirement, but only for the hopefully brief periods when this is required). Again, this is not a technical solution, other than having lock boxes or rooms with multiple keys.
It's important to realize of course that you'd have to apply this solution not just to your sysadmin role, but to your physical infrastructure (no one person can access the power button, the fuse box, etc. by themselves), and to each application running on the system, at least for any role that has access to protected areas. This is nigh impossible in a usable system, so the problem gets pretty complicated, but if you really need the security then it may be worth the effort.
Maybe I am just having a rough day, but that's how I initially read the title...
"You canna take ma root passwoord away, but you'll 'near take mai FREEEEDOOOOMMM!"
A very talented, and very honest person will not put up with layers of approvals and constant monitoring.
Have you actually worked at a company composed of very talented, very honest people who put up with this very thing every day? Setting up an ITIL-compliant change management system -- and getting everybody on-board with using it -- is a very daunting procedure. Speaking as someone who has been on both ends of this, I can say that in the end, it's worth it. My day as a sysadmin is no longer all about putting out fire after fire, dashing around and pulling crazy hours at the whims of vice-presidents who think the latest thing they heard about is the "highest priority".
Work is scheduled, executed, and followed-up on. It's not perfect, but requiring approval from the stakeholders prior to making changes has been a HUGE improvement in my quality-of-life and that of my fellow admins. The principal cost is manpower: I spend far more time managing changes than I used to.
It bugs me when people equate "managing change" with "being a bureaucrat". The process of getting to the point that we could manage and track all our changes was a royal pain in the butt. Five years later, even if you factor in the time-cost of documenting all changes to all systems, we're running more efficiently than before.
I've played the game of being the chief sysadmin in a startup before. Heroic effort, hectic schedules, and obscene hours at low pay aren't what I'm interested in anymore. And truthfully, the average quality of sysadmin I work with in a Fortune 500 company is head-and-shoulders above the admins I worked with before. There's a minimum standard we expect, and if you can't hack it, you're out.
That minimum standard includes knowing that any change you make as root on a system will be monitored, catalogued, and may be subject to a later Root Cause Analysis (RCA). Learn to behave ethically, carefully, and competently at all times when you're working as root, and it's no big deal. In truth, I'm GLAD for the monitoring. During any audit -- and I've been through many! -- I can point to the service request number, change request number, and approvals for the work I performed. It's good pay, my ass is always covered when I make important changes, and I get to work on machines in a data center that makes the warehouse at the end of Raiders of the Lost Ark look small. What's not to love?
Matthew P. Barnson
I learn what I think when I read what I write
I once read a story about a company who had a widget making machine. One day the machine breaks and they have to call in a specialist to fix it. An hour later the specialize arrives and they take him to the machine. He looks it over for a minute, pulls out a wrench, and turns a bolt. The machine springs back to the life and the specialist hands the company rep his bill. The rep is astounded when he reads it and exclaims, "$5005.00?! But it just took you a minute! How can you charge so much?" The specialist replied, "It's 5 dollars to come down and turn the bolt, but $5000 for knowing which one to turn." My point is that companies need to realize that superusers/admins get paid what they do and have the responsiblity they hold for a reason. Looking for a "quick fix" and trying to cheap out is just going to get egg on your face and put your company's infrastructure at risk later. My suggestion to employers is to make sure they do adequate background/security checks on the people they're handing the keys to the company to.
"On a scale from 1 to 10, people are stupid"
If you actually physically possess the box and can get to the power switch, console and input devices to boot off then you "own" the box. Any "takeover" is probably a good enough excuse for an outage, pulling network cables and locking out whoever has misbehaved. Balance that over having to have multiple people available at all times instead of just one of several. The idea as stated above has many drawbacks and would be better done with physical access restrictions. Set up some functions to only be possible from the console, then put that in a locked room with the two keys from out of the movies or whatever. Then be prepared to rebuild the wall every few years when emergency single person access is required :)
"In a DAN, each instance is administered and maintained by a separate administrator or administration group with unique passwords."
"When new data is introduced to one instance (as when a user creates or modifies a record), that data is then replicated to other members of the DAN which accept that instance's hash."
"When an instance fails to properly conform to the hash or other checking-mechanisms, it is deprecated from the network."
O RLY?
So you're telling me, there's a guy who has Admin privs on a machine, and every "change" on that machine is automatically accepted by every other machine, yet if something is "different" on one machine that machine is automatically taken out of the cluster? This is stupid.
First of all, if every change is automatically accepted the admin can just make it look like a normal change and poof, it's propagaded. If you have admin rights you can do anything you want. All keys, all everything is exposed. The game is over.
Second, this is an awesome Denial Of Service attack. Just somehow cause a corruption in one or all of the other machines and your sole machine is now the master of all. Huzzah!
You don't need 3 independent systems, this is so stupid. You just need one machine and grant root to *nobody*. Create admin accounts which can do virtually anything, shape the roles with selinux or capabilities+permissions or a number of other access control methods. Install services and configure sudo or other tools to run in unique security domains or as unique users. Install tripwire, a configuration management app, monitoring, etc that all run in a security domain OUTSIDE of the admin role's control, and have it verify everything on a regular basis and send alerts if anything has been locally edited. Use checksums on binaries (again, outside the admin role) so only authentic applications can be run.
Granted, all of what I said is possible under Linux and is probably possible under Windows, and should not take too long to set up. Also it requires nothing more than Bash scripting skills and the ability to use things like SELinux (or alternatives). One server, multiple roles. This isn't rocket science, kids, and it's not vaporware.
At work, we are 4 sysadmins (two dedicated, two time-shared with other tasks) working in a number of systems. To mitigate this particular problem, we've setup an infrastructure comprised of Kerberos, LDAP and AFS with sudo for privilege escalation.
The intention is to manage the risk of unintentional mistakes causing problems, as well as some degree of traceability when problems occur. No single admin knows all systems, so one may by mistake cause conflicting configs in unknown systems, but in those cases, you can pretty quickly determine who did it, and roughly what was did by examining logs. Enables the shame-factor, education, and acceptably quick recovery.
The highly critical functions runs in an isolated system, upgrades to both code and config are version controlled (as is the bulk of the config of every individual service), further improving traceability, with the praxis to only do updates to those systems in with at least two admins present in order to avoid mistakes and ensure everyone keeps well familiar with those systems.
However, it's very important to note that all of this is intended to improve maintainability, NOT security. All four admins have individual physical access, so any one of us could naturally while doing onsite jobs, perform an "upgrade-reboot", and sneak in init=/bin/bash on any system and do whatever without logging. In the end, you pretty much have to trust your co-workers, or pay in both money and flexibility for a non-centralized organisation.
Your company already has people that clean the place that have keys to every room. That should put some of the unjustified paranoia and misplaced jealousy in some of the comments here in perspective. It's not POWER it's responsibility - just like the responsibility to clean the urinals and sometimes just as unpleasant.
The people on the "review board" (my boss and his compatriots) don't even know enough about my job to be able to determine whether what I've written is accurate.
They're more worried about having paperwork so that they can blame me if something goes wrong (they have the same boss) than they are about something not failing in the first place. Remember, blame shared is no blame at all.
So massive amounts of paperwork are generated ... and nothing gets done. Until something breaks and then then all the updates are applied at once in an effort to see if that will fix the problem.
We had a SAN and attached blade chassis with ancient firmware on it. But I couldn't get approval to update it because the blades were running too many "mission critical" servers. Until one of the disks failed and the entire thing went into alarm. Approval was granted for THAT WEEKEND. Anything. Just get those servers stable. Pay the overtime for the consultant to come in and help with it.
No amount of paperwork worked before the disk failure.
After the disk failure, A single sheet of paper with a SINGLE PARAGRAPH justifying the outage was deemed sufficient.
If there's any chance at all that something might NOT happen "on their watch" then they're going to dump all the effort onto you to get any change pushed through. Just because your change MIGHT cause a problem that they'd have to "deal with".
And by "deal with" I mean "tell you to fix" and "explain in a meeting with their boss why YOU broke something".
A single sysadmin is a seriously problematic anti-pattern even when you don't want to be the single point of failure. There's a theoretically infinite amount of work, and there's never enough time for documentation that would make you not indispensable.
So while you're waiting for a second sysadmin, or at least a PFY, you have to work out how the hell to make yourself not indispensable.
The key point in terms of this article is: anyone with root on a Unix box can actually do what the hell they like - but almost no sysadmins do, because we're nerds and actually care about getting stuff right, or we wouldn't be doing this for a living. Yes, you can haxx0r the logs. No, no-one will care.
* It is the job of every sysadmin to automate themselves out of existence. The trick there is that you are never actually automated out of existence, because there will always, always be new things for you to do.
* Maintain an intranet wiki. Just use MediaWiki, it looks like Wikipedia and people think they understand that, and even if they don't it's quite usable out the box to sysadmin standards. I use ours as my personal notebook and if anyone else benefits that's just fine.
* Document by reflex, even just notes. Cut'n'paste the terminal to the wiki. Everything. The search function will save your arse. I may be unusual in being one of the few sysadmins I know who likes documentation.
* If paranoid managers who don't know what you do for a living want audit logs, just get in the habit of using sudo. sudo -s in extremis. Autosave history.
* The ideal is that every service can just be restarted if it fails, and that a crashed box will come back with all services. This tempts people to administer Unix like Windows, but robustifying the services is enough discipline to feel like you're doing things properly. My boss and I agree this is an excellent ideal and sometimes we even bother. Hiccups are documented on the wiki. *cough*
Anyone else got ideas on the simple notion of getting a damn PFY when you need one?
http://rocknerd.co.uk
Yep. Because when it was enabled, the port was not ready for when the database came up and since the database server couldn't authenticate to the AD controller ... it threw an error.
Why not tie the database to a process that completed LATER in the boot process?
Because it is easier to blame the network and have the network guy find the REAL problem than it is to be competent at your job in the first place.
Once the network guy is gone, you can rewire the switch however you want to because the network guy is an idiot.
Why'd does every port I plug into go into an error disabled state?
I think it comes down to a level of understanding. Perhaps one solution is to make many administrator functions require user authorization. For example, if I worked at an accounting company and had a ton of sensitive documents, an administrator might need a sponsor's permission to view those files. Perhaps the documents are not ACLed for them or my manager, but combined, they could access them. I think the threat is data security. At the moment, any domain admin can access anything on my machine. Since they really don't need access to anything in my user folder, I think it'd be fair to limit that. The people with physical access to data-center machines are all screened by serious background checks. They're trusted about as much as possible. But down the road, I can see full disk encryption with distributed services making physical access much less of a concern. At that point, if a user can understand what an administrator is doing, they can self-audit what people have done/looked at on their work machine. After all, the end-user is the best person to evaluate if something should have been done. It's just that logs and authorization requests are too vague and technical for those users. In most cases, a plain english explanation exists. If the system could phrase access requests in a task oriented fashion, that'd be much easier. A dialog asking for permission to 500 random files vs a "install latest version of Word" or "gathering documents for legal audit" would be a huge change.
They're called botnets.
The only way to do this is to send the file (or a good copy) off to another box that root on the first box doesn't have access to. It's like keeping multiple copies of a ledger and flagging any detected differences.
http://rocknerd.co.uk
There are a number of different objectives people might have for splitting up superuser powers, and depending on what you're trying to accomplish, there are different kinds of solutions out there. For instance
You really need to nail down your problems and objectives carefully before looking for a solution. Security can really improve your operations if it matches your goals, but it can also really interfere with work if it's preventing you from doing things you need, whether that's directly blocking appropriate actions or whether it's by making you use a Linux distribution that's inappropriate because it's the only one with your required security buzzwords all checked.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Sorry, but the meme was just sitting there....
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
"trust but verify"
To get some transparency / accountability, just set up an authlog black hole that includes all of the sudo activity from your servers.
Ahh, but couldn't a local admin disable that logging?
They don't grade fathers, but if your daughter's a stripper, you fucked up. --Chris Rock
It's been a couple of decades since I messed with it much, so I don't remember if it was B2 or B3 that couldn't have root, but you could definitely still have root on a B1 system, and the AT&T System V MLS version did. Networking has become such an essential component of computer use that you'd need Red Book compliance, not just Orange Book, and during the 80s that was still Way Speculative Research. There were some limited-purpose systems that could comply with Red Book or Orange Book A1, but they didn't really provide user environments.
Of course, lots of the government procurements that wanted an Orange Book secured system also wanted Ada, and Posix-compliant with the latest Posix spec, and Posix Real-Time, and Specially NSA-Tweaked X.25, and the GOSIP ISO protocol stack, and it all had to be Commercial Off-The Shelf (so the government didn't have to pay for custom development costs, even though they were the only customer), and yes, NASA, I'm talking to you, Orange Book systems didn't really support networking in their certified configs yet, and messing with anything in the Trusted Computing Base required re-certification, and at best you could get two of these checklist features at once, and the only way this stuff ever got to be COTS was that component vendors would agree to buy these things from you so that you could sell the Feds boxes with their parts in them.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
You know, the "problem" these "solutions" are meant to address rarely occurred back when the company didn't think it was "okay" to treat their vital personnel badly. That really wasn't that long ago (before late 80s-early 90s). But with the "B Ark" people firmly in charge... so it goes.
So far, I don't know of anyone that's figured out how to implement Mandatory Access Controls without using cryptography. I don't think you can enforce privileged two-person integrity without MAC, and based on multi-part keys held out of band. Since most environments are designed to attribute actions to a specific individual, nobody implements a TPI technique that would result in two individuals being 'jointly and severally liable' for an action on the system. The closest thing I've ever heard of is environments like OpenVMS that allowed you to set up a second password for an account.
Let me get off the floor and stop laughing so hard.
I wished that I had more than one person here so I can take a good vacation without worries about systems at the office. Also for the other person so they can take real vacation also.
In this "dismal" economy, I wish that are companies that are willing to hire more than one competent & trustworthy person per shift (if they shifts) so that people can (if they want) have a personal life.
Let me see reality first of companies willing to hire & keep competent & trustworthy people so they can distribute work properly so that no one is a "dictator" of the system.
If disabling that logging would be logged (because it's initiated before the logging got disabled), and he doesn't have access to the logs themselves (e.g. because they are on a computer he doesn't have access to), I guess disabling the logs would be a sure way to get fired immediately.
The Tao of math: The numbers you can count are not the real numbers.
How about an integrated change control system sitting on a document oriented database relying on an underlying Public Key Cryptography Infrastructure with passwords on server IDs that are only available ephemeral and once the servers topples over it is considered compromised. And with the root certifier locked away and allowing only multiple admins to change passwords on Server IDs. Oh yes the system does exist. Guess what it is called? Starts with "N" and ends with "otes". Happy New Year.
So suppose you require 2 users to make any change. One user suddenly decides to be an asshole. Then what?
When you have 1000 servers, in 30 or more different configurations, typos, changes, and unexpected conflicts are inevitable. They're aggravated by complete fear of clean-up and integration operations, where ordinary system upgrades and software cleanups cannot occur until after complete project planning, which triples their timescale and makes them far, far more likely to fail catastrophically.
It's a complex balancing act: I appreciate my role's ability to come in from outside and support the competent people who've been struggling, unthanked and ignored, to actually get their systems working with external validation.
assuming the administrator keeps proper documentation you should be fine, even in the event of their sudden demise.
Please do tell me which Fortune 500 companies you worked for, so that I can short their stock.
I've worked with most of the major oil and gas companies in my role with a "Big 3" IT company. I can tell you that of the companies with over 1,000 employees, for the most part they do a solid job of protecting themselves from the single root problem.
You simple create and "Admin" group, each admin's actions are logged. It doesn't prevent them from tampering, but it does prevent the "Oh darn the admin walked in front of a bus or is on his honeymoon" problem.
That said, over 15 years ago, I was an admin for a Fortune 1 company. They had a system for recovering forgotten passwords that was...unique.
1) The user forgets their password and calls the help desk.
2) The ticket is transferred to the admin team.
3) The admin calls the auditing department.
4) The auditing department issues a one time password for a specific account, this could take 3 - 5 minutes if they were there, or you'd have to wait until they returned your call, usually less than an hour.
5) Reset the user's password.
6) Call the auditing dept and tell them you were done with the Admin ID.
Total average time to reset one password, 30 - 60 minutes.
Do this for every single forgotten password. This is for a company with over 100,000 employees globally, we supported over 10,000 in our little group.
We demonstrated to them, and tried to explain to them how the system logs every user's action, that their process simply broke the audit trail since every single password reset showed up as "Admin" instead of our individual UserID.
So, we did what any sane Admin would do. We used the tape backup software's userid and password which also had admin privileges.
But that was a loooong time ago. I've been told they fixed that kind of silliness years ago.
sudo vim
I need Vim. And with it, I can still do pretty much everything, like spawn a shell. Your Vim is patched to not allow that? Then who installed that version of Vim? Can run a different executable? Like my custom Vim?
Long story short, in a reasonably complex system (complex enough to do actual work) it's next to impossible to guard against a root user who wants to do damage, just any kind of damage. And reinstalling from backups does not count as I will still have done some damage.
It boils down to: "I can access the email of anyone in my company. I can access the email of 99% of our customers as we are an ISP. But why would I?" Work and personal ethics. Find people you can trust and work with them.
"trust but hold their children hostage" seems to work better i find
RedFlayer, you're a stupid asshole and that is all there is to it.
Sounds like your CEO's laptop should have been an Etch-A-Sketch....