Slashdot Mirror
How Do You Protect Servers From a Rogue Admin?
Treborto writes "I work with a non-profit that has an extensive collection of photos and videos. These are used in publications and on the web. We have several levels of privileges: read-only of small, watermarked images; read-only of large, clean images; edit of the site; and admins who can confer privileges. It has happened that people leave the organization in anger. So far, no Admin has done so. Is there a back-up, site mirroring, privilege, or other strategy you'd recommend so we have protection from an Admin gone bad?"
FS snapshotting and backups are the only way, but make sure your backups are protected (locked up) etc.
If people routinely leave your non-profit organization in anger, then the organization's leaders probably need to address a more fundamental problem than server administrative rights.
And then set up a sane sudo environment so that you can remove users who should no longer be able to run commands as root.
First you have to make sure you have a decent quality admin, don't treat him like shit like many companies tend to do. Make sure you don't take on more admins than you need; this reduces the risk of one 'going bad' because you decide to get rid of him and not another one.
You can have someone check the backups and keep them offsite so it at least won't take too long to recover from a rogue admin attack. TFA fails to mention if these admins have physical access, if they do you are pretty much scrude apart from legal recourse.
Create a encrpyted password protected snapshot archive of your server and name it something catchy like angie jolie secret sextape 1-29-2011 and upload it to piratebay. Safe secure lifetime backup retention online.
And usually that's the admins. Most admins gone bad would be smart enough to bone the backups if they were going to do deliberate damage. The best way to protect yourself is an off-site DVD backup, but that's a lot of work to keep current.
That's our life, the big wheel of shit. - The Fat Man, Blue Tango Salvage
On a serious note if its cost effective to safeguard against a possible bad admin implement a peer review system of the workload each admin does.
Don't treat them like mindless little robots that live in a closet somewhere whose sole purpose in life is to be summoned by you, fix whatever you screwed up within 5 minutes, and then disappear.
rsnapshot on a regular basis to a off-site service, that's read-only to the organization. I run that kind of service for several organizations for exactly that reason.
you leave in anger and ruin the backup...
There should be more than one person worrying about this, keep the physical media in somebody's hands (preferably management)
how long until
The best thing you can do is plan to mitigate any damage done. Of course this is easiest by not giving anyone any rights at all, but when you do have to give someone any kind of power try to wall them in as much as possible, so what damage they can do is very limited. Offsite backups that they dont have access to is best for recovery, especially if they have physical access to the site. I know some people will complain that treating everyone like a criminal will encourage destructive behaviour, but at the same time using smart/sane security precautions shouldnt scare away any reasonable people, and those who do react badly to being walled in probably arent the people you want on your site to begin with...
drunk chemists
Rogue admins are extremely rare. So rare that there are many other more likely threats you will encounter, such as hackers or data breach. Worry about those first.
The reality is that most people work in a spirit of cooperation and don't want the black mark on their reputation. They would rather walk away without burning bridges.
That being said, bad admins (and employees in general) spring from two causes: bad treatment and pre-existing jerks.
The best way to handle both situations is to talk to your employees regularly, and find out how they feel. If you know that some policy or other is bothering them, you can avert a crisis very easily if you know about it beforehand.
Some people are just jerks. Don't let these people continue in your organization, even if they are brilliant and highly capable, and even if you don't have an equally brilliant replacement. A mediocre replacement who can work well with others will be much more productive.
(Often said: About 15% of your productivity comes from innate ability, 85% from working with others.)
That having been said, if you're really worried about someone doing you in, make sure you have regular backups and that you personally have access to the backup system. Reformatting a disk and copying data is easy - position yourself so that you can recover completely from the maximum damage they can do.
Hi, This is one of the classic questions of insider misuse mitigation "who watches the guards". One way to deal with this is to use very good logging using a third audit party. Traditional audit/logging engines are not well suited to this task. You might like to take a look at LUARM (http://luarm.sourceforge.net/). It is an effort to provide very fine grained logging into your systems. The idea is you setup engines like that and your logs are then placed off-site and managed by a third party auditor, away from a potentially rogue sysadmin. Thus, if something happens, you have the means to prove what your bad techie did. Preventing this to happen is another story. Some people say that the knowledge of being monitored deters people from doing stuff. I do not support that view. Simply, my experience in dealing with sysadmins is that they are often underpaid, not appreciated and take all sorts of crap for other people. Make sure you pay them well, support them and listen to what they have to say. (a sysadmin) :-)
I'm sorry. I think I'm missing something and I don't mean to appear to be sarcastic, but would not changing the password suffice?
Make it so that you need to be two admins to delete backups, and log all access attempts? Or any such model, where you need to be two (or more) to take destructive actions and it's clearly evident in logs who those people where?
Emotions! In your brain!
How do you protect servers from rogue admins, they same way you protect passengers jets from rogue pilots, they say way you protect ships from rogue captains, the same way you protect buses from rogue drivers, the same way you protect trains from rogue engineers and even the same way you protect patients from rogue doctors.. You don't, any protection you put in place to protect a server from a rogue administrator will be broken by that rogue administrator if they are in any way competent. I suppose you could always seek to hire the most incompetent admin you can find a person who lacks the expertise to break the servers but somehow that seems rather pointless. So how do you protecct your servers from rogue admins, don't hire them in the first place. Consider a full psych evaluation (stay away from the anal types), pay a food salary and, make them part of the executive team.
Chaos - everything, everywhere, everywhen
Don't let clueless PHB's run IT.
Don't make so there only 1 guy doing the network admin
Don't ask for admin password over a conference call
What is you backup method. Many more things can happen than a rogue admin messing up files. Disks fail, equipment gets stolen, users accidentally delete items - all of which point to having a robust, redundant backup strategy. Absent that, rogue admins are the least of your worries.
We've kept rolling backups - i.e several weeks worth, on duplicate media. On-site for fast access and off site for ensuring its availability if something happens on-site. I know others that mirror the entire operation to another secure location.
My suggestion - figure out how much data needs to be backed up, how often does it change, and then develop a redundant backup strategy with teh ability to roll back several generations.
You can't protect against any and all employee actions, but at least you can make it hard to totally destroy your data.
Also - as others pointed out - find out why people leave mad and fix the underlying cause.
I'm a consultant - I convert gibberish into cash-flow.
Schedule weekly backups of your data and have the admin hand them over to you. Also any admin should be legally responsible for any damages inflicted on purpose. If you make that clear nobody's gonna bother damaging your data out of anger or revenge and risk being arrested
Much of life comes down to risk assessment.
If you don't trust someone with full access to your property you'll have to limit their exposure. Maybe cut the property (data, files, access) into fourths and hire three more admins. That way the most damage they could do is release/destroy 25% of your data.
Do you really think a rogue admin is going to destroy all your data? Release your images into the wild? Ask yourself: What is the damage? Can you minimize it? Can you recover?
I think you may have an over-inflated idea of the value of your property. Keep several backups, off-site. Hire people with a track record of doing good for other companies. Verify their employment and run a background check. After that, trust them but insure yourself.
Have new admins sign a NDA in which it is clearly said that they will not attempt to hack/perform wrongly... during and after their time in the company.
Slashdot, fix the reply notifications... You won't get away with it...
Push 'em hard, verbally abuse them, denigrate their efforts, and threaten prosecution of malcontents. That'll show 'em.
..Flickr?
Every suggestion posted so far mentions making extra backups, using third party software for audit and tracking to adding extra, bureaucratic steps into the mix that will do just that: piss someone off.
I'm a sys-admin my profession and even in the area that I live in, there are places (by word of mouth via networking or friends in the field) that just have a bad reputation when it comes to wanting to be a sys-admin there, which lies almost 100% on management. I can almost guarantee this non-profit organization either has some really idiotic management or simply under-mind the talent and expertise they brought on board (e.g. the admin) to do the job and think they have better solutions. Most of the time, people prefer to work in a smaller shop because you get that flexibility to do outside-the-box work, set things up the way you want and push the limits of your resourcefulness. Ya, the pay/benefits might be lower, but your flexible schedule, stress and environment are probably more than ideal.
Can't be done, you're screwed, move on...
[b]Rogue[/b] admin will come stealthed in the dark and stab you from behind. * shadööwwwwww * (skillz)
Read radical news here
The only real "protection" against rogue admins is to have multiple admins who can monitor each other and (if required by audit) sign off on each other's work. Most organizations of any significant size have more than one person at the top, so that (at the very least) if any one admin is sick or leaves in a huff, one or more of the other's can take his place and/or revoke what permissions that admin had. This can take some forethought to prepare.
...En að Besta Sem Guð Hefur Skapað Er Nýr Dagur
No matter what solutions you use for backups, the admin will be able to corrupt or bypass them in some way given enough thought and motivation.
However, for sane though disgruntled people it would be sufficient for them to have the common sense understanding that malicious actions will have strict consequences - people generally don't risk going to jail just to annoy a manager or company. And in the cases where someone would really be prepared to risk that, I'd rather worry about them coming to office with a gun, not tampering with a pile of pictures.
What was the aftermath of the previous cases you say of people leaving in anger and presumably doing something damaging? Your previous reaction in these cases forms the expectations in your admins about what they can get away with when leaving in anger.
If the rogue admin can't access them he can't break them!
How do you protect your servers from a rogue asteroid?
I see there's escalating levels of access, but it doesn't sound like those levels are tied to law. They probably should be, i.e. it's not so much file size as whether the file is about an adult person or a minor, whether the file contains medical information or not, and such things that should be the first consideration in defining those privileges. A single dental photo sounds like a small image under your definition, but its treatment depends on HIPAA first and foremost, never size or image format.
Who is John Cabal?
I just finished all eight seasons of 24 in a period of two weeks, so I feel I'm qualified to suggest that you hold said admin's family hostage and then use enhanced interrogation techniques in the event that he fails you.
Before you get to any details, there's a sort of logical problem in protecting against admins: Who are you going to get to set up the protections? If you hire me as an admin and then ask me to secure the network against myself, there's nothing to prevent me from putting in some kind of alternate access (i.e. a secret backdoor). If you hire someone else to secure the network against me, then there's nothing to prevent that person from keeping some alternate means of access. That's before you even get to the problem of an admin securing things against himself which he'll need continued access to.
It's a difficult problem, and there are things that you can do to mitigate the dangers somewhat, but ultimately if you're not able to handle the security yourself, then you're going to have to trust someone. Make sure you hire trustworthy people, and maintain good relationships with them. If you are able to, make sure you have 2 IT people who keep each other informed about security issues and changes in configuration. That way, if you have a less-than-amicable break with one of the IT people, the other can help you lock him out.
Send regular backups tapes off-site to someplace like Ironmountain. Only give authority to retrieve tapes to collection managers and/or company executives, not to server admins. This also protects your collection in case your office or coloc goes up in flames.
Keep at least 6 months of tapes off-site so you have 6 months to discover a time-bomb or hidden corruption left behind by the rogue sysadmin.
Test restores regularly.
If you truly are concerned about the trustworthiness of your systems administrator; you definitely don't have the right person in place and you need to take steps NOW to ensure the continuity of your systems. Start implementing strict documentation standards for everything - passwords, system maintenance procedures, run books, network diagrams, etc... This information then needs to be stored in location accessible by senior executives and audited by an external firm to ensure completeness and validity. You have to be careful about this though, because it can be a tip off that the administrator's tenure is coming to a close shortly. It can be very costly to have your admin walk off the job with all the passwords. Your systems will be unmanageable and if the passwords can not be recovered by a forensics firm, you'll have to wipe and re-implement the affected systems. Better to have a discussion with all employees and say that the company has come under regulatory scrutiny, or some other excuse, and that all departments must now thoroughly document everything they do. Then everybody is on an equal playing field and employees are less likely to think more into it.
As far as backups go, bring in an external firm to configure, perform, monitor, and audit the backups. The best system would be an off-site mirror of your data center managed by this firm. But tape archives can be effective as well. Either way, your administrator would be discouraged from tampering with the backups, as an audit would immediately show any attempts at sabotage. But even with backups, you could be talking about days of downtime before all systems could be restored, so best to fix the human problem first before you even get to this point.
I went into a local community college with a team of talented system engineers after they were forced to fire their hands-on IT manager. They neglected to get typed and validated documentation from him before they kicked him out, and it took us days to crack all of the passwords and document all of the systems and networks. I estimate it probably cost the college at least $20,000 for this forensics work because they didn't handle the situation properly.
Don't worry about your infrastructure so much. Having been in this position, I noticed that companies seem to worry quite a lot of it.
But it seems to me that it's an unlikely situation. Let's suppose there's an admin really pissed off at you for some reason. What could they do to your photo collection?
All those options are pointless and ultimately suicidal for the admin involved. All you need to do is to have readonly off-site backups (which you should have anyway, what if the building gets flooded or burns down?). If properly done the rogue admin can't screw that up, and while the things above might hurt, they'll be perfectly survivable. Even the torrent isn't a big deal. A serious publication isn't going to touch an illegal collection with a 10 foot pole. As a public organization they're an easy and profitable target.
However, those things are terribly stupid and suicidal for the rogue admin. Who will be the first suspect in line when any of the above happens? The recently fired angry admin. Law enforcement treats such things harshly, and word of mouth gets around and it's unlikely they'll get another job after that.
All the admins I've seen leave (and I took note and did it myself when leaving a job) tried to leave in an as non-threatening way as possible. For instance on my last day on one job I discussed with a coworker what I had been doing, where the files were, what was unfinished, the lists of passwords and access control methods to be changed, etc. I did everything I could to make sure that nothing in my departure could be interpreted in a "screw you" of any kind, and to make sure my successor could take over.
Now, what should you be worried about? The legal ways an ex-employee can screw you over. For instance, the BSA. It's easy to report to them. From what I hear they're most eager to show up, offer rewards to the reporter, and it's very hard to deny them entry. And I hear that their visits can be very expensive. So make extra sure you're in perfect licensing compliance (which is pretty hard), or switch to Free Software.
You could do it the same way they protect the nuclear launch sequence. Nothing can be done unless two admins do the same thing at the same time from different locations. Software changes left as a exercise.
At a small company I used to work for ("used to" being the key phrase here), the bosses, who both insisted on full admin rights, had a bit of a difference with each other. One of the bosses came in one Saturday night, killed the backup (they never took my advice of having multiple backups, including one off-site), and ran off with the server.
I tried recovering the backup, but he did a remarkable job in killing it.
The company didn't exist for more than a week after that.
Does it make you happy you're so strange?
Not to be confused with the rouge admin, who is rather less stealthy.
I've worked at a few non-profits where I was the only server administrator and I know the hardships of pretty much no budget. One place I worked for had a yearly IT budget of 1500$ a year, which wouldn't even cover my visits throughout the year. Anyways, one of the CEO's I worked for was paranoid about losing their data due to server failures/the building burning down, or whatever. We had daily onsite backups, but there was obviously no money for offsite. The solution I came up with was that every night, all the media was backed up to a portable USB hard drive, in addition to the regular backups, and the CEO would carry it home with him every night. Then in the morning he would plug it back in for the next days backup. I set it up to shoot him an email every time the backup finished, and hes been doing this for years. I'm not sure if this fits your scenario, but maybe it will spark some ideas, good luck!
The best you can probably do is the two-man rule: high-level commands can only be done with two admins around.
http://www.google.com/search?q=two-man+rule
This only goes so far as it runs up against the Ten Immutable Laws of Security. Specifically number six:
* Law #1: If a bad guy can persuade you to run his program on your computer, it's not your computer anymore
* Law #2: If a bad guy can alter the operating system on your computer, it's not your computer anymore
* Law #3: If a bad guy has unrestricted physical access to your computer, it's not your computer anymore
* Law #4: If you allow a bad guy to upload programs to your website, it's not your website any more
* Law #5: Weak passwords trump strong security
* Law #6: A computer is only as secure as the administrator is trustworthy
* Law #7: Encrypted data is only as secure as the decryption key
* Law #8: An out of date virus scanner is only marginally better than no virus scanner at all
* Law #9: Absolute anonymity isn't practical, in real life or on the Web
* Law #10: Technology is not a panacea
http://technet.microsoft.com/en-us/library/cc722487.aspx
Of course having the "overhead" of needing two people runs into number two of the 10 Immutable Laws of Security Administration:
* Law #1: Nobody believes anything bad can happen to them, until it does
* Law #2: Security only works if the secure way also happens to be the easy way
* Law #3: If you don't keep up with security fixes, your network won't be yours for long
* Law #4: It doesn't do much good to install security fixes on a computer that was never secured to begin with
* Law #5: Eternal vigilance is the price of security
* Law #6: There really is someone out there trying to guess your passwords
* Law #7: The most secure network is a well-administered one
* Law #8: The difficulty of defending a network is directly proportional to its complexity
* Law #9: Security isn't about risk avoidance; it's about risk management
* Law #10: Technology is not a panacea
http://technet.microsoft.com/en-us/library/cc722488.aspx
If you can't figure out how to use the two man rule you shouldn't be in charge of backup solutions or administration of anything that is considered system critical. Read about it.
Get a web developer
Privileged Identity Management software exists that helps to solve this problem. Investigate.
While it is not yet standard practice, there is absolutely no reason why your server cannot be completely under version control. The only point of contention is the password/groups file. Aside from that, you should be able to use something like TinyCoreLinux to get a minimalist boot image, with a version control system, (SVN, CVS, etc) configure the version control and save that image. Then once you boot the image, you issue a get/sync/update command which gets the most recent version of everything.
TCL Linux will with slight modification of the filetool script, give you a way to automatically check your changes in. Once they are in the source repository you can then have a reviewer review the changes and push them to the approved main/head branch.
The only way a hostile administrator can attack this is by moving things out of the filetool script purview. In order to overcome this vulnerability, you must re-image your server periodically based off the approved mainline/head branch. Any unsubmitted changes will be lost. To do this safely, load a new VM or real hardware until the new image is verified that nothing is lost. Then move the old hardware to spare, and use that for the load in the next cycle.
Slashdot's rate-of-post filter: Preventing you from posting too many great ideas at once.
Back it up to the cloud, sites like www.evacloud.com have automatic versioning, unlimited storage and discounts for non-profits.
was this exact scenario. NDS (and possibly other directory services) has a concept of an "Organizational Role" which is the source of the privileges, rather than the actual user him or herself, and the user's account in the Tree is given the "role" of ... say, "Admin." There wasn't any privilege outside of that role, the user accounts were all pretty well stripped bare and derived all ability to function from the role they were said to "occupy."
How does that help? Well, if LDAP or some other free-as-in-beer-and-speech directory service will allow your organization to control that level of access better than granting superuser/sudo privs to particular admins, who could in theory leave behind shadowed user accounts, that might be something worth looking into. I haven't been a NetWare admin in several years, and haven't followed their current progress with NDS, but I do recall that for a while there was a version of it that would sit on top of Linux/Unix as well as Windows and Mac workstations, and Linux/Unix and Windows servers, and could be managed from most of them as well.
Only two things are infinite, the universe and human stupidity, and I'm not sure about the former. (Einstein)
First, you need to stop drinking the coolaid. You are paying the sys-admin to keep your systems up and running. They do have "the keys to the kingdom", because you are paying that person to hold them. If you don't trust that person to hold the keys, then you shouldn't have hired them in the first place.
The ways you mitigate the issue of "rogue" admins, is vet them, listen to what they are saying in terms of technology, don't micro-manage them, and pay them well. The good ones without a doubt will know the technology better than their manager/management structure will ever know it. The reason the admin says something about the setup/configuration/technology is almost always because it is needed change. If you can't afford to make those changes, then you need to explain that is the reason, don't make up some BS about how you want things to stay the way the are, or you want to change the organization/structure to something else, because they will "call" you on it. Again, they know the technology better than you ever will.
The other thing to do is to pay them appropriately. You are trusting them with running some of the most complex systems in your entire company, as well as safe-guarding your data, your processes, and your daily operations. The reason why you don't see many rogue CEO's is because he/she is being paid well to run the company, choose its path, and steer the ship, so to say. The system admins in today's information based businesses are the guys keeping your entire company running. If your servers/data were all destroyed, and your business would not survive, then you might want to consider paying the people who keep that data/servers a more appropriate amount of compensation since they are so vital to your business.
Again, there are very few admins who go rogue, and even fewer who did not do so after being mistreated by their bosses/management. If people want to point out at the case of Terry Childs, they need to get a clue. Were mistakes made, sure. Did Terry have some issues? Yes. Did he actually go rogue? No. In his eyes, he was protecting the network from idiots and incompetents, and following the rules as currently defined. He wouldn't give out the passwords in a room of strangers, over the phone, or via email where it can easily be intercepted and then misused, as well as be cause for firing him because policy stated not to do any of those things. So he was placed into a situation where he would be fired if he handed out the passwords, or fired if he didn't. And once fired, he really had no obligation at all to give it out anymore, why? Because he didn't work there. Same as if you fired your top salesman, or stock broker, or process manager. They don't have any obligation to tell you anything about the contacts/client relationships/methods for picking stocks/how things work. If you fired them before you obtained that information, then you should have been fired. In the Childs case, were they trying to obtain that information, sure. But in the wrong way according to policy. They should have taken Terry into a one on one conversation, in a private room, with no one the phone and asked in that setting. Even then, he might have refused to have the manager have the password because the manager didn't have the knowledge or skill to know how to properly vet someone as being capable of having the password. The only thing that would happen is that it will cause someone to screw up the settings and create work for Terry since he will be the one called in to fix it, and most likely not paid for that extra time he had to spend fixing someone else's screw up.
Again, it comes down to properly compensating the admins, listening to them, and not trying to play office politics with them. You treat them well, and they will do whatever it takes to keep the systems running because they take pride in their work. You treat them like crap, blindly disregard their expertise in terms of operating the servers/network because "you know better than they do", you are asking for th
We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
If you have workers who are leaving in anger then you have a organizational issue that security is not going to fix.
Got Code?
then you might as well pack up shop right away. Is there nothing else that might go wrong with devastating consequences for the organisation? Really?
Never forget that admins are people too, and if they are a problem, then you have a people problem on your hands. Do not delude yourself in believing technology can solve this. It might help, but it cannot by itself solve the problem.
You have to trust someone and that's your admin. So if you can't trust someone, don't make him admin. It's that easy. Then again, if you treat everybody like you don't trust them, they'll run away. Especially in a non-profit. Why are you in that business again?
Lay off the whole in-house IT staff and use consultants (but only if its cheap). NPOs are imploding right now, and for many tech staff is not program, so they want their tech without the cost of support, but also want the benefit you only get with proper maintenance and planning.
Figure that out.
My thought on the OP is the boss plans to drop a lot of their IT staff. My perspective is most of the IT guys won't be looking back (NPOs usually pay lousy but as a benefit you gain tons of marketable experience doing everything than you would have in the private sector).
step one don't piss off your admin. Step two Don't Piss off your Admin
As a side note a good step three is the have a co admin who hates the admin that works well and is mostly entertaining to the bystanders.
Is there a back-up, site mirroring, privilege, or other strategy you'd recommend so we have protection from an Admin gone bad?
Sounds like you already have a technical solution for cleanup. If it were me, I'd have two locked server rooms, and each sysadmin is only allowed into one server room. Each room has half of the original servers, and half of mirrored servers from the other room. The mirrored servers rsync from the original servers regularly, with a resticted user account with sudo access only to rsync (plus the options in /etc/sudoers to restrict rsync to only backup particular directories, otherwise it could overwrite /etc/passwd and /etc/shadow on the original server).
Also, a third offsite place to store lots of long term backups.
In other words, if you want a technical solution to a simple HR problem, you're looking at spending a lot of money. I would suggest instead to keep your sysadmins happy, either with flexible work schedules, firing PHBs who infuriate them (or putting a technically competent middle-manager in between the idiot and your IT staff), or increasing their salaries slightly (less than it would cost to double/triple your hardware expenditure).
Disclaimer: I am not a sysadmin It seems to me that your best bet would be to distribute authority. Does the guy in charge of email need admin for the webserver? etc. Look at it from the perspective of a hacker compromising an admin account, pitch it this way and the admins will likely be able to help you. Limiting an admin in the range of damage they can do before they become disgruntled is the key. Obviously you can only take this so far, and it will likely make thing more difficult for some of the admins, but if you are really concerned about rogue admins the head ache may be worth it.
...there's actually a pretty easy method.
Simply set up a file server somewhere that the admin do not have physical access to. Setup a server in a locked office. Put it in the president's office, it makes him fell important. (Of course, don't get him any login to it or even attach a screen.) It's so simple and does so little, you don't have to worry about overheating or anything.
No remote login or anything. All it does is have one file sharing point (SFTP or something), that gets logged into and files uploaded. Presumably every night, when the backups run.
Then, once a day, after the backup will be finished, the files are automatically moved to some other, remotely inaccessable, timestamped directory and directories older than a month are deleted, and it emails out what it just did.
It's something you literally can set up in thirty minutes, on 'non-server' hardware. Grab some hardware you're throwing out, buy a new, large hard drive, and throw Linux on it, and spend two minutes writing a script to put in cron to move the directory and delete old files. (Five more minutes work with rsync can result in you hardlinking the unchanged files and saving space.)
Don't worry about 'restoring' from the server, or how to access the files. If shit goes horribly wrong, you'll have to physically go to the server and copy the files somewhere else, or open up remote access, but shit should not go that wrong, ever.
All server admin should visit it in pairs, if they need to, which they shouldn't.
If corporations are people, aren't stockholders guilty of slavery?
you can be angry without being unethical
Hire competent people. Treat them with respect. Pay a competitive living wage.
Ultimately, you cannot be sure you won't get screwed, ever. Not even by hackers outside of your organization, let alone ones inside. It is possible to -- reasonably -- secure a system using methods described above (offsite backups managed by a third party commercial affair, onsite backups under lock and key, careful logging and so on). However, in nearly any network there is one toplevel admin that doles out the privileges and so on, that set the system up, that works on the system many times more often and at a much higher level than the people that typically have permission to do a few things enabled by sudo. There, no matter what, you will be vulnerable.
/. and it is going to cost you money). Basically, the more you try to secure things on the cheap, the more likely it will be that you have a setup with holes you can drive a truck through given the root password and access.
This is a classic problem: Quid custodes custode (who will guard the guardians)?
Paradoxically, you are probably slightly safer if your admins are not uberkinder supergeeks. If I, or any of a dozen people I know, were your toplevel sysadmin and was not the completely honest and trustworthy person that I am, there is no measure you could take for protection that I could not suborn in such a way as to cause you great pain and loss. After all, who would be implementing the measures? Log files are pointless ways to reveal the activities of the person who set up the logging system. Subtly corrupting the backups for long enough to roll over the offsite images (which could be as simple a measure as installing an encrypted filesystem "for security reasons" and making sure that I'm the only person that has the real key). An amateur (or less skilled professional) is less likely to know enough to do dirt and hide their tracks.
There is no real protection against hiring people to do mission critical work of any sort who have a serious personality disorder. So your best protection of all is to hire toplevel systems staff who are, as far as you can tell looking hard, completely ethical and personality disorder free, and then treating them with respect.
Good advice for keeping ordinary employees from going postal, good advice for any organization or task, actually.
There is one more solution -- the NSA sort. Throw an enormous amount of money at it, and hope that the people you hire aren't smarter than the (unknown) one you are defending against and that they leave no holes in what they set up. Hiring ten top sysadmins all tasked with watching each other is good. Having commercial consultants who know what they are doing help you set up a system is good (in other words, if you have to ask the question you need to get an answer somewhere other than
rgb
Even when the experts all agree, they may well be mistaken. --- Bertrand Russell.
A rogue admin will create a back door before they leave. Often they will do this midway in their career to try and ensure continued employment, but that would never work out. Eventually they will be found out. All "Good" admins realise this, so it shouldn't be an issue. Just try to ensure you hire "Good" admins. Personality tests may help in that venue, but history of previous actions taken during "stressful" times may prove to be a better indicator of how they will behave in the future. People often repeat bad mistakes if they don't realise that they are the ones making the mistakes.
maybe not treating your admins like shit would be a good start?
its kinda like not pissing off the person who serves your food, just common sense...
...there's actually a pretty easy method.
I really thought you'd say "put a gun on their familly's head and say if data is gone, they're gone too".
But then you started writing about something not as easy!
I also thought that Duplicity should be mentioned. It uses librsync, its dead easy to setup for backups and supports everything you can think of (encryption, deltas, recovery per time period, various upload means going from regular copy to sftp, scp, and the list goes on for a while)
Excellent tool http://duplicity.nongnu.org/
The protection you're looking for is called "prison." Wiping the files on the way out would be like burning down the building or returning with a gun. It does, rarely, happen and those folks go to jail. The only things you can do about it ahead of time are: treat your staff well and, when it is necessary to fire someone, make very sure that both their privileges have been thoroughly revoked and you have a current, tested backup.
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
The answer to this question is incredibly simple. Backups. If you really don't trust your sysadmin, or simply want to audit them, there are lots of options, like having the backup software email backup reports to both the sysadmin and the executive director. Store them off-site with your chairperson or executive director. Or hire a company like Iron Mountain, where everything is audited, access is controlled, and only certain people are allowed to request tapes outside of the normal rotations.
Please help metamoderate.
Sysadmins generally don't start out with malicious intents but tend to take pride in a well managed environment. Undermining them by consistently ignoring their expertise or restricting their ability to effectively do their jobs due to corporate red tape or any number of other reasons could escalate dissatisfaction and result in a rogue admin. So the first thing is to ensure that your sysadmins are content. Note that going rogue doesn't happen over night. It's something that builds up gradually until it reaches a tipping point. So things could be occurring well before anything has been detected.
There are several things to consider when dealing with a rogue admin:
- can you recover the data? i.e. do you have backups?
- can you be certain that this data has not been tampered with?
- are there any processes in place to validate backups?
- what effect will an admin copying sensitive information (trade secrets, credit card info, etc.), source code, databases, etc. offsite have on your company?
- do you have systems in place to detect and alert on suspicious behaviour?
- can you ensure that these systems haven't been tampered with?
- what do you do if the admin installs keyloggers or replaces programs on employees systems with hacked versions to retrieve passwords or access keys to other systems and networks (e.g. personal email and networks, encrypted files or network shares)?
- how do you detect backdoors in either software or the network?
In most cases, there are few recourses available to prevent a knowledgeable admin from wreaking havoc if he so desired, especially if he's covering his tracks well. Detection is key, whether that be systems monitoring for suspicious behaviour (not tamper proof) or routine security audits by other admins. The earlier the better of course. Separation of duties may also help a bit, but is easily circumventable if the motivation is there. Outside of that, keeping your sysadmins happy is the surest way to ensure that this doesn't become an issue.
3 or 4 hungry pit-bulls in the server room should be enough to take care of your average rogue admin ...
Cool, then all they have to do is messup the backups before they get to that server.
http://www.youtube.com/watch?v=rX7wtNOkuHo
Well, to make the same point in more sober fashion, here we see an important boundary condition of the security problem. Somebody has to build and maintain these systems. Ultimately, the privilege necessary to do that is the same privilege necessary to defeat any security measures they might embody. That's the state of ultimate guardianship. And then we must ask, as Juvenal did a couple of millennia ago, who is to guard the guardians?
The answer, though imperfect, is as good as anything we can devise for other responsible functions. The senior system manager reports to a director or officer who - if they're any good at their job - mediates between the requirements of the organization and those raised by the system manager. It is, without question, a trust relationship, but it's constantly being informed and refreshed through working experience. It's not blind trust. If there's something wrong with this component we call the "senior system administrator" here is where that would normally be detected.
The Terry Childs incident is a recent example where the relationship itself was broken. This would be a failure of senior management. And who watches them?
Parity: What to do when the weekend comes.
Where I work, we have a system of backup where Microsoft DPM 2007 is used to backup to drives, and Norton Backup is used nightly to back up to magnetic tape. DPM runs in auto mode and keeps 7 days or so of records, two replications a day. Domain admins have access to DPM, but we have a separate individual with no machine rights swap tapes daily. They have access to the physical machine (which is by itself, in a different part of the facility than our data center). In addition, the tapes are not in the same room as the backup server, but in a locked cabinet in a secure office. The admins could get access to the tapes if we wanted to, true, but its one more layer of security. We also create one to two WORM tapes a year, of a full data center backup, so even in a mega-catastrophy, we wouldn't be totally wiped out. The whole system runs like a well oiled machine because all the players know their roles.
I have always marveled that there are so *few* rogue admins. In most companies we have access to *everything* -- Employee records, financials, AP/AR, and whatever anyone is doing on any PC or server. Yet reports of rogues intent on stealing or causing damage are rare. It usually goes the other direction -- the rogue takes action out of a concern that management isn't taking security seriously enough. I bet you could think of a few examples.
The question I would ask is: What is it about the job, or about the people you hire for the job, that causes the great majority to take their responsibilities so seriously?
Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
Because someone is in a position of trust, with privileges raised to do their job, doesnt mean you cannot do anything if the trust is breached.
You need to account for the commands and time spent on a box that an admin might do, so that if there ever was a breach of trust there is sufficiently strong logs to detect how and when and what happened. If people know that their work is (if needed) being recorded theres less incentive to do damage that might be criminally motivated.
You also need to detect and be reported of activity that would not typically fall within the boundaries of an admins daily routine (such as deleting large quantities of files perhaps, or execution of of programs (like shred) that you wouldn't typically use.
You have not mentioned the platforms you are working with, or if your talking about a platform - or just some CMS but Linux for example has audit, you can set this up to monitor virtually anything you might want to watch for. It takes a little more creativity to audit from a thresholds perspective (where work is permitted but too many events at once is a threat) but it can be done. Audit can be locked once you've finished setting up the ruleset meaning the box needs to be rebooted for you to change the ruleset at all.
There are also pam modules for linux (like pam_tty) that can log literally every character a user pressed into their terminal (including non-space characters like escape and backspace) which can be useful to help determine the impact of incidents that you might be after avoiding.
SELinux on Linux on newer distros (typically thinking enterprise linux 6) has flexible support for role based access controls, which can further restrict an admins abilities exactly down to least privilege needed to do their job. Learning SELinux to the extent you can really do this efficiently might be a commitment though you might not have the time for - although I certainly recommend learning about Mandatory Access Control policies, especially for situations like this.
Transport these logs to a remote machine, if necessary one nobody has access to without some form of local authorization (like using pam_usb). Theres no point doing logging just on the audited box that a potential admin has access to.
Detection can be more difficult. Prelude is a open source security application that offers some stuff you might find of benefit here, other than that rolling your own scripts might help too - depending on your skills and experience in such things.
Finally, and more importantly - people who are given positions of trust like this should be trustworthy. This is purely a management problem, but screen your guys effectively. Dont hand the keys to the city to some bloke you pulled in off the street without doing at least some background checking.
Publish all the database information?
One can cause massive damage to the organization without destroying data. Simple breach of security is enough if there is any private data (e-mails between staffmembers? or from supporters? any financial information in the databases? is there a reason why some of the supporters wouldn't want their name connected to the cause? etc...). No amount of backups protects you from that, now does it? So stop acting like an arrogant prick. ;)
Yes. If you keep proper logs, you can sue the admin later on. But we are talking about things that people do in fits of rage (when alcohol could well be involved) so that might not prevent things from happening and once the damage has been done, it has been done.
The first mistake is assuming that when an employee leaves, he/she isn't leaving in anger. How does the outfit know whether there is anger or not? Also, anger isn't always the only motive an admin might go rogue then up and quit. Mental illness. Drugs. Personal problems. Who knows. They certainly aren't going to announce "hey I'm angry because [enter reason here] and I'm going rogue then quitting".
The data is an asset, just like anything else in the company, and needn't be treated differently just because it's digital.
In my personal experience, mission critical data has always been backed up and kept off-site, usually with the "big cheese" - the person with whom the buck stops. How often is the answer to a simple question: "How much of this can go missing before trouble starts?"
If the answer is "none", the solution is mirroring - real time backups all of the time.
If the answer is "a little but not much", full backups and prescribed intervals with incremental backups filling in the gaps can be considered.
If the answer is "pfffft, doesn't really matter, just not too much", then a manual backup at set intervals would suffice.
The danger here is not the finding of an adequate solution.
The real danger is assuming an employee is/will be leaving on good terms and isn't intent on causing damage.
Assume the worst always, and don your teflon. When Murphy's Law strikes (and it will), you're bullet proof.
FWIW, I have a standing policy - when I accept a notice of termination from an employee with adminstrator[-like] privileges, I say thank you for service and escort them out the door. On the spot. No exceptions.
Mike
-- Karma whore? You betcha. --
There will probably be conflicts between IT and the less technical members of your company. Make sure that your admins feel they can report these conflicts and get them fairly resolved. If people are leaving your company in anger on a regular basis, it sounds like you need to establish some better policies and hire some HR people who can allow your staff to operate smoothly.
Keep in mind that clear policies and good security protects your admins as much as it does your company.
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
Who watches the watchmen?
90% of all serious damage is caused by normal employees not admins. An admin gone rouge has 3% chance for getting work as admin again, he/she is simply not trustworthy anymore.
One of the best things that you can do, in my opinion, would be to take away their keys and escort them out, making sure they return any company equipment they have. I'm not a manager myself but once they know they have no more job with you, if they want revenge, they use their credentials to get at whatever they can. Disabling those credentials would probably be your best bet.
In a former job, I once found a tape in the back of a cabinet. It had a locking device in its hub which needed a key. It had a label that said "Original System Files, See Lab Director for key". You don't need that, you just need to occasionally put away some backups on physical media
Yes, if you have admin behaving maliciously for a month and no one notices, you'll be in trouble.
Of course, if they can do that, you're pretty much fucked anyway.
If corporations are people, aren't stockholders guilty of slavery?
If an admin deletes all the files.... how quickly do you need every single photo back?
The longer your business can live without them, the less expensive a solution you will require, and the more reliable a solution you can pick.
Some of the least expensive solutions are.... burn every new file to DVD and backup every new file to tape or traditional film. Translation: backup as you go.
If you want your collection to survive thermonuclear war and EMPs, then record every frame to film using a Film recorder; have the roll fixed and developed, and lock the film up in an underground bunker, in an airtight safe with minimal humidity.
For faster recovery, you will need regular full backups. Lock them up in different places. Make sure to never ever reuse a tape. Always use fresh media for every new backup.
Yeah, that's something that could probably handle the entire backup, and just, at the end of it, copy it somewhere else.
Hell, if the backups are encrypted, that's one less thing to worry about on this server.
Then you just need a script like in cron:
mv /backups/shared/ /backups/`date +%m-%d` /backups/shared/ /backups/ -maxdepth 1 -type d -ctime +30 -exec rm -r {} \; /backups/ |mail admin@example.com
mkdir
find
ls
Or, you know, something a bit more complicated, but essentially that. Just moving the backups out of accessible areas (Which sadly means you can't use the backup program's expiration of backups.), and not having any way to log in remotely at all.
You could make it more complicated, but as this is the emergency 'crazy admin destroyed all the backups' setup, it's unlikely it's going to be used that often. With the sort of small organizations using a setup like this, it's going to take several days to rebuild the servers anyway. The fact the backup is not immediately accessible is fine.
If corporations are people, aren't stockholders guilty of slavery?
Executive Summary: Separation of Powers. Make sure that the person in charge of your servers is NOT in charge of backups and auditing. You want an environment that permits "no malevolent activities without collusion."
I worked for a bank auditing company for a while, and installing anything (or any administrative work) was a pure PITA. There was a mandatory "four eyes" principle in effect. Logging in without a second person (every admin login caused a text message to go to all admins, just in case you're wondering whether nobody did it "stealthily") was grounds for instant firing. You would grab a fellow admin (or, if nobody was around, anyone who could "supervise"), fill out a form that you and him are going to log in, then you started a protocol (pencil and paper type) of what you are going to do. Every keystroke, every click of the mouse, was to be written down, then executed. Installing a program or an update by protocol could well take an hour or two, and certainly not 'cause the machines were slow. Termination was told to you the moment you were let go, the same moment two admins were sent with high priority order to revoke your admin privs. On the upside, you were let go instantly, i.e. take your stuff, do not log in, you may spend the rest of your working days at home (i.e. effectively another 1 month of paid vacation). If you had to clean up anything on your machine, two admins did it for you.
This is a level of security and paranoia that borders on insane. Personally, I'd say it's a wee bit beyond insane already. But it gives you an idea that banks tend to take security and the threat of rogue admins VERY serious.
But there is one thing you should definitely do when firing an admin: Revoke his admin privs INSTANTLY the moment he learns that he is gone and send him home. Even if laws demand that you have to tell him 2 weeks before firing him, send him home on 2 weeks of paid vacation. It's cheaper than the threat of having him do something to retaliate at you.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
To be serious about security, you have to eliminate every last single point of failure. Although I seriously doubt a non-profit would have the cash to justify paying rather than simply trusting, if they were serious about limiting the damage an admin could do, they would outsource the backup, requiring that the backup be regularly monitored for suspicious changes and tested both by the outsource and by someone within the company.
Admins are like aircraft pilots. The bad ones don't get hired.
see this: http://ask.slashdot.org/story/11/01/10/1418249/Disempowering-the-Singular-Sysadmin
Correct. In practice, make sure periodic backups are kept in safe storage (to which the admins must NOT have single access: make
sure at least one other person is there for them to be accessed) for relatively long periods - say, a couple years anyway.
If anything happens to a site that is not detected, it may have unwanted functions embedded for a rather long time, and they will
be in the backups also. Imagine the kind of "dead man" switch that has been done before, where some code looks for the pay number of the rogue admin, and crashes if it is missing.
Such a thing can be in place for years. You will want some way to get a system, even an old one, going again that lacks it.
Key logs of what admins do, or other detective controls, are potentially useful, but remember that if the admin knows about them, and the admin is at all competent, he can avoid them. Simple key logs also are not really telling. I could for example make up some commands that would make a normally harmless command .(say, "df" in unix) execute all manner of destructive results (and perhaps for good appearance put out the results of the real "df" command as well). Unless your log captures what is actually DONE by commands issued, you have only a poor record from key logs. Remember, these logs get huge, and if something strange happens, you will need to find where the command was in the logs. It can be very hard to find out from keystroke logs, where what was typed can look innocuous.
That is why backups are needed, stored in secure places.
To avoid enormous storage, it is possible to arrange that most of these backups will be of areas that have been written since the last
backup. This can be done at device level if you like; it is much less efficient at file level. However you still need full backups periodically
so the state of the system can be rewound to some prior time.
I once built a system that would keep such logs, timestamped to a granularity (I used 15-20 min.) and sent to remote storage,
that would allow a physical image of a disk to be put back by multiples of that interval. It needed only linear scans across the disks
to put anything back, but did require writing both to the target and to a timestamped log. (This could be done to write once devices so that it was not tamperable.) Note though that the recovery in such cases gets a little involved.
The less likely this is to happen, of course, the longer you might be able to go between backups. But have enough so that the system can be brought back without destroying the organization. Sometimes people just go nuts. Also sometimes, some nut decides to hold an admin's family hostage or otherwise force actions that would not normally be done.
Finally, it should be presented to the admin that having such backups is also a way to save the admin's hindquarters if he makes some destructive mistake. Accidental deletion of important things need not be forever if there is a procedure to get them back (albeit with admission to someone else that you did it). The safety net effect is one that would tend to reduce the temptation of admins to disable or circumvent the system.
Also, be sure and check now and then that your backups can be restored. I have known several occasions when an attempt to restore a backup found it had errors, or data just wasn't there. Happens sometimes with flaky hardware, sometimes for other reasons. The above "undo screwup" function will tend to exercise this now and then, but make sure it is done in any case.
The answer is simple, but it's also fairly unpalatable.
1) Hire an admin (team) to manage and audit your admins and to implement their own failsafe measures to ensure that errors with their subordinates can be quickly rectified with minimal downtime.,br>
2) Repeat #1 ad nauseum.
The bottom line is that the security of your assets will always have a lynchpin of trust somewhere. Even if you decided to automate administration with some incredibly intelligent AI, you'd have to trust it, or it's developers, or the people that certified them to be trustworthy, or the people that gave them the authority to determine trustworthiness, etc.. It all boils down to how much you're willing to spend vs. what you consider to be a reasonable level of risk (which is incredibly difficult for anyone to define). In addition, anyone that wants to live in a world of total assurance needs to pass the bong pipe along.
this new, cutting-edge, concept, involves duplicating several times your important data. Backups are:
- off-line, so a single event (virus, hack... ) cannot wipe both your live and you backup data
- off-site, same reason with fire, water, theft...
- multiple, 'coz Murphy's law is twice as effective regarding backups, and data corruption may take days-months-years to surface, so several backups from different times/dates are required, a few of those archived long term.
- tested, 'coz it's rumored that sometimes things don't work as advertised.
I'm working on popularizing the concept, and trademarking the name. I can tell you more about this advanced technique for a very reasonnable fee, considering.
The Cloud - because you don't care if your apps and data are up in the air.
They have the keys to everything and could burn the place down at night. The chances of it happening though are not much more than getting killed by a block of green ice falling from a malfunctioning aircraft toilet.
A rogue sysadmin is in the same basket but even less likely because there are less of them. Also if they are any good they have probably solved that one for you themselves as part of the "what if I get hit by a bus" planning and a disaster recovery scheme (passwords in somebody's safe and offsite read only backups).
Of course just as you take the keys away from an ex-employee it's a good idea to change the passwords and restrict remote access when a sysadmin has gone (unless it's a steaming heap of shit like old versions of MS Exchange where changing passwords breaks it entirely - how's that for a lack of testing and quality control).
Everyone is commenting on the nature of this question and not addressing or answering it at all. Politics and BS aside, it is feasible for any organization to have this problem, and any management should be rightly worried about this type of event occurring. In fact, to attain certain technical certifications as an organization you MUST have systems in place that will deal with this type of internal poisoning.
As an experienced system administrator I setup a few systems that do exactly what the author of this question has asked. Here are some options and ideas...
#1: Physical Backups (AKA, offsite tape backups). Nowadays it's rather overly expensive to actually have a tape drive and write tape backups, so a lot of companies just use the hot-swap nature of SATA/SCSI to pull a few drives that are backups. This combined with a strict, regular scheduled backup scenario can help ensure your data integrity in the event of a catastrophic loss. I recommend at least a monthly full backup, and a weekly incremental. Depending on the nature and importance of your data, you may choose to do weekly full backups and daily incrementals. There are numerous companies that will send representatives to your office on a regular basis to pickup and store these backups for you.
#2: Online push-only based backup service. Similar to physical above, but much more new-age. This depends on the amount of data that you have and how quickly you get more data, and how much it would cost for extra bandwidth to process. There are numerous rsync-based services out there, or some custom software-based services that exist for companies to backup entire data-sets to servers. Most transfer your data encrypted to their servers, some even store your data based on your own private key so they cannot have access to any of the data. Ideally, you'd choose a service that keeps full backups for a period of time in the past, I recommend at least 30 days so in the event of total loss, you have (at least) 30 days to retrieve the data before it may be overwritten by invalid data.
#3: Data integrity/retention/versioning software. Basically, setup Tripwire and instead of using it for your operating system and configuration files, use it for your data files. This can be combined with a backup mechanism to ensure that your data is not poisoned thus corrupting the offsite/push-only backups.
#4: Reporting and Accounting. Ensure that all the systems you setup, your scheduled backups (on or offsite) your tripwire/data integrity checking, etc all email ALL system administrators and possibly manager(s) of the sysadmins about the status of this task. Also setting up a monitoring system (like Nagios/Zabbix/etc) to ensure that your backups took place properly, regularly is important as then you can fire off warning/alert emails when a backup procedure fails. It's just as important to get an email that something worked, as it is to get an email about something NOT working. Eg: if someone accidentally turned cron off, the backup procedure might not take place. If your backup procedure was engineered in a way that it "touched" a file after a successful backup, you just need to check the age of that file from an external monitoring system, and if the age is too old, fire off emails.
Implementing procedures well thought out and well documented will let your investors sleep well knowing that even if a jerk sysadmin ran scripts on all your servers to corrupt data, or delete data, or anything, you can recover (nearly) ALL the data. The hard part after suffering from something like this is being able to ensure this sysadmin has not made numerous backdoors back into the system and/or still has damaging software installed on the servers. Something like Tripwire can help ensure that someone has not unexpectedly modified the kernel for example, and then would reduce the likelihood of having to consider reimaging all your servers.
Good and sound practices make up for a lot of your issue I think.
It's also not just a rouge admin that can cause issues with your systems and devices. What if the one guy who knows everything, or the entire team are flattened by a bus? Do you know or have root/admin passwords? Do you have configuration archives? Do you have backups? Backups are important! This above all is critical.
Sometimes finding out what the rouge sysadmin did helps a whole lot in fixing it. In the past I've used OS Fingerprinting locally. There is software out there that will scan a systems' configuration files, critical libraries and so forth. You get nice reports daily, about what changed.
Change Management is helpful in this as well! If you always know exactly what is changing, you can anticipate that in your reporting.
Hire the right people!
Look, I know many folks pad their resume/CV with the good stuff, and perhaps embellish a little on things. But you can uncover much of this during the interview process. The infamous happenings with the Wi-Fi network in San Francisco should NEVER have happened. I'm not terribly surprised though.
If you treat your people well, they will work well and are much less likely to do anything 'rouge'.
I've seen peers leaving the company write tell all e-mails that get sent to the entire company mailing list. :p )
I've seen peers quit suddenly and leave a trail of broken servers, in thier wake.
I've seen peers, just stop coming to work suddenly (yet still get paid for a month before anyone noticed)
I've seen peers insert logic bomb type things, to break things a set time after the facts (caught that one though!
I've seen peers 'accidentally' push an Emergency Power Off button in the data centre on the way out the door.
I've had to call police in to remove someone from the organization
I've seen backdoors used to remotely manipulate systems and devices
I've seen peers sneak a modem & POTS line in, to remotely kill the core network
Myself personally, the most I've ever done on the way out was wipe every bit of data on my laptop off the hard drive. ... I had password files and other documents on it. I had no idea who would have that asset after me, what level of authorization they'd have, and I could not guarantee anyone would reformat and reinstall to baseline levels. So I made sure of it :) ... had I left unresponsive systems in my trail.
You know
Sure, I could have taken an entire national network down as I resigned suddenly, but I didn't. It's just bad form!
That and I could have gone to jail for a long time
The people coming in to replace me, will do quite enough breaking of things :p
I've gone as far as to setup encrypted files on another system, that contains the root/admin passwords for all systems and network devices installed. With a master password (that gets you into all the group password files) written on a peice of paper and then sealed in a dated envelope, which the Boss' have.
In an organization done right, I think this should never be an issue.
This is a well-established risk. There's many things large enterprises (like the military) do. I work in a group that R&D's technical solutions. In short, the machines are custom and expensive. They require two persons from separate rooms to pretty do much of anything really bad. For example, data owners handle data inside the bounds and admins handle the system and approving data in/out. For a non-profit, regular back-ups by a 3rd party are likely your best, cheapest bet.
Good points - much better to prevent a rogue admin than defend against one.
On the backups, I'd also strongly suggest an offsite network backup that operates by "pull" from your main servers, i.e. only the backup server admin can login to the backup server. That way if a rogue admin decides to delete critical data, they can't also delete the backups. The backups will need to go back many versions to guard against someone corrupting data or source code then waiting months or years. For Linux and even some Windows, rsnapshot is a great way to do pull backups using SSH key-based login.
Ultimately this model is still vulnerable to a really talented rogue admin who will simply trojan the SSH server on one of the main servers in order to break into the backup server.
Hence an offsite logging server that captures remote log events from all your servers is important - though really you would need some admins with read-only rights to this as well, and their logins could be captured by trojaning another server or a client system.
For the wider audience (I know you got this right): it's rogue not rouge (French for 'red', English for a type of makeup. This entertaining typo meme has been spreading a lot. The idea of malicious admins with rosy cheeks is entertaining though - would make them easier to spot at least,,,
There is software at here which does that for you: run several copies of you data under supervision of different admins in byzantine agreement. 30% rough admins pose no problem.
as you are already doing .
The root access keys to the kingdom thing is not inevitable and there are several systems out there . It is however very inefficient, adds huge latency to most tasks and for that reason rarely used .
- is to avoid pissing off your employees. This is not so much about paying loads of money, but about making everybody feel they are valued and respected.
It is a strange thing, really; modern businesses, and especially technological businesses depend very strongly on being able to hold on to their experienced staff, since it commonly takes a year or more before a new employee is fully up to speed. So why is it that management so rarely try to understand the really quite simply basics of encouraging and nurturing qualities like loyalty, team-spirit and respect?
Make every day sysadmin admiration day and keep the admins happy.
Executive Summary: Separation of Powers. Make sure that the person in charge of your servers is NOT in charge of backups and auditing. You want an environment that permits "no malevolent activities without collusion."
Thank you. I can't believe the number of posts about "keeping your admins happy." That's nonsense - the kind of person who's willing to do millions of dollars of damage to your business without thinking through the probable consequences is not the kind of person you can keep happy. He's unhinged in some fashion.
Separate powers. On highly critical environments, you can install software that prevents admins from taking certain actions without having another admin also supply credentials, creating a 4-eyes environment, but the first step is separating duties.
There are technical measures that can be taken; it's not all about keeping an antisocial person happy.
We have a policy for that at my company. Actually, it isn't specifically related to rogue admins, but in the case that an admin leaves and doesn't pass on vital passwords, or an admin is on vacation when a problem happens, etc. Basically, the policy is all about redundancy. We maintain multiple off-site backups AND every system, procedure, etc is maintained by multiple admins. In our case, if the main admin is on vacation when a problem occurs, we have a fallback. The same strategy can be used for rogue admins, too. But if you really fear someone trashing a server on the way out, your only protection is many, many backups where each backup is under the control of a different admin, so the rogue employee can't trash the server AND destroy the backups.
Also, keep in mind that the law is very useful in this scenario. If an admin does anything bad on his way out, call the cops and file vandalism charges.
Finally, if you're going to fire an admin, call him into your office, and while he's in your office getting fired, have another admin disabling all his accounts. Make sure his ability to do damage is eliminated the second you utter "you're fired."
yeah what matters is that the systems can be rebuilt at all, even if it takes a week, it's like the really "last chance" backup scenario that isnt supposed to happen (but if it does, there's a fallback)
You are asking how to stop the people who have god access from performing a task that they have privilege to do.
Hard backups such as DVDs/Tapes work to a certain extent so long as you keep them current but you still run into the problem of a rogue admin doing damage over time. That is setting up something to ruin your backups in your backup cycle.
The only real way around your dilemma is to require double authority on admin tasks. This is unbelievably cumbersome and basically says "I don't trust you" to your admin(s). But it means that anything an admin does is countersigned by another admin. It is still very hard to implement.
Backups and checking are your best option. If someone leaves in a huff, cut their access immediately, change your admin passwords and look at their last commands executed before they left for dodgy ones. Assuming you have command history on as well. Highly recommended.
It is up to management not to be stupid, everything can be done in pairs, so why not pair up admins too....have 2 people have the same knowledge of what is needed to keep the system going, so that if 1 is let go, you have another until you find a replacement. Also, if 2 heads are better then one, 2 can bring better ideas to the table, as long as they understand they are not in competition, but working together.
Sun Tzu....
Give them incentives *not* to fuck your system:
1) Blackmail them.
2) Hostages.
For unsociable geek types, with no friends and no offspring, these methods may be less effective.
and shouldn't be treated as one. Reliable backups, separation of privileges, and security of passwords are all things that should be going on regardless of the possibility or actuality of disgruntled employees. What you need is protection from behavior, and the only pro-active method for that is a contract. Whether you have the Admins as W-2 or 1099 workers, you need to have a contract in place that clearly delineates lines of authority and responsibility, with clearly stated consequences for crossing those lines--up to and including civil and criminal prosecution.
First of all ignore all the comments above. You are asking for a technical solution and all the previous posters are just comming with platitudes and common places. You can trust your admin but the day they breach that trust all the above posters will not be there for you to explain to your employers, bosses. regulators and supporters of your organization how that breach happened and why you didn;t do anything about it.
You have the right attitude in regards to security of your data, which can be encapsulated in a very simple and straightforward maxim: trust no one.
The moment you trust somebody you are half way to lose your data, lets say your SysAdmin has umpechable integrity, but that he has a gun pointing to his head (OK, it could be anything else that coherces him into breaching your trust, but allow me this example for dramatic effect). Then what? If your data is not ssecured you are screwed. It is that simple.
Now, the realities and practicalities of implementing such a draconian policy are obvious: the more your data is secured the more difficult it is to access it, you need to balance the risks and decide if the costs of those risks actually materilizing justify the expenditure and effort of mitigating them.
Now for the technical bits.
Several modern operating systems include a model for the management of roles. So basically you can define roles for different people, so some people can do administrative work (create users, modify groups, permissions, etc) while at the same time they can't modify files not related to these tasks or even read them.
Solaris RBAC (role based access control) goes a long way to solve this, so in principle no user needs to become root to do system administration tasks, which means you can control access to regular data by normal permissions, or even better, access control lists (which makes access rights definitions more granular).
You should also consider separating the roles of the machines: nominate a host as a data host, and nominate other hosts for other functions, that way you can narrow down the system administration tasks that need to be performed on each host (so for example a rogue SA can't give himself access to a machine with privileged data, since the people managing access are part of another team with different roles assigned to them, equally you could control which hosts or desktops can actually access the data, which is an additional barrier you want enacted).
There are also 3rd party applications that deal with this, Computer Associates produces access control solutions which do all the heavy liftin for you adding a layer of control to the normal operating system's controls.
You may want to check this: http://www.ca.com/~/media/Files/ProductBriefs/access-control-product-sheet_135407.pdf
I am not familiar enough with Secure Linux (SELinux: http://en.wikipedia.org/wiki/Security-Enhanced_Linux ) in order to provide any advice, but this is something you may want to investigate.
Lastly I will say that if I was hiring a concientious System Administrator you would be one of the one of two people on this thread I would consider, based on all the posts sent so far.
The wishy-washy "trust your SysAdmin, don't make him angry" is wholly unprofessional, I really wonder how some people can actually get away with this cavalier approach to security.